Yes. It should fit as long as your case can fit 10.5 inch cards.
And why would you get a GTX260 over a 5770? 5770 is newer tech, runs much cooler, uses much less energy, only needs 1 power connector, and supports dx11 and eyefinity. Only very few games use physX anyways.
If you have windows 7 and you really want physX, get the 5770 and a cheap Nvidia 8x00 or 9x00 card so you can run both ATI and Nvidia drivers (older Nvidia driver crack).