The GTX 460 should work, but performance will most likely be constrained by the older pci-e 1.0 slot (but pci-e 2.0 is backwards compatible) and by whatever CPU you have which most likely will be underpowered for this card - based on the mobo specs for the CPU.
The mobo is a microATX form factor so you need to ensure yourself that the card will fit within the physical dimensions of the case. Further, the GTX 460 requires a lot more power so you need to check that as well. You did not say what system you had but my guess would be you will need to upgrade the PSU as well. The thermal design limit for the card is 150w.
Finally the card runs hotter than your old one so you need to be sure that your cooling system is adequate.
The GTX 460 might be the best buy even if your system does not take use all of its power. It has more recent technology that will run cooler than the GTX 260 - and will outperform it substantially for only a little more in cost. And it will outperform the 5770 for only a little more.
A standard 750w PSU is more than ample for the GTX 460 but I could not find any specs on yours. Make sure it has the two 6 pin connectors required for the 460.
The 460 is a small card at only 8.5" but I could not find any specs on the 8800 GTS and off course know nothing about your case or mobo.
I could not tell much about your computer case from the pic - other than it appears to be micro-ATX form factor. But if you know the model number you should be able to go to the HP website and get more info on size and cooling. You can also go to their support site and search on the model number and "overheating". You might also just try a Google search to see if it reveals anything. You might also Google for the model number and "reviews" to see if any other info is available.
With the lack of info and my lack of familiarity with HP systems, I do not know if stock cooling is adequate, but certainly and OC version of the 460 will run hotter than one built to the standard spec - but you might find one that has special cooling and runs no hotter or even cooler. You would need to read reviews in more detail to find if there is one.
The PCI-E 1.1 mobo is compatible with the 2.0 - but compatible just means it will provide stable operation. The 2.0 standard - on the card or mobo - provides for twice the capacity of the 1.0 standard. So if you get a 2.0 card, it is still limited by the mobo running at the lower capacity. Further, your CPU may constrain a 2.0 card. Exactly how this all works out in your particular system I don't know. THG did a recent article on balancing CPU and game card for some select models here:
The article reports on neither your CPU or the newer 460 graphics card. You will need to use the THG charts to try to estimate which are the closest approximations. As the article indicates, this is one area that little hard data is available. Also don't forget that you will be constrained by the PCI-e 1.0 mobo and assumes adequate cooling.