Physx isn't a ram hog, 256mb is all you need and second what are your total system specs? A 775 era system can not meet the needs of a dedicated physx card due to i/o limitations when paired with a card that already maxes out the system.
I tried different drivers and got better performance... But performance was still better with the GTX 460 alone.
It might just have something to do with the 8800GTS. It was a graphics card that was given to me. When I got the card it didn't work... But I baked it in the oven and got it working again. Thought I'd try it out as a PhysX card. I think that my problem might be faulty hardware..
Well I got a 460 my self that being the 256bit edition from an early batch and used a 8800gt as a dedicated physx card for mafia 2. I gained around 15-20% performance on average with the physx settings set to max. I used a quality 550w unit from antec that I noticed that handled higher loads than it was rated for pretty well.
I used a p2 x4 820 on a am2+ era board so over all my system would have been much weaker than yours until I later upgraded the board out to a 990fx and 16gb of ddr3 1600. As for the drivers the newest ones usually have a lot of problems for Fermi and even older cards due to the usual bugs and Nvidia trying to nerf older cards to drive sales of Kepler parts.
What you can do is test the 8800 gts on its own and see if it has any major issues.
On Nvidia's SLI compatibility charts it shows my PSU is certified to run GTX 460/560 and GTX 470/570 in SLI. It is a 750 watt power supply, 80+ bronze certified, and it has a single +12v rail with 62 amps of current. It can overload to 900 Watts and 64 amps on the +12v rail. Any more than that and the power supply shuts down. It's a very high quality PSU.
So I doubt it is a power supply issue either.
I think that it's a bad graphics card that's causing the issue.
I remember reading somewhere that with the later versions of the physx drivers really don't favor the older G80 era cards so in some of the guides they recommend G92 or newer like a 8800gt or a 9800gtx ect. Try overclocking the card with riva tuner instead of msi afterburner. Unlink the core from the shader clock and try to lower the core clock as low as you can then raise the shader clock to 1.6-1.8ghz. That should give the card a nice boost.