oh, getting another 5770 from a relative of mine...tiz a present
I had an 8800 GTS running with my single 5850 on an X38 mobo without issue and PhysX was working.
Since you are hacking the Nvidia driver, and the mobo does support 3 GPUs, I think this will work. I bet the PCI-E slot speed will not be a factor as this is going to just be running PhysX, but I could be wrong (girlfriend says I've been wrong before...)
ok thas something to dig into...i'll check if the PCI-E lanes matter for physx