A few months ago on the Nvidia forums, there was a group of people debating whether using a dedicated PhysX card when you already have an Nvidia card as your main GPU would help at all. The results seemed to indicate that all you accomplish is to use more power and generate more heat.
At best, it was a couple percent improvement -- and in many cases, such as when the main GPU was fast already, using a second card for dedicated PhysX actually decreased performance.
I wish I could find that thread again, but it seems like it's dropped off the face of the Internet. Bottom line though, not worth the trouble IMO.
Also - yes, it would only have any effect in PhysX-enabled games to begin with. It's not splitting the general workload...