...yeah, no. You would get WAY worse performance in games, unless the game was extremely graphics-heavy but very light on the CPU. The CPU isn't made for Physx calculations, graphics cards are... and the CPU has work of its own to be doing for the game.
Also, it's not difficult to change. You could have found out the answer to this one yourself by just doing that and playing a game and seeing... and you still can if you want to check us - run a game with Physx on both its lowest and highest settings while doing the calculations on the CPU, and look at both your FPS and the CPU load.