Really, it's entirely up to you, but from what I've seen I can say that PhysX don't add THAT much to a game. It's a nice feature, but in the end it's not like you loose all kinds of actuall detail by not having it. (Really the biggest thing you'll loose is frame-rate). The most i've ever seen PhysX used for is little chunks of concrete or glass flying off in Mafia 2, or papers in Batman. Honestly, i don't see how something like that could make or break a game, i recommend playing the mafia 2 demo and see for yourself. PhysX is also a frame rate buster, so that's something to think about.
Also, you'll a lot of heated opinions Ranging from "Sucks" to "Best thing ever", so be warned, it might get hot and heavy up in here. Just remember that it's your choice and you can do whatever you want. It might be worth it to go searching for youtube videos that show what PhysX adds to Mafia 2 and Batman. It also might be a good idea to look up what kind of FPS those people are getting.
Hope i made at least a little sense,