When should I upgrade my 8800GTS(320M)?

The generally held wisdom is that you upgrade when performance starts to become an issue.
Can you define what you mean by not very smooth ? it could just be a set up issue, Do you use V-Sync/tripple buffering etc, What do you have the AA etc set to.
Let us know how you are set up and it may be that we can save you some money.
Im not sure but i think youare well within the specs recomended for the game.
Mactronix

The problem is the 320MB on your card isn't enough for the high resolution you're running in. Mass Effect will be a slideshow for you if you want > X360 quality visuals.

512MB is the minimum you need. 640 or 768 would be even better if you want to use AA/AF.

Like you I have a 24" monitor and I just upgraded from a 8800gts 320 like yours to a 8800gt 512 using the step-up program from EVGA so the upgrade only cost me shipping fees.

Believe me it makes a difference, games are smoother, fps went up in most games I play (q4, bioshock, hl2, crysis, cod4) and the extra memory helps with a large monitor. But if you don't have an EVGA card (no step-up program)I would not upgade yet, I would wait and see when new cards come out. If you are having problems with only one game you could try and lower some settings (No AA) or try a lower resolution.

BTW just out of curiosity what monitor do you have ?

Yeah, as stated already, the only thing holding you back at that resolution is your memory. What you can try to do in the meantime is reduce the texture quality. That should give you a bit of performance back.

Btw, that is a great monitor you've got and for a great price.

I'm in the same boat. I am still holding out hope that the 9800 GTX wouldn't be G92 based, however, at this point it looks like I'll have to see what Ati's RV770 has to offer this summer (something good pleaaaaase!)

I purchased my eVGA 8800 GTS 320 MB in early May 2007, so I didn't get the opportunity to step up to the 8800 GT and the GTX was ~$300 more from eVGA.