[Advice Needed] Would I be better off building a new system?

Your best bet at this time would be to get a new GPU and later move it to a DDR3 mobo as you mentioned. In you price range:
http://www.tomshardware.com/reviews/gaming-graphics-card-review,3107-4.html

You are looking at:
Low end of your range: Radeon R9 270
High end of range: GTX 760

The 760 is more powerful of the two cards, but it's up to your to figure what you can afford.

Adding more DDR2, wouldn't really help, you are limited by its speed. Just make sure your PSU is good enough to handle whichever GPU you decide to get.

Your next move should be to a new mobo, CPU and DDR3.

http://www.newegg.com/Product/Product.aspx?Item=N82E16814133498
PNY 760 OC with massive OC out of the box 1111mhz/1176 boost. $229 on sale (rebate) +watchdogs

http://www.newegg.com/Product/Product.aspx?Item=N82E16814150705
Radeon 280 side...3 games. XFX Black Double D edition is great. $239 after rebate

The only two I like in the list at these prices. Ref designs are against my religion...LOL. No self respecting guy can argue with anything Double D related...ROFL.

That depends actually, I've heard XFX has had a lot of quality control issues with their graphics cards. Especially the Radeon R9 series. Now the Sapphire Tri-X on the other hand I've heard is bar none the best stock / non modified R9 cooling solution on the market right now.

Your GPU is the biggest bottleneck. Replacing that with a GTX 760 would work out well. The Phenom II 965 and RAM would limit you somewhat, but not severely.

Eventually you'd need to upgrade to a backwards compatible AM3+ DDR3 board; there are plenty of DDR3 boards that'll still work with the Phenom II 965, and the 965 isn't too weak yet. But that can probably wait a year if necessary since replacing the video card will still be an immediate performance increase.

I'd say it's worth upgrading.
If you do decide to replace the whole thing, then at least you'll be able to keep some of the components anyway.

I sold them for 8yrs as a pc business owner and am running a 5850 as we speak for years now. I'm not sure if they still do this, but they used to have a lifetime warranty which then when to 10yr IIRC and it was transferable! That is huge. Even if that isn't true now, 4/5 eggs and most other xfx cards rate that also.

http://www.amazon.com/XFX-RADEON-1000MHz-Graphics-R9-280X-TDFD/dp/B00FSC5N66/ref=sr_1_1?ie=UTF8&qid=1401187462&sr=8-1&keywords=xfx+radeon+280
4/5 at amazon with 51 reviews too. Not sure where you got this quality issue. Also a big one is you can contact them before modifying their cards and ask if they OK your mod. They'll update your profile saying it was OK, so you don't have a warranty issue later (they will even install your mod for you for a price - not sure how evil the price is but most just void your warranty period).

Just checked - LIFETIME if you register within 30days of buying otherwise 2yrs. So again, they have no fear if they are giving you LIFETIME.

"If you register any of the specified products noted above online at http://www.xfxforce.com/ within 30 days of purchase, your limited warranty will be EXTENDED for the duration of your life. Registration within 30 days of the date of purchase is a condition precedent to receiving the lifetime warranty."
http://xfxforce.com/en-us/support/xfx-warranty
Doesn't really get any better than that. That said, I sold a lot of sapphire also My card can be transferred once to a new owner it says. Of note, I never heard from anyone I pushed these to, and I did every chance I got because of the warranty back then, and the transfer which allows an easy upgrade for people dumping their cards (you can tell the person they have warranty still). Sapphire was always 2yrs, which was like most motherboard makers who do vid cards too (msi, gigabyte etc, I never pushed any mo-board makers cards back then due to the warranty). Obviously that doesn't mean a card (or any product) will fail early, but I like being able to call up 5yrs later and say, hey send me a replacement mine just bit the dust. This type of warranty means you'll have a working card probably until you don't want it (I have 2 others here...LOL just keeping around for testing crap, a AGP/PCIE - I still do builds so they're handy).

Thanks for making me look it up, I'll buy another since it's still the same policy it seems At any rate, I don't see complaints massively on any model I look up. I have no fear saying buy XFX.

http://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Daps&field-keywords=xfx+radeon&rh=i%3Aaps%2Ck%3Axfx+radeon
I don't see one model with less than 4/5 stars at amazon I suspect the same at newegg. You can't really fail with a warranty like this IMHO. It's certainly better than most I'm aware of and I was a partner with all of them. I pushed the ones that never came back and since I was friends with my fav distributor (ASI back then) I had their RMA info which helped me have nearly zero returns. I believe most people cause their own problems, with most being largely ESD which either kills you outright, or more often over time once zapped shortening the lifespan. People think they get away with it because it didn't die that second. They are mistaken. I still wear straps & gloves (at times shoes) to combat this to this day. I was a small business so I didn't have time to FIX stuff or stand in RMA lines. Only two people ever touched the parts in our builds (the other was my dad...LOL). Over 8yrs and watching my vendors complain about others in RMA lines, I can tell you not many parts fail when properly handled. Of course, assuming it's not a HD that was beat to death in shipping or something, but everything else was pretty rock solid.