And the 9550 is better than any AMD quad-core, as well as most of the i5s and the lower-end i7s.
Basically, you won't notice a hell of a lot of improvement going from DDR2 to DDR3, and the trouble of ripping out the motherboard (which also requires installing the OS from scratch) is tough to justify unless you're rebuilding the entire system. The CPU is the worst component in your current system, and upgrading it will give you a machine that's well above average for a few years into the future.
Also, changing the motherboard to an AM3 socket is not a very good move, since AMD announced that the next generation of 6-core and 8-core CPUs to come out in 2011 (Bulldozer) will need a new AM3+ socket that isn't out yet, and won't work on regular AM3. So if you do that, you limit your upgrade path right away. I like AMD processors over Intel any day, but for the next 3-6 months, it is not a good time to buy an AMD motherboard unless you're going for cheapness alone.
For your next project after this, you might consider upgrading the video card, since a single 5770 tends to get overtaxed when you get into the 1600px+ range.
Sure, as long as you don't mind your machine grinding to a halt with irreparable driver conflicts. If that's what you want, then you just go right ahead.
Yes, there are a few tricks that you can use to get around a full reinstall sometimes, but honestly, they're almost as much of a pain. And yes, I've heard of cases where people get lucky when they have similar enough hardware that the new board works. Going from an Intel s775 to an AM3, I'd give that about zero chance. Windows 7 might be more tolerant with swapping out devices and drivers, but it's far from guaranteed when you replace the motherboard. I've heard of it working and I've heard of it not working, but bottom line, I wouldn't count on it.
But anyway, the real point of what I was trying to say is: Why even bother with any of that when you can simply replace the CPU and end up with a better machine for less money? If you can answer that, then you're going somewhere.
Now, the Phenom 955 isn't going to be terrible by any means, and will still be a significant upgrade over what you have now. Basically, the top Intel LGA775 quad cores (q9950, q9650 and up) are last-generation, but the best of the last generation, so they are still going to beat any AMD quad-core and just about any i5. The only thing you can put into an AM3 socket that beats a 9550 in benchmarks is one of AMD's Thuban six-core CPUs -- and that's another whole can of worms to open. Because while the AMD six-cores are technically faster, you don't really get much advantage over a quad-core because most of the time, the additional cores just sit there unused, so going by benchmarks alone is a little misleading.
Three months ago, I would've said go ahead and switch to AMD anyway, because it offers the better upgrade path. But AM3 is now a dead technology just like LGA775, because the next generation of AMD processors (Bulldozer) is going to use a new socket.
As far as the RAM goes, no I don't think you'll notice a lot of difference there. RAM speed plays a lot smaller role in gaming than the power of the CPU and GPU. Also, there's the issue of latency. To figure the comparative speeds between two types of RAM (roughly), you take the clock speed and divide it by the latency. With DDR2, latency down in the 5s and 4s is pretty common ... while DDR3 is more like latency of 6 or 7 for the high-end models, but 8 or 9 for the cheap stuff. So if you had a stick of DDR2-800 with a latency of 5, and a stick of DDR3-1600 with a latency of 9, the "relative" speed of the DDR2 is going to be 800/5 or 160 MHz, and the DDR3 is going to be 1600/9 or 177MHz. If you got high-end DDR3, you'd notice more of a difference, but my guess is you're comparing the following two sets, in which case it's probably not worth the trouble:
Having said all that, there is the issue that if you spend your $300 on the Q9550, you're going to be on last-generation technology with the DDR2. Is DDR2-800 going to be laughable three years from now, while improvements to DDR3 make it noticeably better? I don't honestly know. Fairly likely. On the other hand, if you go for the upgrade you originally mentioned, you'll be on the current version of RAM, but three years from now, you'll be in the exact same situation with the CPU as you are now.
What I'd REALLY do if I were in your shoes is wait six months, save up a few hundred dollars in the meantime, then get an AM3+ motherboard and a Bulldozer CPU. That's what I'm doing for my next machine, anyway. It will get you the best combination of power and longevity if you can tough it out with your current system for a little longer.