I am planning on upgrading my current system (a shuttle) with a q8200 and an ATI 5770 1gb.
Will it overheat?
Is the card worth it?
If I had an extra £150 (I do), to spend, what would you suggest? A q9550?
It'd help to know the full specs. We can't really give you advice without knowing all the details. We also need to know what you're doing with the build and how much you have for the entire upgrade.
On thing I can say for certain is that upgrading the CPU would be a waste of money. You're already looking at an obsolete CPU (the LGA775 socket has been dead for over two years), so you shouldn't sink any more money into a dead end part. Depending on how much you're wanting to spend and what you currently have, you could either start overclocking or leave it alone and start planning a new build.
The 5770 is a great GPU, but without knowing what you have, it's hard to say if it's worth it. Since you have money to spare, a better choice to upgrade is to go to the HD 5850, 6870 or 5870 (assuming gaming as the main usage). The 5770 is good, but only at lower resolutions/details. These other cards are going to be a lot more powerful, and should fit the budget.
Ok, im home now and so canbe more specific.
I have a Shuttle XPC SX48P2E
e7400 2.8 dhz
2x 2gb Ballistix Crucial ram ddr2
Sapphire Radeon HD 4850 1gb GDDR3
I am looking to upgrade the CPU and the GPU if necessary.
A new motherboard is fine, if it fits the budget.
The budget is not certain, but depends on how much money I have after Christmas to spend.
I think the budget is about £400 give or take £50.
Your build is just fine, assuming the problem is easily to fix (i.e. overheating). I see no reason to recommend an upgrade. If I had to pick something to upgrade, I'd go with the biggest GPU you can afford, and that's it. With what you're looking to spend, that means either the HD 5970 or HD 5870.
However, you should start planning a new build. You're budget is already a good start. With about $700 (converting to US, as I don't think in pounds), you could easily afford a X4 955 with a good 8xx chipset motherboard (the ASRock 870 Extreme 3 and Asus M4A89GTD Pro/USB3 are excellent ones), 2x2 GB 1600 mhz CAS Latency 7 RAM (any brand except OCZ would be fine) and a HD 5850, 6870 or 5870 (depends on prices). Of course, that assumes your PSU, HDD and case are all still good, which they should be.
I'd say fairly soon, likely within six months to a year. More and more games prefer quad cores (BFBC2 is one of them), but since the socket is dead, I can't recommend spending any money on upgrading the CPU. You are in luck that Intel's new sockets should be out fairly soon, as will AMD's new Bulldozer CPUs, so you'd be in prime position to get the most life out of your next build. Or at least get really good parts for cheap.
Intel's new LGA1155 (replaces the LGA1156) is/was supposed to be out by the end of this year, with the LGA1365 (replacing the LGA1366) is/was supposed to follow a few months behind. I never actually saw a release timeline for Bulldozer, but it should be expected in the same timeframe. I'd start watching for news on the CPUs, but it might be hard to find.
On the GPU side, nVidia just released their high end GTX 580 and AMD just released the mid range HD 6850 and HD 6870. AMD's set to release a new high end card soon as well.