For gaming or something that takes advantage of parallel processing, then the GPU is an important choice.
In multi-threaded applications like CAD/CAM design programs or major photo/video editing applicationg then the CPU is an important choice.
Take systems that require a lot of storage, the Hard Drives would be an important choice to make in terms of speed, reliability, capacity etc
All parts in a computer must be balanced to create a good machine, you can't just skimp on one part to beef up another because that part you skimped on may bottleneck the rest of the system and constrict the potential of the beefed up part, thus wasting your time, effort and money.
Well, like what Griffolion said... It depends on what you use the computer for. If you want it for gaming you might want to consider the resolution and acceptable frame rates to decide on a graphics card. Then you want a CPU that won't slow down that card and enough memory to run the game you want to run. Then of course a power supply that can accommodate all the component's power requirement.
If it's not for gaming, figure out if the application you are running takes avantage of multi core cpu (if it support 2, 3, or 4 cores) or memory intesive and emphasize on what it needs.
Well it seems your primary concern is gaming (as streaming videos and web surfing is less intensive and thus not what i would consider the primary concern like gaming, CAD, music production, design would be).
In this case you'd want a pretty decent graphics card to run your games at good settings smoothly. On the CPU side, you'll want preferebly a quad core but if budget is a concern then you could go down to a triple core but dual core isn't massively recommended as modern games are now starting to thread to 2 cores or more nowadays.
Memory-wise, DDR3 1333Mhz with nice timings is the recommendation i'd give (4GB for dual channel and 6GB for triple channel).
Hard Drive is up to you, a nice big storage drive like a 1TB Spinpoint F3 will do nicely, but if you can stretch to it, a small Sandforce based SSD containing Windows will certainly help your boot times etc.
In terms of power supply, unless you're going to be crossfiring or SLI'ing your graphics cards then 550-650W is what you'll want to be looking at. If you are crossfiring or SLI'ing then i'd say 750 - 850W for ATI cards and the GTX 460 and 900-1000W for the GTX 470/480 (playing it safe).
Hope this helps, come back with more questions about other parts if you need it.
You can buy the fastest card and it would be slow if you don't get enough CPU power and enough bandwidth on your motherboard. Most games that aplly physics uses the cpu for now and only a few that actually uses the GPU.
I would say the PSU is most important since if you get a cheap one it could potentially fail and destroy the rest of the system at the same time. However that said, assuming you get a decent one, getting a 1000W for the sake of it gives you no real advantage over a good 400W if that's all you need, just don't be tempted to get one of those ultra-cheap ones.