yea, pretty much said above and i'll add my 2c worth.
Depends what you want to upgrade for, if there's a requirement needed to run a certain application/game then refer to the system requirement information of that program, either by searching via the web (google) or looking at the back of the box. That will serve you as an estimate guide of what is needed.
With games though, there is a minimum system requirement and recommended system requirement. Rule of thumb regarding games is treat their (game maker) "recommended system requirement" as minimum requirement if you want any chance of enjoyment in the game you're playing.
Also can look at treating the aspect of upgrading like a console in some respects. You have an xbox, to play xbox360 games, you need to upgrade to an xbox360 console. Similar regards to pc, parts inside are updated/upgraded to comply as games advance.
Every part in the computer case will either be upgraded or retired.
To put simply,
Motherboard - basis of your computer, will last as long as its needed.
Cpu - Depends on the motherboard.
Gpu - Upgraded depending on user, approx 3yrs lasting, also depending on motherboard and gpu pcie compatibility.
Memory - Can expand, add more memory, will also depend on motherboard and availability of memory.
HDD - Need more space? Maybe faster storage device like an SSD - depends on motherboard and sata connection compatibility.
PSU - Its preferred to replace the power supply unit between 3-5 years depending how much it's used.
Its an interesting hobby, nicer when you know whats what. But the internet will always be there to answer your questions.