There are only 2 things with moving parts in a PC:
HDDs, and Fans. I really hope it is just a fan as they are really easy to replace... If it is a HDD starting to go then you will want to back up as much data as you can.
To test, unplug the HDD's power and see if it still makes noise when you turn on and shut down the computer. If so, then you are in luck! It's a fan! If not, then you have my apologies.
If it is a fan, then turn the system on and either lightly touch the fans (in the center of the fan, NOT the edges) until it stops moving, or unplug the power to the fans until you find the bad one. Most fans (except for the CPU and GPU) you can live without while you wait for the new fan to come in.