Overclocking CPU

Generally, yes. The amount you overclock will be directly representative of how much of an improvement you see. If a program requires different hardware to do different things (IE games with video cards) then sometimes overclocking the CPU alone won't give as big of an advantage, but it should still be there.

Running windows at 25% OC, it boots faster, loads firefox faster, converts video files way faster, etc etc. just for an example.

Depending on what you're working with you could see something like a 25% increase in performance without going too wild.
Here's an example Overclocking Intel's Core i5-750 @ TechGage
Feeling wilder? You can get aftermarket CPU coolers and get even more performance.
Makes a $200 CPU perform like the $1000 CPU.

E5200 - 50% OC
Q6600 - 50% OC
Q9550 - 27% OC