I'm not looking to spend much more than $40, so if you could limit your suggestions in that regard, please. I also have to say that this will be my first build, and have ZERO experience overclocking. I don't know if there's such a thing as overkill when it comes to cooling (...right?), but will the coolers on that short list be too much if I decide to take the 955 from 3.2 to 3.5-3.7? Plus, are they too big physically?
None of those i would choose to be honest. The best budget cooler is the Hyper 212+ from Cooler Master. The great thing about Newegg is its prices, but don't by the Hyper 212+ from Newegg, they over price it 25$ from everywhere else. Check Amazon or Frys.com they sell it for 25-30$...
IF your worried about dBa then just get two fans that are quiet, that or if your worried about cooling add another high CFM fan on the Hyper 212+ for dual fan and it'll cool like a champ, even after the extra fan it'll run you at maybe like 35-40$ Right in your budget, the stock fan is good enough to be honest.
That review shows that the Hyper 212+ remember a 25$ cooler can be compared with 50-75$ coolers So you should really choose this.
Also since your a newbie XD at OCing temps do matter in a way, before my new IC7 paste i couldn't reach 3.9 stable, but now i can. So the lowered temps really helped. There is no such thing as overkill, but to a standard user, you really never need to go to water cooling =P to be honest.
Here's a AMD BE OC guide, read it and you'll understand.
You can't really argue against the virtues of the 212+ for price and performance and also it's ability to fit in most mid-towers without a hassle.
It's your best bet cuz the Mugen 2 is a better cooler imo but it's huge and a PITA to install especially for first timers.
You really can't go wrong with the 212+.