On my PC, I have a 500W PSU with 22A on the 12V rail. I have 2 HDD's, 3 fans, an Nvidia 8600 GTS, and an Athlon X2 3600 @ 2.4GHz with a 1.30 Vcore.

I believe that I should have enough power for everything; however, my 12V readings are always 11.78 to 11.84V and the 5V is always reading 4.89V, while the 3.3V is reading 3.39V. Is this normal or is the MB reporting even accurate? Also, are these voltages withing ATX specs?

ATX 2.2 spec Look on page 22

You didn't mention the make and model of your motherboard.

The readings are normal depending on the power supply. A cheap to mid range power supply will be rated at around + or - 10% which your readings would fall into. As you get up to higher quality supplies they start to guarantee rates around + or - 3%. As long as everything is working fine and you're not having any issues related to a power supply (random lockups, rebooting etc.) then I wouldn't worry.

Asus M2A-VM

I do have a problem but I don't think it is the PSU but it could be. I have obviously pushed the CPU faster than stock; however, if I set my clock to 264MHz or greater, the PC may or may not just shut off...no warning, no error, just off. Also, it does this consistently if I launch GPU-Z 0.2.6, not with 0.2.4 though.

That could be power supply related. I had to replace my power supply about 3 weeks ago. Everything ran fine until I tried playing crysis. The computer ended up shutting down and refused to come back on for awhile. At first I thought it was heat related but it ends up that when running in crossfire and things got real active that the video cards were to much for the power supply to handle so the power on my 12v rail would drop. I figured it out by running a stress test in a window while monitoring temps and voltages with asus probe. The 12v dropped to about 10v then the pc reset. Replaced the PSU with a Thermaltake toughpower 750 watt that is guaranteed 85% efficiency with + or - 3% on the rails. Monitored and stressed it as much as I could the first few days and the voltages never budged. Haven't had a problem since.

However since your messing with overclocking this could just as easily be heat related or simply you've hit the limits of what you can do with your current overclocking setup/method.