PC turns on minutes after pressing the on switch

Hello,

DELL Optiplex GX520
2 GB RAM
2X hardrive Sata and IDE
Intel Pentium 4

When I use my PC for everyday use, all is good in the world.. Until I turn the damned thing off at the wall!

When turned off at the wall, then back on (regardless of how long for) the following happens:

The Motherboard light is on, I press the on switch and nothing happens. I press it continuously and nothing! My only bet is to sit and wait. After a few minutes it just whirls into life! This delay can be anything from 2 minutes to 2 hours! Seriously, after nothing happening whatsoever for two hours, and nobody touching it, it just decides to whirl into life!

When it finally does, the CMOS time has reset itself and needs setting again, the CMOS options have all been reverted to standard settings, and it give s me the 'Diskette failure' notice and I have to press F1. I set the CMOS and the time, then startup and shut down. I turn it back on again and it starts up without a hitch with no CMOS errors of F1 warnings! Then off at the wall, all hell breaks loose again!

I've replaced the on/off switch, and I've put new CMOS battery in (Twice), but I'm not sure where to go next with this!