What power supply do you have? Did you plug in the 6-pin PCI-E power connector when you tried using the 5770?
I'm having a similar problem. video card had 6pin plugged in. it is getting power but not sure from board or plug. tried it in a different slot and same thing happened. I have an ASUS p6t se motherboard. pulled it and put in an old ati video card and it booted.