Have you taken the H60 off and tried your old cooler? Or 2, do you have another video card you can swap in and see what it does? Or 3, do you have on board video that you could use and take out your 5770 and try?
EDIT: The tubes should be at the front bac or side going into the H60 radiator to limit air trapping. You used a good thermal paste when you put it on right? And finally, the H60 pump AND fan plugs are plugged into the mobo correctly?