Because your system actually runs the ram at PC2-3200. The reason PC2-5300 will work is because it can fall back to PC2-3200. The problem you have is that PC2-6400 will only fall back to PC2-4200 speed, hence, the error lights.
6400 will fall back to 4200, you are correct. However, your system actually RUNS the ram at 3200, so the 6400 would need to fall all the way back to 3200, which isn't going to happen. Hence, the reason they specified PC2-5300 as the fastest it will allow (because it can fall back to 3200)
One thing i didn't mention is that with that particular Precision workstation, you need to use ECC Registered memory. I don't know what you ordered exactly, but when you get the correct speed, make sure the new stuff is PC2-5300 ECC Registered.
I do have the most recent BIOS update(ao7) which, like you said - enable the use of pc2-5300.
One more question- to ease the cost of the higher-prices ecc registered memory, could I couple it with some of the existing dimms, and therefore buy less. Would there be any issues with say- buying 4gb of the ecc registered stuff, and using it with 4 of the 512 dimms that are currently installed, filling all 6 slots with 2gb in two slots each, and 512 dimms in the other 4 each?
To do that, you would need to MAKE SURE that the RAM you order is SINGLE RANKED, ECC REGISTERD. Just make sure to match the speed of the existing RAM to prevent any compatability issues. While technically you should be able to order any compatible speed and it should fall back to match the rest of the RAM, I would just match them all right from the start.