Only if one or two of the graphics cards aren't dual slot, AFAIK, µATX only has four expansion lanes, and the vast majority of motherboards have their first and second PCI-e lane have a PCI or PCI-e 1x slot between each other, and one right after the second PCI-e 16x slot.
What would happen if I did CFX with a dual slot and single slot card? I mean, mostlikely the dual slot card would have higher clocks than the single slot, so would I have to underclock the dual slot card?
If you CFX with a single slot card and dual slot card of the same type, then there wouldn't be any performance difference, the dual slot sometimes has higher clocks, but the majority of the time, they are the same. The cards you suggested are actually dual slot because of the cooler which partially blocks the slot directly underneath the fan, however this is not a problem if your expansion card is short. Also, both HD 5770s should support OpenGL, except that XFX may not have shown such information on their website.