Above poster doesn't know what their talking about. I have over a dozen rack mount systems with "graphics cards" installed in them.
Check to see if its a vertical or horizontal mount. Being a 4U case it will most likely be vertical and full slot length.
You might have to worry about two things, first being slot length. Server motherboards often only have a bunch of x8 slots that a x16 card won't directly fit into. You can do a small mod to remove the backstop off the slot to fix the card in, but it'll only operate at x16 and would most definitely invalidate your warranty. The other is the power connector, I've rarely seen a server chasis / powersupply that provided a 8x/6x PCIe power connector. You may need an adapter.
Other then those two issues, there is no reason a GPU can not go into a rack mount case.