Strictly regarding size, you need a micro ATX socket 775 motherboard and a micro ATX power supply.
However, as taken from another post regarding replacement of the Optiplex GX280 motherboard:
"I doubt a standard motherboard will work in your case. I work on the GX/SX280's, GX620's and soon the GX740's. The motherboard uses metal mounting/grounding clips that are molded in to a metal surface. There might be one screw that holds the board to the surface. This makes swapping the board a breeze if you get a Dell replacement (board is usually already mounted). Unless you modify this surface and remove these clips, the board will likely have grounding issues which may cause a variety of interesting problems (especially with anything that uses RF). I've seen a lot of people do what he did. It makes sense but you end up having to puchase a case, power supply (unless the case has one), motherboard, and an OEM copy of Windows. So plan on spending $250-300, more or less."