Most server boards are built with reliability in mind and are not necessarily faster for games but should work fine.
For example, some server boards are only spec'd for the slightly slower DDR2 3200/400Mhz ECC memory and will not run with non-ECC memory.
Also, most server boards will not come with a sound chip or a PCI-E slot. A lot of them will come with a basic onboard ATI Rage IIc video chip as well since a true server does not need a fast graphics chip.
If you can find a server class board with all of the features you want it should work just fine for you as a workstation or for games.
Hope this helps!