Your motherboard is a very important part of your build. Pick the CPU brand (AMD & Intel), your Socket set (example: AMD=AM2, 939 ; Intel=775, 771). Each CPU and Socket set will have a Chip Set on the motherboard that will allow you to have better timings, more over clocking capabilities, SLI or Crossfire, etc... Also you will have different ranges of FBS. This will also affect which CPU you pick and are stuck with for example Intel Core 2 Duo are 800/1066/1333 FBS so you will need to find a motherboard that can use all of the FBS or at least two of them so you can possibly upgrade to the highest one later. Also the new motherboards have a PCI Express 2.0 which is more recent and will allow a little boost in your graphics card over the older PCI Express x16 motherboards.
The brand is also a major thing to pay attention too. I have had great motherboards from Abit, Asus, Gigabyte, & evga. I have had horrible motherboards that have given me nothing but problems such as MSI. So read up the reviews online from tomshardware or anandtech to see how they rate the motherboards. Also check out some of the user feedbacks on neweggs for the motherboard your interested in.
A 32bit OS can use 3gigs effectively but a 64bit OS can support up to 16GBs of RAM if I am correct. I have 8 gigs and I'm using Vista 64 and I have had no problems out of it so far. As far as I know I'm one of the few that has not experienced any problems with it. Vista is on my gaming machine and I have played Crysis, World of Warcraft, BF2, BF2142 and COD4 with no issues yet.
I hope this helps somewhat, you can read some articles here at tomshardware under the motherboard section that will go into more depth that this brief explanation