It's less a matter of a limitation on the motherboard (though there clearly are some that do have limits), and more a matter of how much address space the OS has. The limit for a 32 bit Microsoft desktop OS is 4GB of address space - but this includes *everything* that's installed, and not just RAM: devices, communications, video.. Anything with Memory Mapped I/O takes a portion of that address space, and RAM is last in line for it's share. I've seen people argue they want their memory addressed first, and although that would give RAM it's full allocation what would happen is nothing else on the computer would work - because the OS doesn't have an address to use to communicate to/from it.
Now - Going to a 64 bit OS will *very* effectively solve the address space issue {XP64 supports 128GB of address space, which is waaaaay higher than anything commercially available for desktop usage}. A 64 bit Windows OS will run your 32 bit Windows code pretty much unsullied. While not perfect, IMHO Microsoft deserve a LOT of credit for making this work as well as it does. Performance~wise, it's pretty much a wash because there's a little bit of extra work that needs to be done by the OS to run 32 bit apps, and 64 bit registers consume a little extra address space than 32 bit ones. It's not a big overhead, like some may want you to believe, since the binaries run still natively. But there is a second set of libraries and some 'traffic cop' redirection going on in the background.
The catch is that a 64 bit OS requires native 64 bit drivers. For newer hardware, this isn't an issue because Microsoft issued an Edict along the lines of "Thou Shalt Write Native 64 Bit Drivers For Everythhing With A Windows Logo, Forever and Anon. Praise Be to The Great Gates". But for older devices, it's.... different. In your case, for the hardware you listed, I would strongly advise you stick with XP. If you are going with a new build from scratch, then I would advise going straight to Win 7 X64 (available on pre-order - Oct 23 release date), and planning your build for the end of the month.
As far as "taking memory out of the RAM" - I'm unclear what you mean by this. RAM is memory. The limitation I referred to in the beginning of the post covers this: You have a 4 liter(GB) bottle that all the computer's 'stuff' needs to fit inside. So you put some in the bottle for communications. Put some more in the bottle for Installed devices. Then put even more in the bottle for Memory Mapped Video... At that point, there's clearly much less than 4 liters worth of space left over. If you then try to add 4 liters of Memory to the already partially full bottle, it doesn't all fit. That's the 'issue' with 32 bit desktop Operating Systems: The address book is only so big.