The only real reason I would say it's a bad idea is that you're limited to 4GB of RAM total, which would have to cover the OS, game, and everything else running. But aside from that, there's a negligible performance difference between the two versions.
That total of 4Gb from what I've read is that is it shared to your vcard also, so if my vcard has 1Gb, 3Gb would only be available or have I understood it wrong?
Yeah pretty much. Windows uses PAE so it's not "technically" limited to a total of 4GiB but Microsoft enforces this on 32 bit consumer versions of Windows (32 bit server versions do not have the same limitation). A single virtual address space is still limited to 4GiB of which at most 3GiB is addressable under Win32 addressing with the /LARGEADDRESSAWARE compiler flag. This limits the shared kernel memory to only 1GiB which can cause some serious issues. A certain amount of the non-PAE address space is reserved by Windows for hardware and compatibility purposes which limits the total amount of application-usable memory to less than 4GiB
64 bit versions of Windows are far more efficient and far more secure due to the availability of the NX bit and other security measures that are only available in long mode. Applications are similarly more efficient as they have access to 16 general purpose registers and 16 XMM registers which means that the CPU has to do far less data management. Likewise, 32 bit wide instructions can be macro fused into a 64 bit instruction when compiled for a 64 bit platform.
Overall, going to 64 bit from 32 bit on an x86 platform can increase overall efficiency by up to 20% but can increase an application's memory footprint by up to 100% (although this is the extreme case, 50% is not unheard of). If you have 16GiB or more of system memory then the gain is usually worth it.
Most games are still compiled for 32 bit environments which is odd because 64 bit gaming had a surge betweel 2008 and 2010 before dying off.