Bear in mind what the future brings. Memory is pretty cheap these days, and often a good choice to improve performance (just threw out the original 2GB memory in a Macbook and placed 8GB inside - it does wonders).
If you have 64 bit Linux VMs you can later just copy them onto your new machine with more memory and be done. My old PC had a 32bit Linux installed and I never wanted to go through the trouble of a new install for using more memory, though I did have use for it. With regards to performance, I can't tell the difference between the two, but that's just because I never upgraded a 32bit machine to 64bit. I'll probably do that in the coming days with my old PC as I prepare it for my kids. If I see any noticeable difference, I'll post here.