I'm running Vista 64 and have 4 ddr2 slots so 6GB or 8GB would be no problem at all. I would opt to ditch the swap file, but I have seen a few places online where people have said that some software depends on the swap file and not having a decently sized one causes instability.
As for the card reader, I'd probably get a device like this
which shouldn't bottleneck the card.
I wish Vista were like Linux where you can reduce the "swappiness".