The GPU does not mount in a Ram (ddr2) slot but in a PCI-express slot where your current card is. The HD5770 is a good card for your setup. System ram and installed ram on a dedicated GPU have nothing to do with each other.
The Pci-e slot accepts card with gddr/gddr2/gddr3/gddr4/gddr5 regardless of what ram your motherboard is made for. The installed ram on the Graphics card only communicates with the GPU on the card, it has nothing to do with your system memory.