The system memory and graphic card memory are separate and distinct from one another, so just because one uses DDR2 and the other DDR3 has nothing to do with the problem.
Make sure your video and chipset drivers are fully up to date. What make/model of motherboard and graphics card are you using?