I recently acquired a newer motherboard and 2 more GB of RAM, bringing my total physical memory to 4GB DDR2-667 memory. I run several programs that can benefit from the 3GB switch, so I tried it last night.
Note that I used to run the 3GB switch on my office workstation, which has dual dual-core Xeon processors and a Quadro FX550 128MB card running two 1280x1024 displays, back when I was running XP x86 at work. It worked fine most of the time.
Now when I tried it at home, graphic refresh seemed EXTREMELY lagging. For instance I would hit the Calculator button on my keyboard, and it would take nearly a second for the calculator to pop up and render all the buttons. When I hit the little arrow in the system tray to reveal all the hidden icons, it took nearly 5 seconds for that arrow to slide over and reveal everything.
Graphics card is a Quadro FX540 128MB card, currently running my 1680x1050 display on analog since the DVI output doesn't work. Could there be some kind of issue with the memory address space available to the graphics card when the 3GB switch is used? I was using a June 2010 nVIDIA driver, but I just updated to the latest one. I may try it again and see what happens.
More about :horrible graphic performance 3gb switch
The 3 gig of RAM should not have an issue at all with your video card. However the fact that you stated the DVI output does not work is a bit of a clue. Since the card is already damaged, it could easily be not working well in other areas. Try a different video card, maybe the onboard video as well.
Also make sure you have installed all of the drivers for your motherboard since it's a new one, especially update the Chipset drivers and BIOS.