Realize that you already have multicore GPUs, a GTX275 has 240 cores, a 5870 has 1600 cores. A CPU and a GPU are designed for two different types of tasks, the majority of a CPU's work is highly serial, each instruction relies on the result of the last one, so each core can only really process one thing at a time, GPUs are massively parrallel, it needs to perform the same instruction to a massive amount of data. There is quite a bit of prediction that goes on cache and onboard memory to decide what should be stored where for best performance, with two GPUs you have some overhead with the GPUs talking to each other and lining everything up properly so you dont get frame 1 2 4 3 5 6, probably about 5-10% is used there.
Also one thing that is important to note, doubling the processing power or speed of something rarely results in a 2x performance increase, if you were to test a core 2 duo and a core 2 quad both running at the same speed with the same cache per core in a quad threaded app i doubt you would see more than 80% scaling. You still have to allow for latencies with each componen., With dual gpu's the CPU now has twice as many GPUs to drive, so its a bit more load on it, and once you hit two high end GPUs you are likely at a CPU bottleneck which will limit your efficiency gain. The 80% we are at now is pretty damn good.