I believe its a problem with the fact that ATI and Nvidia drivers just don't like to work alongside each other. There are some sites/threads on the internet that explain how to get them to work together nicely, but I don't know what/where they are. Long ago I was researching a similar (actually reverse) setup, and found that while some people could get it to work, most just had all sorts of problems.
Its not a hardware limitation, because PCI and PCI-E are on completely separate sets of hardware and in no way conflict with each other. Proper drivers work nicely with each other, but with ATI and Nvidia specifically, there are some conflicts.
Same thing, only with AGP inserted where PCI-E is above.