Just stay with PCIx16 in BIOS setting.
Have you uninstall the previous card driver?
1. Uninstall the Nvidia driver
2. Boot into safe mode and use driverSweeper to clean out the remaining of nvidia.
3. Restart and boot normally.
4. Install your ATI driver.
Try it out...