Horizontally for sure in that case. Keep the air flowing one way. I'm guessing that's what you were asking? Vertical mounting isn't awful but it tends to drag air from the GPU which in turn increases temps.
Orient the cooler exhaust hot air toward the back exhaust fan. You want a clean airflow where the air goes from the front of the case, across the CPU, and the hot air sucked out the back. All fans in this line should be blowing the same direction (out the back of the case).
You can also do the same with a top exhaust fan config as well. I prefer the first option because you also should have front case fans pulling cool air into the case.
Doesn't the GPU exhaust out of the PCIE slot on the back of the machine?
Yes, but it still generates excess heat that remains inside the case. That heat dissipates and removed from the system via the airflow of your fans. You don't want to pull that excess heat directly into your CPU heatsink if you can avoid it.