Which aspects of a CPU can potentially bottleneck a GPU?

The IPC (archetecture) is what bottlencks in your case.

Your 5ghz clocked CPU vs say a 3.5 ghz i3 (which has same amount of cores and threads as you) is like 4 fast teenagers vs 4 experience mechanics. The experienced mechancis (archetecture) are just going to work more efficiently despite the energy (clockspeed) of the teenagers.