Yes, actually. Almost all GPU's are in one way or another bottlenecked by the CPU... though for most cards (particularly the Nvidia ones) the difference between an i7 and an old Phenom II is negligible, unless you turn the settings down to the point where the GPU isn't the limiting factor... in which case, you're wasting a lot of GPU.
Xeon processors, however, are NOT designed for gaming. They should have the memory access and wot to keep the GPU fed, but it will likely struggle with the game logic itself, meaning that many games, particularly CPU-bound physics in games, will not run at a reasonable speed.