It boils down to something called IPC; Instructions Per Cycle.
For every 1Hz of clock speed Intel CPUs can process / execute more instructions than AMD CPUs. For example, say AMD CPUs can execute 6 instructions for every 100MHz while Intel CPUs can execute 8 instructions every 100MHz; to keep the numbers manageable. At 4GHz the AMD CPU can execute 240 instructions. However, at 4GHz the Intel CPU can execute 320 instructions. In order for the Intel CPU to compete equally with the AMD CPU it would only need to run at 3GHz.
So why doesn't AMD simply improve the IPC of their CPU? Much easier said than done. They need to spend money on research and development to figure out how to do it. Depending on the year you are comparing, overall Intel generally spends 4x or 5x more money on R&D for all their products compared to what AMD spend on R&D for all their products.
There is also software optimization, some software are optimized to run better with intel CPUs.