It does this to conserve power because it doesn't need to use those cores.
Yeah, Modern CPUs underclock themselves to save power. The only time you'll see all 8 cores at max is when you are running something that needs all that power.
It was doing this while I was doing anything. I only noticed when Assassin's Creed I started running real bad. I changed affinity to the first two cores, and continued on to find they were the only ones running at normal speed, even when the others were being heavily taxed and slowing the game down.