It's a combination of some technical reasons but mostly it's marketing. Intel certainly could host the Cedar Mill in a 478 format if they wanted to, but by continually changing the socket, they assure themselves and the industry of continual upgrade and renewal, and that's worth a lot of money.
Technically, the trend over time has been to put more pins on the CPU. Most of these pins put power into the chip and you can understand why they need that! Dual channel on chip memory controllers on Athlon's also drive a lot pins to carry those signals along with the separate pins for the FSB - hence 939 pins.