Yes, your computer has to load your desktop. The more data that is stored on that desktop, the longer it takes to bring it all up.
One solution, is to take large folder (such as music), move them to a folder (such as My Documents), and make a shortcut on the desktop.