Here is an illistration.
1. Open a command prompt.
2. type tracert www.youtube.com and press ENTER.
What you are seeing is the routers your connection needs to pass trhough on it's way to youtube and possibly on it's way back to your computer. You will notice that some are in different cities. If any of these routers is congested or is not performing at it's best, then throughput is effected - I.E. speed slowdown.
Also, many people watching th same video simultameously may effect it. If your ISP uses burst, you will notice your connection slowing down after the first minute or so of downloading.
All of these are out of your control.