An internet connection to a site/server is not a point to point connection. There are many intermediate connections. Your download speed will be representive of the slowest link in the chain. This is beyond anyone's control.
What you are seeing is the routers your connection needs to pass trhough on it's way to youtube and possibly on it's way back to your computer. You will notice that some are in different cities. If any of these routers is congested or is not performing at it's best, then throughput is effected - I.E. speed slowdown.
Also, many people watching th same video simultameously may effect it. If your ISP uses burst, you will notice your connection slowing down after the first minute or so of downloading.
How do you know it's going through the same routers? Youtube has many routers. Also, if your ISP is bursting, download speeds will always drop after a few minutes. You will have to argue this with your ISP.
You already said that speedtest.net shows you can get 9Mbit, so your connection is good. There's nothing else you can do.