Do you have a limit set on your download speed in the settings? Otherwise you are most likely being throttled by your network adapter (Killer NIC for example is capable of this) or your ISP. Possible your router has some QoS settings enabled or you are trying to download something else while downloading with Steam.
Due to my location it always picks Washington DC as my server but it is frequently overloaded. When I switch to something farther away such as US - Columbus it works normally.