Either the servers you join are hacked or poorly hosted, or you have a firewall/AV or malware software working in the background, flooding the ports your using.
Also, you might consider running a speedtest on www.speedtest.net to check your ISP parameters.
Cheers