You need to be clear on one thing: Latency is governed by the speed of light in a wire. The majority of that 100ms is probably determined by how far you are from the server in question. Unless you can break the laws of physics, you will hit a wall at some point.
Yes I am sure you are right. I tried some servers that are a little closer to me and I can get latency of 50ms of less on them at certain times.
Actually it is not the distance of the server from my connection geographically that would make the difference... but really how many nodes it has to hop through to get to me.
If I get a faster download upload speed do you think this could help latency if I got 16 or 25 mbps and 1 mbps upload for example?