You probably can't do much on your end other than switching to LAN and not wifi... you might have slower downloads if you have 100mbit LAN but it will probably have somewhat lower latency.
You can ask your ISP if they can do something by threatening to switch to another
On ADSL there are usualy "fast" profiles the ISP can set that lower your latency at the cost of your max download speed but you have to ask them.... for instance my ping went from 80ms to 60ms for WoW when I asked my ISP to switch me to fast....and "lost" about 1mbit max speed.
Never hurts to ask, depends on what you have!