Ok, if you're using Windows 7 follow this link & follow the instructions (it works):http://lifeandcode.net/2009/05/reduce-game-network-latency-in-windows-7-or-vista/
Secondly, try port forwarding your router for the ports that WoW uses for gaming. The ports are TCP 1119, 3724, 6112, 6113, 6114 & 4000. The ports for the downloader are TCP 3724, 6112, 6113, 6114, 4000 and additionally ranged from 6881-6999. The port used for voice chat is UPP 3724.
There are also other factors to consider with latency such as cable VS DSL and server location. You would get the best (lowest) latency if you play on a server that is held on a data center that is closest to your location.
Try my above suggestions (I too had to do these) and you should see your latency drop.