How often does this happen? Does it happen on more then one server? I assume you get no connection error message?
When you say "warp backwards", it sounds like you had a connection error. Once the connection stabilizes it put you back where it thought you were. This has happened to me before, perhaps once or twice. (I've solely been playing this game when I came out, and have who knows how many hours logged playing it.)