I assume that this is the only game you are having this issue with. You may want to try to find a server with a lower ping. I don't know what server you are trying to log into. It sounds like your ping exceeds the server max threashold after about 3min. I play on the =EGO= servers and my ping varies from 160ms down to 80ms. Not the best but it still works well.
Make sure that you are not downloading anything or have other programs running in the background. That can make things worse. Good Luck.
Ummm... this is posted in the console gaming section. You don't get to pick which server to join, and you can't have programs running in the background other than what the OS allows.
@Op... is this on an XBox or PS3?