Network issue

Ok this may be long, but I'm determined to find an answer! I'll start with the description.

I have a 2 story house with the modem and router located UPSTAIRS, and my desktop (that I use for browsing and a lot of gaming) is downstairs in my basement. Let me add that I have full range connectivity, and I have had my computer located on the first floor as well. We have a WNDR3400 Netgear router. The connection speed of our internet is 20 Mpbs.

Now I'll start with the problem I've been having.

At first I thought that my latency in games was due to poor service from my ISP provider, and after a few visits out to the house there was a frequency issue that was solved, BUT the high latency within games still remained I just wasn't getting disconnected anymore. Well, after a third call to my ISP provider the tech wanted to check my speeds and told me that we were reaching our projected 20 Mpbs.

The problem now that I face is with the wireless router. I know its a dual-band router, one for "N" and "G" compatibility. The wireless adapter I have for my computer is a WUSB54GSC V2. I read that speeds for the mode should be set to 54Mbps for "G," which is currently what I have done. To no avail I am still receiving latency issues, and I've also changed the channel settings from 1, 6, and 11. Nothing in that department seems to work. I have also tested another router completely to see if the issue remained, which it did.

I'm starting to wonder if maybe the USB adapter I have is the cause of my latency problems. The only issue I have is that I don't want to go out and buy a PCI-e card or another adapter and find out that the issue still remains. Also to be mentioned is that on the network we have 5 computers running, not all at the same time either. My brother and I are pretty much the hogs of our internet connection, but I just don't see 20 Mbps being the problem as not enough speed.

Any input to this situation would be much appreciated! I love playing my games at an enjoyable speed.

Thank you Community!