Home network lag problems (internal network, not internet)

So we have a house with 4 roommates, each with a wireless connection to a netgear wireless G router. Some of the time, our internet would be fine, and other times it would be TERRIBLY slow. I realized that it was the internal network and not our connection out, because when I pinged the router while it was slow, I was getting 300-600ms responses inside our own network.

It didn't seem to matter where we were in the house, it was slow (close or far from the router). We thought the router might be crappy and can't handle all the connections all were trying (NAT table size or something), so we switched it out with a netgear N router we have that has DD-WRT installed. We're running into similar issues.

Any preliminary thoughts? I'm at work now, but we plan to do some more debugging tonight (wire into the router, try linux vs vista, double check proximity, etc). Just curious if anyone has run into anything similar or has any good tips to try out when I get home tonight. We're thinking about just wiring the whole house with ethernet and saying screw wireless altogether.