Yes I would get a cheep router, that would give Internet to all of your computers at the same time, as well as provide as a hardware firewall, witch in my opinion every Internet connections needs. there are some dirt cheep ones at local stores like walmart and target, I picked a wireless G router up for 30$ a wile ago, no deal, no rebate. 30$ out the door.
So far it has been working grate!
That would give you 4 direct connections to plug in computers and there is a another conection for Internet. Not realy any setup required for wired connections, just plug everything in and it works. Wireless you need to turn on and set up if you want it.