Yes I would get a cheep router, that would give Internet to all of your computers at the same time, as well as provide as a hardware firewall, witch in my opinion every Internet connections needs. there are some dirt cheep ones at local stores like walmart and target, I picked a wireless G router up for 30$ a wile ago, no deal, no rebate. 30$ out the door.
So far it has been working grate!
That would give you 4 direct connections to plug in computers and there is a another conection for Internet. Not realy any setup required for wired connections, just plug everything in and it works. Wireless you need to turn on and set up if you want it.
To run both pcs on the internet at the same time you will need to get a SOHO (small office/home office) router. These incorporate a router and a switch into one box. Router handles multiple IPs and the switch handles multiple connections.
Just using a switch will not work for internet access unless your modem has a built in router.