Can I connect a cheap switch off newegg right to my cable modem and have an internet connection for all the computers plugged into it? Or does the switch have to be plugged into a router?

Seems to be one of those simple questions that you can't find an answer to without just trying it.

Thanks

In some cases, your cable modem might be a combo modem+router device, in which case you can just add a switch. Generally they're not, and you should add a router.

Consumer routers generally have a built-in switch, so often there's no need to add another switch. The router will handle communication to the modem/internet, and also give you a hardware firewall.

Most cable modems only give out one address. Check with your isp and see what your modem supports.

Generally you will need to get a cable/dsl router which will have a 4 port switch built in. If you need more ports you can connect a switch to the router.