Modem w/internal router and switch

I have a Efficient Speedstream 5667 dsl modem/router. I had it setup with a Linksys BEFSR41 router, routing to 2 winxp computers. But I've been told that since the modem has a router built in, the Linksys router is unnecessary, and I can use a regular switch instead. I can't figure out how to set this up. I'm guessing that I plug the 5667 modem/router into the uplink port of the switch. Then plug the 2 computers into the switch ports. But it won't work. Do I have to configure the modem differently? It worked with Linksys router at the default settings, I never disabled the router in the modem or anything like that. Do I need to change my network settings? Or set static ips for the computers? I really don't understand all of this, please help!