Is this possible to do? I have 20 computers all connected to the same router and I want to stop access to certain websites with a proxy. Is possible to set up the computers so that it can't access the internet without a proxy?
Depends how much work you want to do and what features your router has.
Block port 80 and 443 in the router for all except the proxy
Force the users to manually put in a proxy...you can do it with group policy if you like.
Use a transparent proxy. These are either put directly in line between the router and your network. These are in a way a form of IDP when installed in this method.
Or if your router has the ability it can route all traffic for ports 80 and 443 to proxy. Very advanced routers have a feature called WCCP that allows the proxy to tell the router what traffic it wants to intercept.
Then you have to stop the creative people who will attempt to use external vpn/proxy to bypass your restrictions. There are lists of these sites, most list call these proxy avoidance as a category.