Both routers are in the same subnet. You need to adjust the gateway address and subnet mask. Something like router A is 0.1 and assigns 0.2 through 0.100. Router B is 0.101 and assigns 0.102 through 0.200. These numbers may not be precisely on a boundary, and you need to calculate a netmask, but that should work for you.
Look in the router config pages and see if there is a way to setup an IP route to each router.
Is there a reason why you have the switch in there? You could run it router -> router. This way the routers would know about each other. You could still set each one up as you have it so you would only have one network.