They are 2 different standards which cannot talk directly with each other. You need a router in between to translate between them.
RJ45 is 10base-t/100base-t/1000base-t which uses up to 8 wires to send data in both directions along the cable at the same time.
RJ11 is basically your telephone line which consist of a 2 wires along which all data must travel, data can only travel along the cable in one direction at a time, either in or out but not both at the same time.
Lastly, there is a big difference in speeds between the two standards. RJ11 was for modems which got up to 57Kb/s, RJ45 is ethernet which started at 10Mb/s.
EDIT:ignore the last bit as I clearly forgot about adsl.