You can't with a domestic router.
Instead if your brother's downloading, access the router's setup screens and reboot the router. That'll chuck him off and as he's probably not watching the computer he won't notice til later. You can get the same effect by setting up Wireless Access List (which includes or excludes users by the MAC ID of their equipment). Brother annoying you, activate Access List.
Tom's takes no responsibility if he gives you a good hiding when he finds out.