could be a number of things:
router setting to throttle specific home network ip address.
voip priority (router), connecting to a voip will set your computer into a higher priority if that is enabled in router
wired vs wireless ... your brother is wired and you are wireless, though this doesnt greatly affect ping unless your router is a cheap isp freebee sitting in another room.
Your computer is full of spam, your brothers is clean. stop surfing for prawn when your brother isnt looking.
you share a network connection 'through' your brothers pc .... making his the master and you get the left-overs.