With 20Mb/s down, you should have no problem with two people on the line. I would think uploading across that 1Mb/s upstream is the choke point. Once you get over 80% upstream bandwitdth it will begin to hinder your download speed as well.
Anyhow, the solution as stated is QoS.
Okay thanks, I am looking into it