Hi!
If you applyed the GPO at the top of domain (Default Domain Policy), It will work to all Domain.
Try to create a OU (Organizational Unit), apply the GPO to this OU and move all to It and leave Your users and the Domain Admins out, at another GPO.