most would not, unless your ears are sensitive enough.
my setup, asus xonar (forgot the model) which i use for my speakers, and corsair 2100 wireless headset (its usb so its not using my sound card)
i just switch from windows when using either headset or speakers, it saves me the pain of reaching behind the case to plug/unplug the speaker/headset jacks