If you have onboard(realtek) audio, you can use the realtek manager to remap lets say BLACK to be your TV audio and when you want TV audio tell it that BLACK is front out(tell it rear out the rest of the time if you do not have surround sound on your speakers). OR you can leave black and green as front audio out and the card will play the same sounds on both at the same time. This will NOT hurt the card and it has no issues doing this.
Now if that option is not very good for you, An AV selector switch can be used in reverse(as long as it is a switch and not an electronic device.). So connect your computer to the OUT port and the TV to 1 and the other audio system to 2. This way when you switch from 1 to 2 you will send the audio(that is from the computer) from TV to your other speakers. Only one can on on at a time with this setup.
The last option(and fun one
). Buy your self a double pole double throw switch and wire the center to the computers left and right(you can use a project box with headphone jacks(female) or cables with those jacks already on the end(male).) The grounds can ALL be tied together in this case. Now wire the top to the TV cable or jacks and the bottom to the other speakers jack or plugs.
Would be something like this, but each channel would be on side of the switch