Yes, I'm contemplating doing that...except that I don't think my TV has a headphone-AUDIO-IN jack. I suppose that the RCA-to-3.5mm adapter cable I linked to in my OP would serve that purpose?
Your original post says you want to listen to the audio out of your TV. All that's required for that is either a headphone jack or appropriate RCA jacks. I can't remember seeing a TV with audio in jacks. If you want to listen to a second device I think you must treat it separately.