You're not doing anything wrong, that's normal when tv is left plugged in.
Once you log in to use the tv, switch to the tv. This setting is lost when you shut down.
I had the same thing when I was using my older lcd monitor connected to the vga port. When I got my Samsung lcd tv I was using the digital connector with a vga adaptor to connect it.
Now I have a new monitor with a lead that has a digital connector, it's connected to the digital connection and the tv to the vga where the old monitor was, problem solved.
You can't do that swap with a laptop, so I think you're stuck with it.
This is a desktop.