I would leave it overclocked. Anytime you overclock part you are pushing it further than the manufacturer's recommendation. So there is inherent risk. And you could shorten the life of the card. But by shorten, it may last 15 years instead of 20. It wont run a potato in 15 years so what difference does it make.
Silent mode is where it will drop the fan speed and may drop the clock speed of the GPU. If it makes a lot of noise, you may want to drop the fan speed to keep it quite. Problem is, the fans keep it cool and it will thermal throttle around 85C and start to downclock.
For me, any noise does not bother me because I game with the sound turned up pretty high. So I would overclock it and never run it in silent mode. But...