Not really so bad. If you don't leave your PC for on for more than 16 hours a day, you shouldn't worry.
Even more so, if your CPU keeps it cool.
If you want to be on the safe side I suggest you find some kind of rubber patch/sheet the area of the heat-sink and put it between the cooper tips and the side panel, or you could just leave the case open.