well yes it does effect it, be down to a hot day for example, or if the heating of the house is on and the ambient temp of the room is higher one day to the next.
But to be honest even at 40c there is nothing to worry about Maikky.
If you were looking at 90c or so as the temp reading then it would be cause for concern but you are fine with 40c. no worries.
You need to check your cooler. If it is a stock cooler with the mounting made of plastic, the mount could be cracked. Otherwise, you may need to remove the cooler, clean it and the top of the processor off, re-apply quality temperature paste, and re-assemble the fan. This cannot get better, and will probably get worse.
I have seen some cases with the stock cooler that it gets bumped and the twist clips weren't locked in place and this causes the HSF to be lose contact with the processor. Simply reach in the case and give each of the twist clips a pull straight up. If you can pull any straight up then it wasnt locked in place and you really need to reseat.