Since it happens only during certain hours of the day it is most likely not your equipment and most likely not even the connection between your house and the telco. The problem more than likely in the device they call a DSLAM. This is where you and all your bandwidth hogging neighbors meet up in the telco office. The ISP may have very badly oversold their capacity.
Call them and have them test your line..it will most likely be fine but ask them if they have capacity issues and if they plan to upgrade. If you are very lucky you may get a tech that knows something.