You're most likely being hit by a high subscriber rate in your area or poor cable routing that is highly successful to electromagnetic interference (in other words other connections colliding with each other) - this is common with the copper cabling schemes.
In any case, evidently based on the time when it occurs, this is most likely due to high usage in your area, you should contact your ISP about the issue and see if they can do anything about it, if they identify a high subscriber rate in your area they might send some engineers out to upgrade the network in your area (if you're lucky).
What ISP are you using?