This is one of the most difficult things to troubleshoot if it's intermittent. Do you have any logs, traceroutes, MTRs, or anything else to prove to your ISP that you are taking a hit across their circuit? If this is a wireless connection, then you have a whole extra layer of troubleshooting to do.
Best thing you can do is either find logs, or at the very least start documenting when it happens, and how long it happens to start isolating what is the issue on the "network". This could just as easily be something on your side of the network as the ISPs.
It is a very time consuming issue, and there is no quick fix unless you are lucky.