Are you using wired or wireless? If wireless, have you investigated (using WIFI Analyzer for android or inSSIDer) how many other wireless networks are competing with yours? It is also possible that Comcast is oversubscribed in your area. You will probably have to get more data points (times) to determine if any pattern exists.
You aren't sharing your internet with roommates that could be impacting your bandwidth? You don't have any torrents running, correct?