Gfail: Gmail Goes Down for Nearly 2 Hours

Google yesterday evening took to its Gmail Blog to talk about what the company dubbed a "Big Deal." Detailing that the outage was because of a miscalculation of capacity, Ben Treynor, VP Engineering and Site Reliability Czar, wrote that the company had already thoroughly investigated what happened, and was compiling a list of things it intended to fix or improve as a result of the investigation.

It's nice to know the team has things under control, but what actually went wrong? Treynor says that yesterday morning the team took a small fraction of Gmail's servers offline to perform routine upgrades, a procedure that normally goes off without a hitch. Ben continued on to explain that this time the team underestimated the load that some recent changes (ironically, some designed to improve service availability) placed on the request routers. At approximately 12:30, a few of the request routers became overloaded and so, the load was transferred onto the remaining request routers. More became overloaded and it sort of went from there until they were all down.

Steps Google is taking to ensure the same thing doesn’t happen again include increasing request router capacity, and figuring out a way to make sure problems in datacenter A don't affect datacenter B.

How many of you were unable to access your Gmail yesterday? Let us know in the comments below how you dealt with the outage!

  • Swivelguy
    Only the web interface was down. Gmail worked fine via POP throughout the "outage."
  • JasonAkkerman
    It didn't affect me. I would like to add that I hardly ever have service issues with GMail. Kudos to them for addressing the problem and being open about it, and for excellent FREE service.
  • citizenx
    It affected me as far as accessing the web site but POP worked fine. Google still rocks for getting it back up so fast. Keep doing your thing Google!
  • FlayerSlayer
    I lost access to the GMail website entirely, but my phone could still synch through POP3 just fine.
  • jerther
    ^ same here. Didn'T notice. However, time here is -4 EDT
  • megamanx00
    Not a big deal to me since I don't need to check my Gmail account every hour like some people. Good to see they are working to make it more reliable though.
  • rooket
    I've seen hotmail / msn messenger go down for longer than this without any explanation offered at all. one would expect that google would have issues from time to time as well. anyone arguing that microsoft is benefitting from this needs to really rethink what they are saying. i've already seen cnet try to say that microsoft was happy about this. i really doubt microsoft would have an opinion on the matter that would bolster their own product. just kind of shows how many hacks there are that claim to be in the IT field.
  • Mr_Man
    That's the price of free. If it's free, you can never get less than you pay for.
  • Shadow703793
    Is it me or does Google seem to have quite a few outages recently? IIRC there was one in June?
  • doomtomb
    I tried accessing it when it was down and it frustrated me for a bit then I just went about my day and did something else. Hopefully it doesn't happen again but 2 hours without email didn't kill me yesterday.