Google yesterday evening took to its Gmail Blog to talk about what the company dubbed a "Big Deal." Detailing that the outage was because of a miscalculation of capacity, Ben Treynor, VP Engineering and Site Reliability Czar, wrote that the company had already thoroughly investigated what happened, and was compiling a list of things it intended to fix or improve as a result of the investigation.
It's nice to know the team has things under control, but what actually went wrong? Treynor says that yesterday morning the team took a small fraction of Gmail's servers offline to perform routine upgrades, a procedure that normally goes off without a hitch. Ben continued on to explain that this time the team underestimated the load that some recent changes (ironically, some designed to improve service availability) placed on the request routers. At approximately 12:30, a few of the request routers became overloaded and so, the load was transferred onto the remaining request routers. More became overloaded and it sort of went from there until they were all down.
Steps Google is taking to ensure the same thing doesn’t happen again include increasing request router capacity, and figuring out a way to make sure problems in datacenter A don't affect datacenter B.
How many of you were unable to access your Gmail yesterday? Let us know in the comments below how you dealt with the outage!
Lexar's latest SD 8.0 card is too fast for its own good — potentially DOA since no current devices exploit the card's 1,700 MB/s speeds
AMD EPYC CPU hacked onto B650 motherboard, hits 6.6 GHz with liquid nitrogen — $159 EPYC 4124P shows immense overclocking potential
AMD’s laptop OEMs decry poor support, chip supply, and communication — OEM complains the company has "left billions of US dollars lying around" due to poor execution: Reports
