Cloudflare revealed that a memory leak in its parser made sensitive information, such as HTTP cookies or authentication tokens, publicly available. Some of this private data was cached by search engines (Google, Yahoo, Bing, and others), but all have reportedly purged the info from their services.
Cloudflare provides a number of services, ranging from distributed denial of service (DDoS) attack mitigation to email obfuscation, to more than 5.5 million websites. The company said this memory leak affected only a small number of users--roughly 0.00003% of HTTP requests made between February 13-18 could have potentially resulted in memory leakage--but the bug still has a vast reach because of Cloudflare's runaway popularity.
Here's the gist of the bug's impact, as Cloudflare explained in its blog post:
More concerning was that fact that chunks of in-flight HTTP requests for Cloudflare customers were present in the dumped memory. That meant that information that should have been private could be disclosed. [...] This included HTTP headers, chunks of POST data (perhaps containing passwords), JSON for API calls, URI parameters, cookies and other sensitive information used for authentication (such as API keys and OAuth tokens).
Worse still was the fact that search engines cached some of this data. Cloudflare said it found 770 unique URLs covering 161 unique domains cached on Google, Yahoo, Bing, and others. The company worked with each search provider to purge the sensitive information before this bug was made public. This should prevent interested parties from trolling search engines to find some of the private data that could have been exposed by the memory leak.
Cloudflare also had to address the bug itself. Here's what that involved:
Having a global team meant that, at 12 hour intervals, work was handed over between offices enabling staff to work on the problem 24 hours a day. The team has worked continuously to ensure that this bug and its consequences are fully dealt with. One of the advantages of being a service is that bugs can go from reported to fixed in minutes to hours instead of months. The industry standard time allowed to deploy a fix for a bug like this is usually three months; we were completely finished globally in under 7 hours with an initial mitigation in 47 minutes.
The incident and Cloudflare's response highlight the importance of companies sharing information with each other. This bug was originally found by a researcher at Google's Project Zero, which is devoted to finding vulnerabilities in online services and making their creators aware of them. Project Zero gives companies 90 days to address the problem before it's disclosed to the public; not everyone (ahem, Microsoft) responds within that timeframe.
Cloudflare did. Sharing information might not be a silver bullet, but at least in this case, it helped shorten the reach of a potentially disastrous bug. Hopefully the formation of threat-sharing groups like the IoT Cybersecurity Alliance and the Cyber Threat Alliance will encourage other companies to take warnings about their security as seriously as Cloudflare did. (Or, at the very least, cut the average response time down from three months.)
