HTTP/2 Zero-Day vulnerability results in record-breaking DDoS attacks

Created on November 12, 2023 at 10:34 am

8 min TIME read

This post is also available in 简体中文, 繁體中文 ORG , 日本語, 한국어, Deutsch ORG , Français ORG and Español ORG .

Earlier today TIME , Cloudflare ORG , along with Google ORG and Amazon AWS ORG , disclosed the existence of a novel zero-day DATE vulnerability dubbed the “HTTP/2 Rapid Reset ORG ” attack. This attack exploits a weakness in the HTTP/2 protocol to generate enormous, hyper-volumetric Distributed Denial of Service (DDoS) attacks. Cloudflare has mitigated a barrage of these attacks in recent months DATE , including an attack three CARDINAL times larger than any previous attack we’ve observed, which exceeded 201 million CARDINAL requests per second ORDINAL (rps). Since the end of August 2023 DATE , Cloudflare ORG has mitigated more than 1,100 CARDINAL other attacks with over 10 million CARDINAL rps — and 184 CARDINAL attacks that were greater than our previous DDoS record of 71 million CARDINAL rps.

Under attack or need additional protection? Click here to get help.

This zero-day DATE provided threat actors with a critical new tool in their Swiss Army ORG knife of vulnerabilities to exploit and attack their victims at a magnitude that has never been seen before. While at times complex and challenging to combat, these attacks allowed Cloudflare ORG the opportunity to develop purpose-built technology to mitigate the effects of the zero-day DATE vulnerability.

If you are using Cloudflare ORG for HTTP DDoS mitigation, you are protected. And below, we’ve included more information on this vulnerability, and resources and recommendations on what you can do to secure yourselves.

Deconstructing the attack: What every CSO ORG needs to know

In late August 2023 DATE , our team at Cloudflare ORG noticed a new zero-day DATE vulnerability, developed by an unknown threat actor, that exploits the standard HTTP/2 protocol — a fundamental protocol that is critical to how the Internet and all websites work. This novel zero-day DATE vulnerability attack, dubbed Rapid Reset ORG , leverages HTTP/2’s stream cancellation feature by sending a request and immediately canceling it over and over.

By automating this trivial “request, cancel, request, cancel” pattern at scale, threat actors are able to create a denial of service and take down any server or application running the standard implementation of HTTP/2. Furthermore, one CARDINAL crucial thing to note about the record-breaking attack is that it involved a modestly-sized botnet, consisting of roughly 20,000 CARDINAL machines. Cloudflare regularly detects botnets that are orders of magnitude larger than this — comprising hundreds of thousands CARDINAL and even millions CARDINAL of machines. For a relatively small botnet to output such a large volume of requests, with the potential to incapacitate nearly any server or application supporting HTTP/2, underscores how menacing this vulnerability is for unprotected networks.

Threat actors used botnets in tandem ORG with the HTTP/2 vulnerability to amplify requests at rates we have never seen before. As a result, our team at Cloudflare ORG experienced some intermittent edge instability. While our systems were able to mitigate the overwhelming majority of incoming attacks, the volume overloaded some components in our network, impacting a small number of customers’ performance with intermittent 4xx and 5xx CARDINAL errors — all of which were quickly resolved.

Once we successfully mitigated these issues and halted potential attacks for all customers, our team immediately kicked off a responsible disclosure process. We entered into conversations with industry peers to see how we could work together to help move our mission forward and safeguard the large percentage of the Internet that relies on our network prior to releasing this vulnerability to the general public.

We cover the technical details of the attack in more detail in a separate blog post: HTTP/2 Rapid Reset: deconstructing the record-breaking attack.

How is Cloudflare ORG and the industry thwarting this attack?

There is no such thing as a “perfect disclosure.” Thwarting attacks and responding to emerging incidents requires organizations and security teams to live by an assume-breach mindset — because there will always be another zero-day DATE , new evolving threat actor groups, and never-before-seen novel attacks and techniques.

This “assume-breach” mindset is a key foundation towards information sharing and ensuring in instances such as this that the Internet remains safe. While Cloudflare ORG was experiencing and mitigating these attacks, we were also working with industry partners to guarantee that the industry at-large could withstand this attack.

During the process of mitigating this attack, our Cloudflare ORG team developed and purpose-built new technology to stop these DDoS attacks and further improve our own mitigations for this and other future attacks of massive scale. These efforts have significantly increased our overall mitigation capabilities and resiliency. If you are using Cloudflare ORG , we are confident that you are protected.

Our team also alerted web server software partners who are developing patches to ensure this vulnerability cannot be exploited — check their websites for more information.

Disclosures are never one and done. The lifeblood of Cloudflare ORG is to ensure a better Internet, which stems from instances such as these. When we have the opportunity to work with our industry partners and governments to ensure there are no widespread impacts on the Internet, we are doing our part in increasing the cyber resiliency of every organization no matter the size or vertical.

To gain more of an understanding around mitigation tactics and next steps on patching, register for our webinar.

What are the origins of the HTTP/2 Rapid Reset ORG and these record-breaking attacks on Cloudflare ORG ?

It may seem odd that Cloudflare ORG was one CARDINAL of the first ORDINAL companies to witness these attacks. Why would threat actors attack a company that has some of the most robust defenses against DDoS attacks in the world?

The reality is that Cloudflare ORG often sees attacks before they are turned on more vulnerable targets. Threat actors need to develop and test their tools before they deploy them in the wild. Threat actors who possess record-shattering attack methods can have an extremely difficult time testing and understanding how large and effective they are, because they don’t have the infrastructure to absorb the attacks they are launching. Because of the transparency that we share on our network performance, and the measurements of attacks they could glean from our public performance charts, this threat actor was likely targeting us to understand the capabilities of the exploit.

But that testing, and the ability to see the attack early, helps us develop mitigations for the attack that benefit both our customers and industry as a whole.

From CSO ORG to CSO: What should you do?

I have been a CSO ORG for over 20 years DATE , on the receiving end of countless disclosures and announcements like this. But whether it was Log4J PERSON , Solarwinds ORG , EternalBlue WannaCry/NotPetya ORG , Heartbleed, or Shellshock, all of these security incidents have a commonality. A tremendous explosion that ripples across the world and creates an opportunity to completely disrupt any of the organizations that I have led — regardless of the industry or the size.

Many of these were attacks or vulnerabilities that we may have not been able to control. But regardless of whether the issue arose from something that was in my control or not, what has set any successful initiative I have led apart from those that did not lean in our favor was the ability to respond when zero-day DATE vulnerabilities and exploits like this are identified.

While I wish I could say that Rapid Reset ORG may be different this time around, it is not. I am calling all CSOs — no matter if you’ve lived through the decades DATE of security incidents that I have, or this is your first day on the job — this is the time to ensure you are protected and stand up your cyber incident response team.

We’ve kept the information restricted until today DATE to give as many security vendors as possible the opportunity to react. However, at some point, the responsible thing becomes to publicly disclose zero-day DATE threats like this. Today DATE is that day DATE . That means that after today DATE , threat actors will be largely aware of the HTTP/2 vulnerability; and it will inevitably become trivial to exploit and kickoff the race between defenders and attacks — first ORDINAL to patch vs. first ORDINAL to exploit. Organizations should assume that systems will be tested, and take proactive measures to ensure protection.

To me, this is reminiscent of a vulnerability like Log4J ORG , due to the many variants that are emerging daily DATE , and will continue to come to fruition in the weeks, months DATE , and years to come. As more researchers and threat actors experiment with the vulnerability, we may find different variants with even shorter exploit cycles that contain even more advanced bypasses.

And just like Log4J PERSON , managing incidents like this isn’t as simple as “run the patch, now you’re done”. You need to turn incident management, patching, and evolving your security protections into ongoing processes — because the patches for each variant of a vulnerability reduce your risk, but they don’t eliminate it.

I don’t mean to be alarmist, but I will be direct: you must take this seriously. Treat this as a full active incident to ensure nothing happens to your organization.

Recommendations for a New Standard of Change

While no one security event is ever identical to the next, there are lessons that can be learned. CSOs, here are my recommendations that must be implemented immediately. Not only in this instance, but for years DATE to come:

Understand your external and partner network’s external connectivity to remediate any Internet facing systems with the mitigations below.

Understand your existing security protection and capabilities you have to protect, detect and respond to an attack and immediately remediate any issues you have in your network.

Ensure your DDoS Protection resides outside of your data center because if the traffic gets to your datacenter, it will be difficult to mitigate the DDoS attack.

Ensure you have DDoS protection for Applications (Layer 7) and ensure you have Web Application Firewalls ORG . Additionally as a best practice, ensure you have complete DDoS protection for DNS ORG , Network Traffic (Layer 3 CARDINAL ) and API ORG Firewalls

Ensure web server and operating system patches are deployed across all Internet Facing Web Servers. Also, ensure all automation like Terraform ORG builds and images are fully patched so older versions of web servers are not deployed into production over the secure images by accident.

As a last resort, consider turning off HTTP/2 and HTTP/3 (likely also vulnerable) to mitigate the threat. This is a last resort only, because there will be a significant performance issues if you downgrade to HTTP/1.1

Consider a secondary, cloud-based DDoS L7 PRODUCT provider at perimeter for resilience.

Cloudflare ORG ’s mission is to help build a better Internet. If you are concerned with your current state of DDoS protection, we are more than happy to provide you with our DDoS capabilities and resilience for free to mitigate any attempts of a successful DDoS attack. We know the stress that you are facing as we have fought off these attacks for the last 30 days DATE and made our already best in class systems, even better.

If you’re interested in finding out more, view our webinar on the details of the zero-day DATE and how to respond. Contact us if you’re unsure whether you’re protected or want to understand how you can be. We also have more technical details of the attack in more detail in a separate blog post: HTTP/2 Rapid Reset: deconstructing the record-breaking attack. Finally, if you’re being targeted or need immediate protection, please contact your local Cloudflare ORG representative or visit https://www.cloudflare.com/under-attack-hotline/. GPE

Connecting to blog.lzomedia.com... Connected... Page load complete