HTTP/2 Zero-Day vulnerability results in record-breaking DDoS attacks

By admin

8 min
TIME

read

This post is also available in 简体中文,

繁體中文
ORG

, 日本語, 한국어,

Deutsch
ORG

,

Français
ORG

and

Español
ORG

.


Earlier today
TIME

,

Cloudflare
ORG

, along with

Google
ORG

and

Amazon AWS
ORG

, disclosed the existence of a novel

zero-day
DATE

vulnerability dubbed

the “HTTP/2 Rapid Reset
ORG

” attack. This attack exploits a weakness in the HTTP/2 protocol to generate enormous, hyper-volumetric Distributed Denial of Service (DDoS) attacks. Cloudflare has mitigated a barrage of these attacks in

recent months
DATE

, including an attack

three
CARDINAL

times larger than any previous attack we’ve observed, which exceeded

201 million
CARDINAL

requests per

second
ORDINAL

(rps). Since

the end of August 2023
DATE

,

Cloudflare
ORG

has mitigated

more than 1,100
CARDINAL

other attacks with

over 10 million
CARDINAL

rps — and

184
CARDINAL

attacks that were greater than our previous DDoS record of

71 million
CARDINAL

rps.

Under attack or need additional protection? Click here to get help.

This

zero-day
DATE

provided threat actors with a critical new tool in their

Swiss Army
ORG

knife of vulnerabilities to exploit and attack their victims at a magnitude that has never been seen before. While at times complex and challenging to combat, these attacks allowed

Cloudflare
ORG

the opportunity to develop purpose-built technology to mitigate the effects of the

zero-day
DATE

vulnerability.

If you are using

Cloudflare
ORG

for HTTP DDoS mitigation, you are protected. And below, we’ve included more information on this vulnerability, and resources and recommendations on what you can do to secure yourselves.

Deconstructing the attack: What every

CSO
ORG

needs to know

In

late August 2023
DATE

, our team at

Cloudflare
ORG

noticed a new

zero-day
DATE

vulnerability, developed by an unknown threat actor, that exploits the standard HTTP/2 protocol — a fundamental protocol that is critical to how the Internet and all websites work. This novel

zero-day
DATE

vulnerability attack, dubbed

Rapid Reset
ORG

, leverages HTTP/2’s stream cancellation feature by sending a request and immediately canceling it over and over.

By automating this trivial “request, cancel, request, cancel” pattern at scale, threat actors are able to create a denial of service and take down any server or application running the standard implementation of HTTP/2. Furthermore,

one
CARDINAL

crucial thing to note about the record-breaking attack is that it involved a modestly-sized botnet, consisting of

roughly 20,000
CARDINAL

machines. Cloudflare regularly detects botnets that are orders of magnitude larger than this — comprising

hundreds of thousands
CARDINAL

and even

millions
CARDINAL

of machines. For a relatively small botnet to output such a large volume of requests, with the potential to incapacitate nearly any server or application supporting HTTP/2, underscores how menacing this vulnerability is for unprotected networks.

Threat actors used botnets in

tandem
ORG

with the HTTP/2 vulnerability to amplify requests at rates we have never seen before. As a result, our team at

Cloudflare
ORG

experienced some intermittent edge instability. While our systems were able to mitigate the overwhelming majority of incoming attacks, the volume overloaded some components in our network, impacting a small number of customers’ performance with intermittent 4xx and

5xx
CARDINAL

errors — all of which were quickly resolved.

Once we successfully mitigated these issues and halted potential attacks for all customers, our team immediately kicked off a responsible disclosure process. We entered into conversations with industry peers to see how we could work together to help move our mission forward and safeguard the large percentage of the Internet that relies on our network prior to releasing this vulnerability to the general public.

We cover the technical details of the attack in more detail in a separate blog post: HTTP/2 Rapid Reset: deconstructing the record-breaking attack.

How is

Cloudflare
ORG

and the industry thwarting this attack?

There is no such thing as a “perfect disclosure.” Thwarting attacks and responding to emerging incidents requires organizations and security teams to live by an assume-breach mindset — because there will always be another

zero-day
DATE

, new evolving threat actor groups, and never-before-seen novel attacks and techniques.

This “assume-breach” mindset is a key foundation towards information sharing and ensuring in instances such as this that the Internet remains safe. While

Cloudflare
ORG

was experiencing and mitigating these attacks, we were also working with industry partners to guarantee that the industry at-large could withstand this attack.

During the process of mitigating this attack, our

Cloudflare
ORG

team developed and purpose-built new technology to stop these DDoS attacks and further improve our own mitigations for this and other future attacks of massive scale. These efforts have significantly increased our overall mitigation capabilities and resiliency. If you are using

Cloudflare
ORG

, we are confident that you are protected.

Our team also alerted web server software partners who are developing patches to ensure this vulnerability cannot be exploited — check their websites for more information.

Disclosures are never one and done. The lifeblood of

Cloudflare
ORG

is to ensure a better Internet, which stems from instances such as these. When we have the opportunity to work with our industry partners and governments to ensure there are no widespread impacts on the Internet, we are doing our part in increasing the cyber resiliency of every organization no matter the size or vertical.

To gain more of an understanding around mitigation tactics and next steps on patching, register for our webinar.

What are the origins of

the HTTP/2 Rapid Reset
ORG

and these record-breaking attacks on

Cloudflare
ORG

?

It may seem odd that

Cloudflare
ORG

was

one
CARDINAL

of the

first
ORDINAL

companies to witness these attacks. Why would threat actors attack a company that has some of the most robust defenses against DDoS attacks in the world?

The reality is that

Cloudflare
ORG

often sees attacks before they are turned on more vulnerable targets. Threat actors need to develop and test their tools before they deploy them in the wild. Threat actors who possess record-shattering attack methods can have an extremely difficult time testing and understanding how large and effective they are, because they don’t have the infrastructure to absorb the attacks they are launching. Because of the transparency that we share on our network performance, and the measurements of attacks they could glean from our public performance charts, this threat actor was likely targeting us to understand the capabilities of the exploit.

But that testing, and the ability to see the attack early, helps us develop mitigations for the attack that benefit both our customers and industry as a whole.

From

CSO
ORG

to CSO: What should you do?

I have been a

CSO
ORG

for

over 20 years
DATE

, on the receiving end of countless disclosures and announcements like this. But whether it was

Log4J
PERSON

,

Solarwinds
ORG

,

EternalBlue WannaCry/NotPetya
ORG

, Heartbleed, or Shellshock, all of these security incidents have a commonality. A tremendous explosion that ripples across the world and creates an opportunity to completely disrupt any of the organizations that I have led — regardless of the industry or the size.

Many of these were attacks or vulnerabilities that we may have not been able to control. But regardless of whether the issue arose from something that was in my control or not, what has set any successful initiative I have led apart from those that did not lean in our favor was the ability to respond when

zero-day
DATE

vulnerabilities and exploits like this are identified.

While I wish I could say that

Rapid Reset
ORG

may be different this time around, it is not. I am calling all CSOs — no matter if you’ve lived through

the decades
DATE

of security incidents that I have, or this is your first day on the job — this is the time to ensure you are protected and stand up your cyber incident response team.

We’ve kept the information restricted until

today
DATE

to give as many security vendors as possible the opportunity to react. However, at some point, the responsible thing becomes to publicly disclose

zero-day
DATE

threats like this.

Today
DATE

is

that day
DATE

. That means that after

today
DATE

, threat actors will be largely aware of the HTTP/2 vulnerability; and it will inevitably become trivial to exploit and kickoff the race between defenders and attacks —

first
ORDINAL

to patch vs.

first
ORDINAL

to exploit. Organizations should assume that systems will be tested, and take proactive measures to ensure protection.

To me, this is reminiscent of a vulnerability like

Log4J
ORG

, due to the many variants that are emerging

daily
DATE

, and will continue to come to fruition in

the weeks, months
DATE

, and years to come. As more researchers and threat actors experiment with the vulnerability, we may find different variants with even shorter exploit cycles that contain even more advanced bypasses.

And just like

Log4J
PERSON

, managing incidents like this isn’t as simple as “run the patch, now you’re done”. You need to turn incident management, patching, and evolving your security protections into ongoing processes — because the patches for each variant of a vulnerability reduce your risk, but they don’t eliminate it.

I don’t mean to be alarmist, but I will be direct: you must take this seriously. Treat this as a full active incident to ensure nothing happens to your organization.

Recommendations for a New Standard of Change

While no one security event is ever identical to the next, there are lessons that can be learned. CSOs, here are my recommendations that must be implemented immediately. Not only in this instance, but for

years
DATE

to come:

Understand your external and partner network’s external connectivity to remediate any Internet facing systems with the mitigations below.

Understand your existing security protection and capabilities you have to protect, detect and respond to an attack and immediately remediate any issues you have in your network.

Ensure your DDoS Protection resides outside of your data center because if the traffic gets to your datacenter, it will be difficult to mitigate the DDoS attack.

Ensure you have DDoS protection for Applications (Layer 7) and ensure you have

Web Application Firewalls
ORG

. Additionally as a best practice, ensure you have complete DDoS protection for

DNS
ORG

, Network Traffic (Layer

3
CARDINAL

) and

API
ORG

Firewalls

Ensure web server and operating system patches are deployed across all Internet Facing Web Servers. Also, ensure all automation like

Terraform
ORG

builds and images are fully patched so older versions of web servers are not deployed into production over the secure images by accident.

As a last resort, consider turning off HTTP/2 and HTTP/3 (likely also vulnerable) to mitigate the threat. This is a last resort only, because there will be a significant performance issues if you downgrade to HTTP/1.1

Consider a secondary, cloud-based DDoS

L7
PRODUCT

provider at perimeter for resilience.


Cloudflare
ORG

’s mission is to help build a better Internet. If you are concerned with your current state of DDoS protection, we are more than happy to provide you with our DDoS capabilities and resilience for free to mitigate any attempts of a successful DDoS attack. We know the stress that you are facing as we have fought off these attacks for

the last 30 days
DATE

and made our already best in class systems, even better.

If you’re interested in finding out more, view our webinar on the details of the

zero-day
DATE

and how to respond. Contact us if you’re unsure whether you’re protected or want to understand how you can be. We also have more technical details of the attack in more detail in a separate blog post: HTTP/2 Rapid Reset: deconstructing the record-breaking attack. Finally, if you’re being targeted or need immediate protection, please contact your local

Cloudflare
ORG

representative or visit

https://www.cloudflare.com/under-attack-hotline/.
GPE