Fastly is faster at time to first byte. It makes a difference.

Created on November 12, 2023 at 10:24 am

Fastly is faster at time to first ORDINAL byte. It makes a difference.

Fastly is 43% PERCENT faster at time to first ORDINAL byte than Akamai ORG CDN, and 32% PERCENT faster than other CDNs, and here’s why this is representative of overall performance.

Measuring CDNs CARDINAL against one another is hard. There are a lot of valid questions like what metric to use, what dataset to use, and what sites to look at, but we’re always up for a challenge at Fastly, so we decided to do it anyway. The numbers speak for themselves, but in the rest of this post we’ll break down our methodology and rationale so that you can see for yourself that you’re faster (and safer! ) with your traffic on Fastly.

Why we chose Time ORG to First Byte

TTFB is the only available metric found in real-world usage data that can be directly attributed to the CDN ORG that delivered it across a large set of sites. Other metrics like Largest Contentful Paint ( LCP ORG ) can be impacted by things like client-side JavaScript execution, poor website configurations, and third ORDINAL party content embeds. And then you have to start thinking about multi-CDN situations where the domain and first ORDINAL byte may be delivered by one CARDINAL CDN, but then one CARDINAL or more other CDNs could be used for other aspects of content delivery. There’s no analogous method to attribute content delivered after TTFB to specific CDNs CARDINAL in the same way we can for the first ORDINAL byte.

It’s usually not that content gets delivered by one CARDINAL CDN this time, and a different CDN the next time, it’s that large organizations (like the Fortune 500 for example), often have multi-CDN setups where different kinds of content are delivered by different CDNs CARDINAL . Video may come from one, images from another, and some other kind of content from a third ORDINAL . And every time you hit that site you get this multi-CDN experience that contributes to the LCP ORG performance. This simply can’t be teased apart.

But just to be sure, we went an extra step and looked at data to confirm that the benefits we saw in TTFB for Fastly customers was related positively to their relative LCP ORG performance, and it was! More on this below, but it means that the performance benefits ( 32% PERCENT better than other CDNs and 43% PERCENT better than Akamai CDN PRODUCT ) were not outliers or at the expense of other performance characteristics – these gains were accompanied by significant LCP ORG performance benefits as well.

We believe the best way to examine this is to look at data that is collected from real users, using real browsers, all around the world, and looking at TTFB allowed us to do that thanks to Google ORG ’s Chrome User Experience Report ORG ( CrUX PRODUCT ) and dataset.

Why we chose the CrUX PRODUCT dataset

CrUX PRODUCT is the official dataset of Google ORG ’s Web Vitals program, which is an “initiative by Google ORG to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.” In short, Google ORG runs this program to share what they think is important, how they measure it, and what they think “good” performance is so that site managers know what they need to try to achieve in order to get credit from Google ORG for improved performance, which comes in the form of better Google ORG ranking scores, and better SEO performance (and therefore more traffic).

Because this is gathered from real Chrome ORG users across the globe, the data can be trusted to reflect real-world experiences when visiting websites rather than something more synthetic. It’s collected from real user data over time. This real-world data is a better reflection of how things like cache hit ratio (CHR), server proximity, optimized routing, and efficient load balancing improve website performance because the data is collected over different geographies and at different times, during peaks and troughs in traffic, and reports performance of how a site handles load at different times. It’s reflective of what’s happening when you don’t set up the experiment yourself–when you don’t have control–and shows how people experience your site in the wild. CrUX PRODUCT also comes with the benefits of having an easy-to-use API ORG to allow us to query the relevant data repeatedly over a timespan, and also being a well trusted source based on a huge volume of data.

TTFB itself is one of Google ORG ’s Web Vital ORG metrics, but not a “ Core Web Vital WORK_OF_ART ” metric for Google ORG , even though it is collected along with the other CrUX Web Vitals ORG data. This means that it doesn’t count against your site’s score with Google ORG if your TTFB performance is outside their recommended window. For that Google ORG looks to things like LCP ORG for the reasons mentioned above, but a solid TTFB is still critical for site performance. TTFB precedes LCP ORG , and therefore affects LCP ORG . By measuring LCP Google ORG is factoring in the TTFB performance along with other important metrics at the same time. The point is that it’s still very much worthy of attention when other metrics are not available (as in this case).

Caveats PERSON on CrUX PRODUCT data:

The CrUX PRODUCT dataset delivers times for things like TTFB and LCP ORG as “P75” measurements. This means that the number that is returned is the worst value for that metric for the top 75% PERCENT of users over the last 28 days DATE . This removes the extreme low scores to give a more accurate reflection of site performance. The Web Vitals metrics drop the lowest 25% PERCENT in order to make room for things that are out of the site’s control. This set could include user experiences from very slow connections or devices, or other transitory problems that impact a load time in ways that should not factor into a performance score negatively. By dropping the lowest quartile Google ORG can trust that the measurement is a better representation of actual performance on the site, and not skewed by outliers.

The data is only collected in the Chrome ORG browser, and not in iOS. While this still covers a huge diversity of experiences, it does mean that the data is limited to Chrome ORG users on Android ORG devices for mobile data because of iOS restrictions on data collection in apps. On desktop it is limited to data from the Chrome ORG browser (no Firefox ORG , Safari ORG , Edge ORG , or others), but it does include all desktop platforms like MacOS ORG , Windows ORG , and Linux PRODUCT . It also does not include traffic or data from China GPE .

We feel strongly that the value of real-world data from CrUX ORG is worth the trade-off of losing the representation of, for example, iOS ORG and China GPE data. Furthermore, we expect performance differences to be similar on iOS and in other browsers if not the same, because TTFB PERSON is about the first ORDINAL bit of data to the browser and device and browser types should not impact these results meaningfully.

For more information, here is Google ORG ’s methodology on which users get included in CrUX PRODUCT data: https://developer.chrome.com/docs/crux/methodology/#user-eligibility

Why we chose the Fortune 500

We know it is critical to represent organizations across many verticals and industries. We also wanted to include representation for large organizations, complicated web experiences, and a variety of use cases for what an organization needed its digital presence to achieve. We landed on the Fortune 500 as a starting point, representing the 500 CARDINAL largest U.S. GPE companies by revenue across all verticals and industries who have different needs and goals for their websites. They tend to be large and complicated organizations with large and complicated digital presences where the role of a CDN ORG might be more important in day-to- day DATE business than a personal website or even a regional company. To populate our list of Fortune 500 LAW domains, we pulled a list from this site on October 17th DATE . Fortune’s original list is behind a paywall and we wanted everyone to be able to see what we used, so we opted for this re-publication of the list.

How we determined which CDN ORG delivered the first ORDINAL byte

After we decided on the Fortune 500 as a starting point we used our internal CDN LOC detection tool to run twenty CARDINAL detections against each origin from Oct 17, 2023 DATE at 11:19 PM TIME to Oct 18, 2023 DATE at 4:07 PM TIME .. For each detection, we queried Google ORG ’s public DNS ORG server, made an HTTP request on the origin, and detected the CDN ORG by checking the following content against a config file of known providers and values:

HTTP response headers (server, X-cache, etc.)

CNAME PERSON records (*.fastly.net, *.edgekey.net, etc.)

IP-to-ASN mappings

We encountered two CARDINAL website origins with multiple CDNs for the origin hostname. This is where even the first ORDINAL byte may be delivered by different CDNs CARDINAL for different requests, so they were excluded from the analysis. We found 35 CARDINAL website origins that exceeded the 60 second timeout TIME period while awaiting headers, and they were excluded as well. (We note that 34 CARDINAL of these appeared to be served through Akamai ORG and one had no CDN. Including these in the analysis would have improved Fastly’s performance advantage to 46% PERCENT vs Akamai ORG ’s CDN ORG , and 34% PERCENT vs other CDNs CARDINAL .)

Faster TTFB PERSON with Fastly also means faster LCP ORG performance with Fastly!

For extra credit we decided to look at the relationship between TTFB performance (Fastly vs. Akamai ORG and others) alongside LCP ORG performance (Fastly vs. Akamai ORG and others) to make sure that the improved TTFB performance for Fastly customers was indicative of improved performance in other metrics like LCP ORG . If companies were using a method to improve TTFB ORG that worked at the expense of the LCP ORG or full load times it would show up here. Or if Fastly’s performance superiority was limited to TTFB with worse performance than the competition on LCP ORG it would show up here. But guess what, we won by double digits on LCP ORG vs. Akamai ORG too. Akamai ORG was 21% PERCENT slower than Fastly for LCP ORG , and all CDNs combined were 7% PERCENT slower vs. Fastly.

The Fortune 500 list was too small with too many multi-CDN deployments to check LCPs effectively, because you can’t give credit to a single CDN for the LCP ORG score when multiple CDNs ORG were involved in achieving it. You can only compare sites with “full site deployments” where only one CARDINAL

CDN ORG is delivering the domain and all of the content. Very few companies in the Fortune 500 had full site delivery, so we decided to test against a larger dataset. We looked at the CrUX PRODUCT top 10,000 CARDINAL site list and ran the same CDN detection method, running 20 CARDINAL

CDN ORG detection requests over the course of a 24-hour TIME period. If all of the domain and asset providers came back from a single CDN then a site was labeled as using that CDN ORG for full site delivery. Even with the larger data set we were only able to confidently identify full site delivery for 176 CARDINAL domains served by Fastly, and a little over 3 CARDINAL times that amount for Akamai ORG .

With that list in hand we then queried the CrUX PRODUCT API for the LCP ORG performance of those full site delivery domains and looked at how the TTFB performance advantage (that was also represented in this data) with Fastly related to the LCP ORG performance differential with Fastly. The data showed a significant performance advantage in LCP ORG with Fastly full site delivery at a 21% PERCENT advantage over Akamai ORG CDN, and a 7% PERCENT advantage for Fastly over other CDNs.

Full results

Each of the aggregate numbers below is the mean of the p75 scores for all of the website origins under each CDN.

Fortune 500 Website Origins (domainProvider):

Fastly 25 CARDINAL origins:

TTFB: 843.0 ms

QUANTITY

Akamai 116 PRODUCT origins:

TTFB: 1208.4 ms QUANTITY ( 365.4 ms QUANTITY ( 43% PERCENT ) slower than Fastly,)

All CDNs except Fastly 354 CARDINAL origins:

TTFB: 1111.3 ms QUANTITY ( 268.3 ms QUANTITY ( 32% PERCENT ) slower than Fastly)

CrUX PRODUCT Top 10,000 CARDINAL Origins (full site delivery):

Fastly 176 CARDINAL origins:

LCP: 2274.3 ms

QUANTITY

Akamai 555 PRODUCT origins:

LCP: 2757.8 ms QUANTITY ( 483.5 ms QUANTITY ( 21% PERCENT ) slower than Fastly)

All CDNs except Fastly 5741 origins:

Connecting to blog.lzomedia.com... Connected... Page load complete