Quickly Reveal Verification Results Uncovering Cloudflare’s Lies “3X Faster Than Fastly” –GIGAZINE

Cloudflare wrote on their official blog on November 20, 2021: “Our service is Fastly’s edge computing platform.[email protected]196% faster thanArticleHas been released. In response, Fastly notes that Cloudflare’s method of measuring performance is flawed, arguing that it is a “nonsensical conclusion based on statistical lies.”

Lies, Damn Lies, and Statistics (from Cloudflare): Debunking Recent Cloudflare Performance Tests | Quickly
https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests

According to Fastly, the tests conducted by Cloudflare have the following six flaws.

・ Measurement service used for measurement “CatchpointIt is suspected that the nodes of “” have been carefully selected and that the ones that lead to desirable results have been extracted.
・ Officially compatible with JavaScriptCloudflare WorkersAnd JavaScript support is still in beta.[email protected]The point to which JavaScript has been operated and compared.
-The Fastly account used by Cloudflare was a free account for testing, which has more limited functionality than paid accounts and is not suitable for load testing.
-Since the test time was only one hour, it was difficult to incorporate the effects of unusual events and specific traffic patterns that occur daily into the test results, but it was easy to reflect random bias.
-Although the test code says “I ran a function that only returns the current time”, the sample code that returns a copy of the received request header is displayed after that, and any of the descriptions are incorrect. Therefore, the objectivity and reproducibility of the test are not guaranteed.
-In a test that has almost no computational load, the payload is not that large and the platform API is not used. “Time to receive first byte (TTFB)In other words, evaluating server response time is not a meaningful performance measure.

Based on the above points, Fastly “expanded the number of nodes from 50 to 673” “extended the period from 1 hour to 1 week” “measured with a paid Fastly account instead of a free trial account” “compiled with Rust on instead of JavaScript “We performed a test to measure TTFB under the condition” use the WASM binary that was created “.

The result is below. Of the six regions, Fastly was shown to have shorter TTFBs, or better performance, in four regions except Asia and Africa. Especially in North America and Europe, it was about twice as fast, and in Oceania, it was about 10 times as fast. Fastly also said that JavaScript is not used: “We know that JavaScript support is important to many clients, but[email protected]I am not satisfied with the performance of JavaScript in. That’s why JavaScript support is still in beta. “

As Fastly points out, TTFB alone is not a good indicator of server performance. So, Fastly said in the Asian region, where TTFB was slower than Cloudflare, “NetworkRound trip time (RTT)“And” TTFB “were measured, and the median value was graphed as follows. Of the two graphs, the one on the left is the result of the Fastly Varnish Configuration Language (VCL) standard service type and the one on the right it is the result.[email protected]Is the result of. In particular, in VCL, the cache access was found to be faster as the RTT (blue) and TTFB (green) network are almost the same.

From this point on, Fastly says, “Cloudflare workers[email protected]It can never be 196% faster than that. ”Additionally, Fastly is working on a benchmark to assess performance in a more realistic environment, and plans to announce the benchmark result again when it comes out of a third party.