Google Retires Octane JavaScript Benchmark Because It Did More Harm Than Good

Google announced it was retiring its Octane JavaScript benchmark, saying most of its tests are outdated and contained bugs that allowed browser developers to cheat and obtain higher scores than intended.

The search giant said that Octane's tests no longer reflected the best way to measure performance in modern browsers, and the way websites load and execute JavaScript code.

JavaScript frameworks like Angular, React, jQuery, or Ember, which are used on billions of sites, have no say in Octane test results, which regularly execute a multitude of short tests for rarely used features. This has led to browser developers focusing on edge cases that never matter in the real world, which has hurt both browser makers and users.

Google replaces Octane with new private benchmark

To replace Octane as a way to assess browser performance, Google has developed a set of new tests that it uses internaly and which work by loading snapshots of various websites. Because these tests also factor HTML and CSS, Google engineers say they reflect a more accurate picture of how browsers behave.

This "page snapshots" benchmark is not publicly available, and Google doesn't seem interested in releasing it, fearing it will follow the same fate as its previous JS testing toolkit.

In the Octane retirement announcement, Google showed its disatisfaction with various developers that have focused on optimizing their products for the sole purpose of obtaining a high score in Octane, at the detriment of real world JavaScript optimization.

Google, who says it started to focus on how JavaScript is being interpreted on real websites, says it improved page load times in Chrome by 10-20%, depending on CPU architecture, based on "real-world performance data," instead of Octane's narrowly focused static tests.

"Comparing page snapshots [tests] to Octane revealed that Octane was a poor approximation of most websites," Google engineers said. "Given the plateau of Octane scores across web browsers and the over-optimization of peak performance, we decided to retire the benchmark as a general-purpose measure of real-world JavaScript performance."

Speedometer is a solid alternative

In the meantime, if developers really want to test their browser's JavaScript perfomance, Google is recommending the Speedometer test, built around the ToDoMVC project.

These tests work by loading various JavaScript frameworks and simulating the same user actions. While not perfect, Google says that Speedometer is more accurate at determining real-world performance, compared to Octane, because Speedometer tests focus on  load time and run time of synchronous operations (how JavaScript is normally used in browsers), and not edge case scenarios and peak performance.