The Stats Accelerator, what is it, and why you should care?

This series of articles sets out to simplify what are typically seen as complex applications; an Occam’s Razor approach, if you like. The objective is to leave the reader with enough information to understand the value of the subject matter/tool without having to understand the complexity of the underlying mechanics.

Before we get into the basics, it’s important to understand the underlying challenges businesses face, which will add context as to why you should care about the stats accelerator and when to use it.

The Challenge:

The majority of businesses that engage with Kin+Carta regarding experimentation talk about their desire to move and learn faster. They want tools to help them increase the velocity at which they can test and learn. The stats accelerator exists for this very reason.

A recent study by cxl.com found that almost half of all businesses running an optimisation program are only running 1–2 tests per month, with just 9.5% of optimisers performing 20 or more tests per month. For those who wanted to increase the velocity of their testing programs but couldn't, lack of traffic was the most common challenge.

Why is increasing velocity Important?

In my experience, the more tests you run, the more you learn about your customers and how to grow your business. The disruptors of this World all share one thing in common when it comes to experimentation: they experiment at a high-velocity across all of their business units to increase learning and create exponential growth. The likes of Amazon, Netflix, etc clearly have an advantage in that they have millions of users per day, Amazon saw some 2.7 billion users in June 2021 which  obviously lends itself to testing at a high-velocity. But, what if you don’t have the luxury of billions of users per month – “Who does right”

The Billion User Alternative:

Now, I’m not saying that the Stats Accelerator is a substitute for Billions of users, it’s not. So, let me add some context regarding how I became aware of its existence. Through years of working with a diverse array of optimisation and testing platforms, spanning numerous verticals with varying traffic levels there has been one contributing factor that has opened the door to increasing the velocity of our experimentation programs whilst retaining the statistical quality we need to have confidence in our results, and that is the optimisation platform we use — Optimizely. Optimizely’s stats accelerator was something I personally stumbled across whilst working client-side in my search for a tool that would help us to maximise our optimisation program when faced with a reduced volume of traffic during the first phase of the COVID pandemic. So, what exactly is the Stats Accelerator?

The Stats Accelerator, simplified:  What is the Stats Accelerator?

• The Stats Accelerator was designed to speed up the time it takes to reach a statistically significant outcome through A/B/n testing. It focuses on the statistical uncertainties of the variations, using confidence intervals to dynamically update the test’s allocation in order to reduce time to finding the best significant variation.

Example:

• Typically, when a user sets up an A/B/n test (not using the stats accelerator), they choose the percentage of visitors that will be exposed to each of the test’s variants - the test’s traffic allocation. The traffic allocation e.g.,50/50, 33,33,33 remains constant throughout the duration of the test.
  • When you use the Stats Accelerator It monitors the ongoing experiments by using machine learning to dynamically adjust traffic distribution among variations—in other words, it allocates more users to the variations that have a better chance of reaching statistical significance.

Why is this useful?

• Stats Accelerator enables businesses to accelerate their experimentation velocity and reach statistical significance considerably faster than the standard A/B approach. An experiment that would have previously taken 2 weeks, could be completed in only a few days using Stats Accelerator. Alternatively, you could double the number of variations in your experiment and reach statistical significance in the same amount of time. The latter is useful, as creating more than two variations can delay statistical significance a lot longer than you might think, which, as a consequence, reduces the time to exploit the winning variant.

• The Stats Accelerator can increase testing velocity, so businesses can drive learnings and iterate at a faster pace, it’s that simple.

 When to use the Stats accelerator:

You can use the Stats Accelerator anytime you want to (accelerate your learnings) reduce the time to finding the best significant variation; however, there are caveats.  The Stats Accelerator requires at least 3 variations, including your control.

Furthermore, when Stats Accelerator is enabled, the experiment's results will differ from other experiments in four visible ways:

1. Stats Accelerator adjusts the percentage of visitors who see each variation. This means visitor counts will reflect the distribution decisions of the Stats Accelerator.
2. Stats Accelerator experiments use a different calculation to measure the difference in conversion rates between variations: weighted improvement. Weighted improvement represents an estimate in the true difference conversion rates that is derived from inspecting the individual time intervals between adjustments.
3. Stats Accelerator experiments and campaigns use absolute improvement instead of relative improvement in results to avoid statistical bias and to reduce time to significance.
4. The winning variation displays its results in terms of approximate weighted improvement.

Stats Accelerator History:

I want to end the post with some historical context regarding the Stats Accelerator.  Created in 2018 the Stats Accelerator was originally designed with two available modes ; accelerated learning (Exploration) and accelerated Impact (Exploitation)  Whilst both modes dynamically changed the test’s traffic allocation, their strategies, and intent where vastly different. The first mode, accelerate learning, is the model that is utilised in the Stats Accelerator we see today. The second mode has been removed. However, it is still available in Optimizely and more commonly known today as “Multi-Armed Bandit” testing. Given the differences and initial link with the Stats Accelerator, the next post in this series will be focused on the basics of multi-armed bandit testing.