Understanding the "Chances to win" indicator

The Chances to win is a statistical index which indicates the odds of a strictly positive gain on a variation compared to the original version. It is expressed as a % for any selected KPI.

⭐ Good to know We recommend following business rules before making a decision after running an experiment: - waiting until you have recorded at least 5,000 unique visitors per variation - letting the test run for at least 14 days (two business cycles)

This measurement is based on the number of conversions collected. The Chances to win enables you to determine the risk percentage (100% minus the Chance to win). It enables a fast result analysis for non-experts and simplifies the decision-making process.

Interpretation

The Chances to win indicator can take on values between 0% and 100%, rounded to the nearest hundredth and should be interpreted only if the business rules are complied with.

• Green: the Chance to win is equal to or greater than 95%. This means the variation can be implemented with what is considered to be a low risk (5% or less).

• Orange: the Chance to win is between 5% and 95%. In this case, the feature is either neutral or lacks data. You can check the confidence intervals: the further the confidence intervals are, the more you will have to wait to have enough data. There is as much chance of the variation underperforming compared to the original variation as there is of it overperforming.

• Red: the Chance to win is equal to or lower than 5%. This means the likelihood that this variation is underperforming compared to the original version is very high. Thus, the variation mustn’t be implemented as the risk is very high (95% or more).

As soon as your reliability status is reliable, this means the data is statistically relevant and ready to be analyzed.

Use case

Case #1: High Chance to win

In this example, the chosen goal is the Conversion rate. The experiment is made up of a single variation.

The conversion rate of variation 2 is 1.55%, compared to 1.49% for the original version.

The Chance to win displays 99.95% for variation 1, which means that variation 1 has a 99.95% chance of triggering a positive gain, and therefore of performing better than the original version. The odds of this variation performing worse than the original therefore equal 0.05%, which is a low risk.

Because the Chance to win is higher than 95%, variation 1 may be implemented without incurring a high risk.

Case #2: Low Chance to win

In this example, the chosen goal is the Conversion rate. The experiment is made up of a single variation.

The conversion rate of variation 1 is 18.07%, compared to 18.21% for the original version.

The Chance to win displays 0.93% for variation 1. This means that variation 1 has a 0.93% chance of triggering a positive gain, and therefore of performing better than the original version. The odds of this variation performing worse than the original therefore equal 99.07%, which is a very high risk.

Because the Chance to win is lower than 95%, variation 1 should not be implemented: the risk would be too high.

Need additional information?

Submit your request at [e-mail]

Always happy to help!