Frequentist Analysis mode

While our primary framework is Bayesian, some users prefer Frequentist analysis. This option uses the same core data and offers a different statistical perspective.

Access the Frequentist mode

AB Tasty suggests avoiding Frequentist analysis for A/B testing due to its complexity in interpretation. Yet, we've created a feature allowing users to utilize it if necessary.

Please note that Reporting Copilot, Analysis Copilot and opportunities views are not available with frequentist mode.

1

Ask your CSM to activate the feature to your account

2

Switch from Bayesian stat engine to Frequentist stat engine

1. Access your campaign.

2. In the reporting tab of your campaign, click on the statistics engine button.

3. Select the Frequentist engine.

   
4. If you want to apply to all reports of the account, check "Apply to all reportings". If you don't, this statistical analysis, will only apply on the campaign your are currently browsing).

5. Click on "Apply".

Impacts on the data views:

• Conversion rate stats changes :

  • “Confidence interval” switches to “Confidence interval” (Frequentist one)

  • “Chance to win” switches to “p-value”

• AOV changes (transaction goals only):

  • “Chance to win” switches to “p-value” (Frequentist one)

Visual aids enhancing the understanding of the results

In AOV gain p-value and in Conversion gain p-value you will face :

• Blue artifact and stats’ font if p-value =< 0.05. Highlighting that the tests results provide potential learnings.

• Orange artifact and stats’ font when p-value >= 0.95. Highlighting that the tests results provide no learnings.

Key principles of Frequentist analysis

Key Metrics

Frequentist analysis uses the exact same conversion metrics as Bayesian tests, including inverse metrics like bounce rate.

Confidence Intervals

Confidence Intervals remain a core output. They are computed differently but maintain the same format and meaning, indicating the range within which the true effect likely lies.

Understanding the p-value

The p-value is the central concept in Frequentist analysis, replacing the Bayesian "Chance To Win (CTW)".

• Definition: The p-value is the probability that there would be no real difference between your variations (A & B) despite the observed difference in your experiment.

• Interpretation:

  • A lower p-value indicates a higher chance that a real difference exists between A & B.

  • A high p-value (close to 1) means there is no statistical evidence that a difference exists. This can occur if:

    • The observed difference is based on a very small number of visitors.

    • The observed difference is very small, even with a large number of visitors.

• Significance vs. Direction:

  • The p-value measures only the significance of a difference, not its direction (whether it's an uplift or a downlift).

  • The direction of the effect is determined by the Confidence Interval.

Unlike Bayesian CTW, the p-value does not use red/green color coding for uplift/loss.

Example: A p-value of suggests a high chance of a real difference, but this could be either an uplift or a loss.

Significance Thresholds

• Standard Significance: The industry standard significance threshold is a p-value of <0.05. This indicates a statistically significant difference has been found.

  • This is equivalent to a Bayesian CTW >95 for an uplift, or <5 for a loss.

• Emotion AI Segments: For Emotion AI segments, a p-value of <0.10 combined with a Confidence Interval showing a positive gain (i.e., the measured gain is >0) corresponds to the 90 Bayesian CTW.