Frequentist Analysis mode
While our primary framework is Bayesian, some users prefer Frequentist analysis. This option uses the same core data and offers a different statistical perspective.
Last updated
Was this helpful?
While our primary framework is Bayesian, some users prefer Frequentist analysis. This option uses the same core data and offers a different statistical perspective.
Last updated
Was this helpful?
. Yet, we've created a feature allowing users to utilize it if necessary.
Please note that , and opportunities views are not available with frequentist mode.
Ask your CSM to activate the feature to your account
Switch from Bayesian stat engine to Frequentist stat engine
Access your campaign.
In the reporting tab of your campaign, click on the statistics engine button.
Select the Frequentist engine.
If you want to apply to all reports of the account, check "Apply to all reportings". If you don't, this statistical analysis, will only apply on the campaign your are currently browsing).
Click on "Apply".
Conversion rate stats changes :
“Confidence interval” switches to “Confidence interval” (Frequentist one)
“Chance to win” switches to “p-value”
AOV changes (transaction goals only):
“Chance to win” switches to “p-value” (Frequentist one)
In AOV gain p-value and in Conversion gain p-value you will face :
Blue artifact and stats’ font if p-value =< 0.05. Highlighting that the tests results provide potential learnings.
Orange artifact and stats’ font when p-value >= 0.95. Highlighting that the tests results provide no learnings.
Frequentist analysis uses the exact same conversion metrics as Bayesian tests, including inverse metrics like bounce rate.
Confidence Intervals remain a core output. They are computed differently but maintain the same format and meaning, indicating the range within which the true effect likely lies.
The p-value is the central concept in Frequentist analysis, replacing the Bayesian "Chance To Win (CTW)".
Definition: The p-value is the probability that there would be no real difference between your variations (A & B) despite the observed difference in your experiment.
Interpretation:
A lower p-value indicates a higher chance that a real difference exists between A & B.
A high p-value (close to 1) means there is no statistical evidence that a difference exists. This can occur if:
The observed difference is based on a very small number of visitors.
The observed difference is very small, even with a large number of visitors.
Significance vs. Direction:
The p-value measures only the significance of a difference, not its direction (whether it's an uplift or a downlift).
The direction of the effect is determined by the Confidence Interval.
Unlike Bayesian CTW, the p-value does not use red/green color coding for uplift/loss.
Example: A p-value of suggests a high chance of a real difference, but this could be either an uplift or a loss.
Standard Significance: The industry standard significance threshold is a p-value of <0.05. This indicates a statistically significant difference has been found.
This is equivalent to a Bayesian CTW >95 for an uplift, or <5 for a loss.
Emotion AI Segments: For Emotion AI segments, a p-value of <0.10 combined with a Confidence Interval showing a positive gain (i.e., the measured gain is >0) corresponds to the 90 Bayesian CTW.
