SRM calculator
Still want to calculate it manually?
SRM Calculator
In order to help A/B testers with their experiments, AB TASTY provides, in addition to its S-SRM service, an online service for SRM analysis. This service can help identify potential SRM issues so that they can be corrected before the experiment is restarted.
Please be aware that this SRM calculator does not account for peeking (also known as p-hacking—learn more here). Peeking can lead to false positives and may produce results that differ from those of S-SRM. Additionally, the underlying calculation models differ: the SRM calculator is based on bayesian approach while S-SRM alerts rely on Sequential Testing algorithms.
Overall, SRM can significantly affect the results of an A/B test experiment, and it's important to take steps to identify potential SRM issues and take corrective action before starting or restarting the experiment.
How does it work?
Input : Type the allocation ratios expected (numbers that sum to 1) and the observed allocation (the one that happened, with integers). If your test has more than two variations, hit the “+” button to create additional variation slots.
Output : After hitting ‘enter’, you will be told whether there is a SRM or not. In case of SRM, a confidence interval of the deviation will be shown as a relative gain. A vertical dotted line shows where the normal value should be (corresponding to 0 deviation).
Last updated
Was this helpful?