Sequential Testing Alerts

Sequential Testing Alerts help you detect underperforming variations early in an A/B test campaign.

The feature continuously monitors your campaign’s primary goal and uses the selected sensitivity threshold to identify variations that show a decrease in conversion rate compared to the baseline.

When an underperforming variation is detected, Sequential Testing Alerts can:

• send an email alert

• automatically interrupt the campaign and send an email notification

This is especially useful for campaigns where a variation could negatively affect important business logic, such as pricing, eligibility, routing, or recommendation rules.

How it works

Sequential Testing Alerts rely on two settings:

• the action to trigger when an underperforming variation is detected

• the sensitivity threshold used to trigger the alert

The algorithm evaluates the campaign based on the selected primary goal and the chosen sensitivity level.

Configure Sequential Testing Alerts

To enable Sequential Testing Alerts: check the action you want to trigger when the algorithm detects an underperforming variation. Then, select the sensitivity threshold.

Select the action to trigger

You can choose one of the following actions:

• Receive an email alert without automatic campaign interruption

• Automatically interrupt the campaign and be notified by email

Use the first option when you want to review the situation manually before taking action.

Use the second option when you want the campaign to stop automatically as soon as a losing variation is detected.

Select the sensitivity threshold

Choose how quickly Sequential Testing Alerts should react to negative signals:

• High Alerts are more likely to be triggered, but losing variations are identified sooner. This is the safest option.

• Balanced Alerts may still be triggered, while losing variations are identified within a reasonable amount of time.

• Low Alerts are less frequent, but losing variations may take longer to be identified.

Important details

• Sequential Testing Alerts are only available when a primary goal is configured.

• Detection is based on the campaign’s primary goal and the selected sensitivity threshold.

• When Sequential Testing Alerts send email notifications, all users in the account are notified.