Reporting Copilot
Last updated
Was this helpful?
Last updated
Was this helpful?
The Reporting Copilot is a module aiming at simplifying the analysis process of your campaign data to enable you to make the best decisions. It indicates whether the campaign has a winning variation, what is the winning variation and why.
The Reporting Copilot is available only if your campaign:
is an A/B Test or a Multipage test
has a transaction goal set as the primary goal
has a readiness that has been reached
has some traffic on the original version and more than 1000 unique visitors
The Reporting Copilot is statistically valuable since it is based on the confidence interval and chance to win both Average order value and Conversion rate of all variations of the reporting. However, you must still consider other goals’ results to make the good decision. For information on the reporting, refer to .
The Copilot is made of the three following elements:
The analysis status
The variation highlight
The analysis dialog box
This is the result of your campaign and is displayed under the main readiness element for the primary goal. You can see at a glance what is the result of your campaign.
According to the campaign result, the winning variation (or the one with the best trend) is highlighted in the goal spreadsheet with specific background colors and captions:
When there is a winning variation: the variation is highlighted in green with the mention “winning variation”
When no winning variation has been identified: the original version is highlighted with the mention “baseline is winning” in red to show that the campaign is “losing”.
When the campaign is neutral: the variation that has the best performance for the moment is highlighted with the mention “trending variation” in yellow.
The campaign analysis gets automatically displayed in a dialog box over the primary goal spreadsheet of the reporting. It can give seven different analysis results.
You can decide not to display the analysis dialog box again by checking the “Don’t show analysis anymore” box (The preference will apply only to this reporting). You can display the dialog box again by clicking on the Report Copilot status button (“Open analysis”).
The reporting can be categorized under three trends:
Positive trends
Neutral trends
Negative trends
Those trends are calculated based on the Average order value and the Transaction rate.
Positive trends indicate that we have a winning variation and happens in two situations.
This indicates that the variation has a great optimization potential and can be implemented on production without major risk, as either the average order value or the transaction rate has a good trend while the other has a balanced trend.
This indicates that both the average order value and the transaction rate have good trends.
Neutral trends indicate that there is no variation recommendation, as either both the average order value and the transaction rate have balanced trends, or they have opposite trends: one performing well while the other is not performing well.
Even when the campaigns have reached the readiness threshold, Copilot currently cannot designate a winning variation. If the campaign has been running for an extended period, further monitoring may be necessary. If the campaign is relatively new, we suggest allowing it to run for a few additional days to gather more data for a better evaluation. If the readiness has been reached more than 30 days ago, the reporting copilot won't be able to display the "needs more data" status, and you will see either a positive (winning variation), negative (baseline winning) or neutral (opposite trends) status.
While a specific variation may be highlighted as trending, we do not recommend implementing it due to the lack of a proper decision-making basis. Despite observing opposite trends in the variations, it's important to note that these fluctuations do not align with the objectives of this particular test.
Negative trends indicate that the original version (baseline) is performing better than the variations, that is, there are no winning variations identified.
In such cases, we recommend pausing the campaign and launching a new one with a different approach, typically by testing a new hypothesis.
When the readiness of your campaign (readiness of your primary goal) has not been reached yet, the Copilot can’t analyze your reporting data.
When switching the baseline, the Copilot’s logic remains the same: we keep on identifying the potential of the variations compared to the baseline. Keep in mind that the Copilot’s advice can change if the baseline changes.
In this case, you must wait for the three indicators of readiness to be reached. For more information on reporting readiness, refer to .