Analysis Copilot

Analysis Copilot is an AI-powered assistant designed to help you analyze campaign data within AB Tasty reports.

This feature is currently in its Beta testing phase. During this period, users may experience changes and updates as improvements are made based on feedback. We encourage users to provide input to help us refine and enhance the feature before its official release.

Instead of manually sifting through data tables and charts, you can simply type your questions, and the AI will process the underlying metrics, statistical significance, and objective performance to deliver concise and relevant answers.

Ask Copilot leverages CRO best practices and statistical data to provide clear, data-backed answers and recommendations.

The assistant can answer with text, table, and/or chart based on your needs.

Use case examples:

• Explain the winning variation

• Challenge my hypothesis

• Give me CRO best practices based on my campaign results

Tips for Asking Effective Questions :

To get the most accurate and helpful responses from the AI, consider the following:

• Wait for readiness : if you are willing to make a decision, wait for the readiness to be ok at least on primary goals.

• Be Specific: The more precise your question, the better the AI can understand your intent.

  • Good: "Which variation performed best for the 'Conversion Rate' objective?"

  • Avoid: "What about the conversion rate?"

• Reference Objectives and Metrics: Clearly state the objectives and metrics you are interested in.

  • "What is the statistical significance for the 'Click-Through Rate' of Variation B versus Control?"

  • "Show me the difference in average revenue per user between all variations."

• Ask for Comparisons: The AI is excellent at comparing performance between variations.

  • "Compare the bounce rate of Variation A and Variation B."

  • "Which variation had the highest increase in sign-ups compared to the control?"

• Inquire About Statistical Significance:

  • "Is the difference in [Metric] between [Variation X] and [Variation Y] statistically significant?"

  • "What is the p-value for the 'Add to Cart' objective?"

• Ask for Summaries:

  • "Summarize the overall performance of this A/B test."

  • "What are the key takeaways from this report?"

• Identify Top/Worst Performers:

  • "Which variation had the highest [Metric]?"

  • "Which objective performed the worst for Variation C?"

• Troubleshooting & Clarification:

  • "Can you explain the 'Confidence Interval' for the 'Purchase Rate'?"

Use the Ask Copilot:

1

Access your report.

2

On the top right corner, click on Ask Copilot button.

3

Select a question or enter your query.

4

Read the copilot's answer and get a better understanding of your report.

Limitations

While powerful, the Analysis Copilot has some limitations:

• It does not work in Frequentist mode

• Relies on Available Data: The AI can only analyze the data presented in your report. It cannot infer information not present in the underlying metrics and statistics.

• Statistical Interpretation: The AI provides statistical interpretations, but it's crucial for you, to apply domain expertise and strategic context to those interpretations.

• Complex Scenarios: For highly complex or multi-variate statistical modeling, you may still need to consult with a data analyst.

• No Predictive Capabilities (currently): The AI focuses on analyzing past performance; it does not predict future outcomes or design new experiments (unless specifically integrated in a future release).