Dynamic allocation

Dynamic traffic allocation is based on a statistical research topic called “multi-armed bandit”. The goal of this algorithm is to limit the loss due to the lowest-performing variations.

However, when the campaign has just been launched, the performance of the variations is not known yet. When data collection begins, the performance of variations is known progressively, but these measures are not fully significant.

At this point, it would be an error to send all traffic to the apparent winner. However, some formulas can help to compute the right amount of traffic to send to each variation with the following goal: sending more traffic to the variation that could win, but not too much, otherwise, there would not be enough traffic to ensure the measurement of the supposed losers.

The solution is to periodically change the allocation regarding previous data, following a statistical formula that ensures the goal mentioned above.

Guide_Defining_Traffic_Allocation_09.jpg

Dynamic allocation is a technique that can be applied when a classic A/B Test is not possible because of lack of time, or traffic. This technique is only applicable in a given context (typically with stable KPI). If you have enough traffic, and are not sure that the KPI to optimize is stable then you should prefer the classic A/B test.

Configuration

Before switching to dynamic allocation, you need to define at least a primary goal in the Goal step of the test creation flow. The Dynamic allocation algorithm is exclusively based on this primary goal. This means that it detects the best-performing variation based on the increase in conversions of the primary goal. For further information, please refer to the Metrics guide & definitions.

The primary goal can be any type of goal except Browsing metrics (Revisit rate, Pages per session, and Bounce rate). To activate dynamic allocation, click the Change/switch to Dynamic allocation button at the top of the page.

A confirmation message is displayed:

By confirming, the traffic allocated to each variation will be determined based on the predictive algorithm and on the primary goal, you configured for your test. At this point, you will not be able to manually adjust the percentage of traffic allocated to each variation. However, you are still able to define the percentage of traffic you want to allocate to your test and the percentage of traffic you want to leave untracked.

If you did not select a primary goal, you will not be able to switch/change to dynamic allocation. In this case, you will be asked to define at least one primary goal and you will be redirected toward the goal step.

Once the test has been launched, you can access the reporting anytime, as you would for any campaign with static allocation.

From the test dashboard, tests using dynamic allocation are easily identifiable thanks to the icon.

Once your test has been launched with dynamic allocation, you will not be able to perform the following actions:

• Change the primary goal

• Add, remove or duplicate a variation within the test

• Switch back to static allocation

If you want to perform any of these actions, you need to pause the test and duplicate it.

Use Cases

Dynamic Allocation can be very useful in the following cases:

• When you want to optimize micro conversions that are expected to occur within a short period after the user has been exposed to a variation. For instance, e-commerce, prefers the Add to cart goal rather than the transaction, as the primary goal.

• When you have very little time to run a test (ex: testing the headline of a media article, testing the message of a countdown banner).

• When you have really low traffic on the page you want to test, but still want to do some optimizations. Dynamic allocation carries out optimization even before reaching enough data to be fully significant. This would not be very interesting because the traffic is low. But it may become interesting if carried out on a high number of pages. For instance, if you have low traffic on each product page, you cannot carry out a classic A/B test. But it may be interesting to implement multiple dynamic allocation tests, one for each product page. The gain will be low for each page, but it can be more significant when multiplied by the number of pages.

• When you have a lot of variations to test (more than six), dynamic allocation enables you to quickly identify the lowest-performing variations to run the test on the most relevant ones.

We recommend not using dynamic allocation in case you want to optimize a website/application where the visitors will be different over time. Since allocation changes over time, the primary goal should be stable during the experiment. Example: a meal-ordering website, or a taxi company, customers and behaviors will not be the same at noon and 3am