How to A/B test a recommendation strategy - For API clients

This guide is dedicated to people who are implementing Recommendations manually via API and who want to be able to run remotely some A/B tests or personalizations.

Requirements

• Recommendations & Merchandising access

• Feature Experimentation & Rollout access

Differences with basic R&M API integration:

• Increased latency versus basic R&M integration (+30 to 100ms)

Overviews

• Usage overview

• 1st time technical implementation overview

Initial configuration

1

Create a recommendation strategy

Create the strategy of your choice and save it.

2

Create a FE&R flag

1. Go to Feature Experimentation > Flags

2. Click on Create flag

   1. Name: Reco_{block_name}

   2. Type: String

3. Click on Save

3

Implement your strategy via API (Need a developer)

1. Select the FE&R SDK that best fits you and configure it

2. Fetch FE&R flags and retrieve the flag Reco_{block_name} which should contains the strategy id that you want to push on your specific recommendation block. Make sure that visitor id is of type a string

3. Call the Recommendations API by passing the value of the flag Reco_{block_name} as the strategy id. Don't forget to manually provide a fallback hardcoded strategy ID

https://uc-info.eu.abtasty.com/v1/reco/${SITE_ID}/recos/${strategy_id}?variables=${query}&fields=${fields}

1. Use the results of the Recommendations API to display your recommendation block

2. Your code is ready to display your current strategy or future A/B tests

4

Add FE&R tracking (Need a developer)

Implement the following trackers:

• click_on_product: via an Event tracker (documentation)

• add_product_to_card: via an Event tracker (documentation)

• purchase : via a Transaction hit tracker (documentation)

Tracking documentation: Tracking data with SDKs or Tracking with custom API calls

5

Create a FE&R "Feature toggle" campaign

This step is optional. If you don't do it, your default strategy will be the hardcoded strategy. It means that you will have to manually edit your code to deploy the winner of an A/B test.

1. Go to Feature Experimentation > Campaigns

2. Click on "Create campaign" > "Feature toggle"

   1. Name: {block_name} deployment

   2. Type: Feature toggle

   3. Folder: Pick the folder of your choice

3. Define your scenario

   1. Reco_{block_name} = Your recommendation strategy ID

4. Define your goals, we recommend adding the 3 trackers

5. Keep default parameters for targeting & delivery strategy

6. Take your campaign live

Running an A/B test

1

Create a variant recommendation strategy

Create the strategy of your choice. You can duplicate your original strategy if you want not to start from scratch.

2

Create a FE&R "A/B test" campaign

• Main information:

  • Name: {block_name} A/B test

  • Folder: folder of your choice for the campaign

• Variations:

  • Original: Set flag Reco_{block_name} to original strategy id

  • Variation: Set flag Reco_{block_name} to your variant strategy id

• Goals: Pick the goals you want to track

• Targeting: All users

• Traffic allocation: 50/50

3

Turn "A/B test" campaign live & pause "feature toggle" one

• Set campaign {block_name} A/B test to "Live" status

• Set campaign {block_name} deployment to "Pause" status

4

Track your results

Open the report for your campaign {block_name} deployment and check your results.

5

Once your decision is taken, turn "feature toggle" campaign live and stop "A/B test" one

• Set campaign {block_name} deployment to "Live" status

• Set campaign {block_name} A/B test to "Pause" status

6

Update your "Feature toggle" campaign with the ID of your winning strategy

If you want to deploy your winning strategy, edit your deployment campaign {block_name} deployment by editing its scenario and setting Reco_{block_name} to your new winning strategy ID.