Define your first experiment

Discover FlowSync example at the end of the article.

Defining your first experiment is a critical step to ensure you generate meaningful insights and business value from your Feature Experimentation and Rollout (FE&R) program. This process involves two main actions: identifying what to test, and setting clear KPIs and success metrics.

1. Identify a Feature or Business Logic to Test

Start by selecting a feature, user journey, or business logic that you want to improve or validate. Good candidates for your first experiment include:

• New features you want to roll out safely (e.g., a new payment method, onboarding flow, or recommendation engine).

• Business logic changes such as pricing thresholds, delivery options, or eligibility rules.

• User experience improvements like button text, layout changes, or navigation updates.

Best Practice: Choose a test that is simple, high-impact, and easy to measure. For example, changing a call-to-action text or adjusting a delivery threshold. This helps your team build confidence in the experimentation process and ensures quick, actionable results.

Tip for Product Managers:

Use your product roadmap, customer feedback, or analytics data to prioritize which features or flows to test first. Collaborate with your developer to ensure the feature or logic is accessible for flagging and experimentation.

Tip for Developers:

Collaborate with product managers to understand the business context and technical feasibility of the proposed test. Align with developers to determine how to break down the feature or logic into appropriate flags (e.g., for a button color change, create a "button_color" flag), and ensure both teams are synchronized on how to structure the test for flagging and experimentation..

2. Set Clear KPIs and Success Metrics

Defining what success looks like is essential for every experiment. KPIs (Key Performance Indicators) and success metrics allow you to objectively measure the impact of your changes.

How to Set KPIs:

• Align with business goals: Choose metrics that reflect the desired business outcome (e.g., conversion rate, average order value, engagement, retention).

• Be specific: Define exactly what you will measure (e.g., “Increase sign-up completion rate by 5%”).

• Select primary and secondary metrics: Primary metrics are your main goal; secondary metrics help you monitor for side effects (e.g., bounce rate, time on page).

Examples:

Experiment Example	Primary KPI	Secondary KPI
Change CTA button text	Click-through rate	Conversion rate
Test new delivery threshold	Average order value (AOV)	Conversion rate
Roll out new onboarding flow	Onboarding completion rate	Drop-off rate

Best Practice: Document your hypothesis in a clear, structured format. For example: “If we [make this change], then [target users] will [do this action], resulting in [measurable impact].”

Example Hypothesis: “If we increase the free shipping threshold from $50 to $75, then new users will increase their average order value, resulting in higher revenue per session.”

Next Steps

Once you have defined your experiment and KPIs:

• Document your hypothesis and success criteria.

• Share your plan with stakeholders for alignment.

• Proceed to the next step: integrating the SDK and setting up your experiment in the FE&R platform.

Ren fictionnal example from FlowSync

When we started with FE&R at FlowSync, the first thing I noticed was an opportunity to challenge and improve our onboarding workflow. I wanted a test that was simple enough to build confidence, yet meaningful enough to deliver real value.

I sat down with Jess, our dev, and we looked at our roadmap and usage analytics. One thing stood out: a surprising drop-off in our onboarding flow.

We didn’t want to rebuild the entire journey right away, so we chose a small, high-impact test: updating one step in the flow to make it clearer for new users.

From there, I defined our KPIs: the primary one was onboarding completion rate, and the secondary was drop-off on step two.

Jess checked that this part of the flow was flaggable and easy to instrument, and once we aligned, I wrote the hypothesis in the first JIRA ticket of what would become a long series of experiments!

Defining this first experiment gave us a shared language. Jess understood exactly what we were trying to achieve, and I had measurable criteria to validate whether our idea made sense. It made our next steps (integrating the SDK and setting up the experiment) fast and straightforward.