A/B Testing in Chatbots

Editor's note: This blog post was originally written on the Flow.ai website. Flow.ai was acquired by Khoros in 2021 to advance Khoros' conversational AI and machine learning (ML) capabilities and data science expertise. This blog post has been adapted to be on the Khoros blog.

One way of optimizing digital experiences is A/B testing. A/B testing allows you to create two (or more) variants for your users. This way of testing is used a lot when optimizing websites. You create variants, select a metric and measure which of these variants has the highest conversion.

This concept is very powerful and can also be used in conversational experiences. Our A/B testing feature for your chatbot allows you to easily create variants of specific parts of your conversational experience.

Metrics and performance

Choosing the metric that matters all depends on your case. When you want to increase your Marketing & Sales with the help of an AI Chatbot then your metric can be the number of generated leads.

The example below will show you how to create A/B tests for your chatbot.

How to add A/B Testing in 3 steps

1. Create a Flow

The first step is to create a flow. Head over to the Flows section and create a new flow by clicking the plus icon. Now drag and drop the first items onto your canvas.

2. Add A/B Test

You can now add an A/B test by dragging and dropping the Trigger - Condition onto your canvas. On the right pane, you can set it to A/B test from the drop-down menu. You will now have 2 branches or variants. Complete each of the branches and your A/B test is ready.

3. Try it out

Make sure to test your A/B test. You can use the Try it out button on top or you can test it with any of the channels that you’ve integrated.

Adding more variants

Are you planning to create an A/B/C test, just drag and drop an additional Condition onto your canvas and set it to A/B test. You will now have a third variant. There is no limit to the number of conditions you can place next to each other.

This also allows you, for example, by adding 10 conditions, where 9 are the same and 1 is slightly different, to perform an A/B test where only 10% instead of 50% get the second option. This trick is also useful for Soft Launching your bot.