Measuring Chatbot Effectiveness: 10 KPIs to Track in 2022

As early as 2019, Gartner predicted that 15% of global interactions will be driven by artificial intelligence, which includes chatbots. They predict that by 2022, that number will rise to 70% in some millennial-focused industries.

 

Since the benefits of chatbots are already proven and more and more businesses are adopting AI technologies every year, a new question arises. How do you measure chatbot effectiveness? What are the top chatbot success metrics? Let’s figure this out in the article.

10 KPIs to measure chatbot performance

All of us have probably communicated with a chatbot on our own, but how good was your experience? What about your company chatbot? How do you know if your bot is really helping customers? What are the main chatbot performance metrics?

 

User metrics

 

1. Total number of users. If these numbers seem lower than expected, there may be a technical problem with your bot that prevents users from using it.

 

2. Number of new users. If the number of new users is decreasing, you may need to reconfigure the chatbot to make it more useful.

 

3. Goal completion rate. This number measures how successful your chatbot is at accomplishing its goal. If the coefficient is low and/or falling, you should reconsider the chatbot’s scripts.

 

4. Number of sessions initiated. This number refers to the number of times someone has initiated a conversation with a chatbot. Compare this number to the number of sessions completed, and you can see how useful the chatbot is to users.

 

5. The average number of daily sessions. Comparing this metric to other metrics, such as average daily traffic to your website, gives you unique data on how needed the bot is.

 

Conversation metrics

 

1. Total number of conversations. This number enables businesses to measure customer satisfaction level. We know by experience that the number should also increase with the growth of website traffic. 

 

2. Agent Request Frequency. Monitor this data to determine whether the chatbot is succeeding in reducing the number of interactions with support agents.

 

3. Average Conversation Time. If this indicator is low or decreasing, then users probably consider chatbots ineffective.

 

4. Bounce rate. Sometimes the chatbot can’t understand a user’s request or answer a question. If this rate is high, perhaps you need to review your chatbot scripts or NLP parameters. For example, AI bots may have a non-response rate of 10-20%.

 

5. Interaction rate. This rate is the average number of messages exchanged during each conversation between your chatbot and a user. This is a key metric for understanding overall engagement. 

The key performance indicators listed above are the main ones for tracking the success of any chatbot. In this article, we want to look not only at the basics but also at the more advanced chatbot metrics for each type of bot. Let’s go!

Metrics for a lead generation chatbot

According to a study by Harvard Business Review and InsideSales.com, if a customer doesn’t get a response within the first five minutes, the likelihood of a lead being qualified as relevant drops by 400%.

 

The effectiveness of a lead generation bot

However, with a chatbot, not a single lead will be missed. To assess the effectiveness of a chatbot as a lead generation tool, you need to compare the cost per lead and the conversion rate before and after running the bot. For fair results, compare the number of leads received from chatbots and registration forms.

X = (traffic cost + chatbot platform subscription cost) / number of leads

X — cost per lead after chatbot launch

% — conversion of traffic to lead after chatbot launch

% = (Leads / Traffic) * 100

 

If X is less than before the launch of the chatbot, it should be scaled to other channels such as Instagram DMs, WhatsApp, WeChat, etc.

 

If X is larger, consider increasing traffic and equalizing the cost of subscribing to the chatbot platform. If additional traffic isn’t bringing in more money, it’s likely that the chatbot isn’t an effective tool for generating leads in a particular business case.

 

The profitability of a lead generation chatbot

To assess whether it is profitable or not, you need to compare CAC (customer acquisition cost) with LTV (Lifetime Value, the profit a customer will make over the time he or she uses a company’s products or services).

 

However, the LTV and CAC are also useful in other cases because comparing them shows the return on investment and the effectiveness of the channel.

n = (cost of subscribing to the chatbot platform + cost of traffic) / number of leads for the reporting period) / (% of conversion from lead to payment)

n – CAC

LTV = average number of payments per life cycle * average bill

If n is less than LTV, that’s great, keep using the chatbot to generate leads.

 

If n is greater than LTV, you should increase traffic to recoup the cost of subscribing to the chatbot platform. If chatbot economics don’t match the additional traffic, it may be worth finding alternative channels for additional lead generation.

 

Metrics for a chatbot to cut operating costs

Measuring time saved by agents

Any company’s hotline receives countless routine inquiries every day. But with an FAQ bot answering them, agents will have more time to deal with more complex questions. For example, The Cardiff insurance company’s bot automates 56% of incoming calls in the first line of support.

 

To calculate how much time a bot saves for agents, you need to use the following formula.

Savings = (number of correctly answered questions * agent’s average response time * agent’s average rate per hour) — (bot creation cost + service subscription cost)

If the value is positive, the chatbot can be scaled to other channels and expand the questionnaire.

 

If the value is negative, you should increase the number of questions that the chatbot answers and check the correctness of the answers.

 

Measuring savings on hiring staff

Before hiring a new customer support person, consider automating the process. A metrology service call centre that provides residents with water metre calibration and replacement services decided to take the latter route.

 

That metrology service call agents handled 35-50 inquiries a day, while the bot handled more than 200 inquiries. The average agent’s salary is £2,300 while maintaining the bot costs about £200 a month. The bot, unlike employees, can also answer questions 24/7, and it never asks for a day off or goes on vacation.

Savings can be calculated using the same formula as for time

If the cost turns out to be greater than the cost of hiring, it makes sense to use a chatbot. 

 

If the savings turn out to be less than the cost of hiring, you should also remember about staff turnover. Note, that the bot can pay off in 12 months.

So how to measure chatbot effectiveness

1. Set a goal in numbers and measure performance before you start the chatbot, so you have data to compare.

2. The money saved by using a chatbot is worth spending on improving the product or user experience.

3. Measure UX. In Google Analytics you can see how many users come back to use the bot.

4. Provide the chatbot with traffic, such as through emails and Ads. This is the only way to measure effectiveness and make your business profitable.