Measuring Chatbot Effectiveness: 10 KPIs to Track in 2023

August 18, 2023

5 min read

Diana Kisling

 

As early as 2019, Gartner predicted that 15% of global interactions will be driven by artificial intelligence, which includes chatbots. They expected that by 2023, that number would have risen to 70% in some millennial-focused industries.

 

Since the benefits of chatbots are already proven and more and more businesses are adopting AI technologies every year, a new question arises. How do you measure chatbot effectiveness? What are the top chatbot KPIs? Let’s figure this out in the article.

 

10 KPIs to measure chatbot performance

All of us have probably communicated with a chatbot on our own, but how good was your experience? What about your company chatbot? How do you know if your bot is really helping customers? What are the main chatbot performance metrics?

 

User metrics

1. Total number of users. If these numbers seem lower than expected, there may be a technical problem with your bot that prevents users from using it.

 

2. Number of new users. If the number of new users is decreasing, you may need to reconfigure the chatbot to make it more useful.

 

3. Goal completion rate. This number measures how successful your chatbot is at accomplishing its goal. If the coefficient is low and/or falling, you should reconsider the chatbot’s scripts.

 

4. Number of sessions initiated. This number refers to the number of times someone has initiated a conversation with a chatbot. Compare this number to the number of sessions completed, and you can see how useful the chatbot is to users.

 

5. The average number of daily sessions. Comparing this metric to other metrics, such as average daily traffic to your website, gives you unique data on how needed the bot is.

 

Conversation metrics

1. Total number of conversations. This number enables businesses to measure customer satisfaction level. We know by experience that the number should also increase with the growth of website traffic. 

 

2. Agent Request Frequency. Monitor this data to determine whether the chatbot is succeeding in reducing the number of interactions with support agents.

 

3. Average Conversation Time. If this indicator is low or decreasing, then users probably consider chatbots ineffective.

 

4. Bounce rate. Sometimes the chatbot can’t understand a user’s request or answer a question. If this rate is high, perhaps you need to review your chatbot scripts or NLP parameters. For example, AI bots may have a non-response rate of 10-20%.

 

5. Interaction rate. This rate is the average number of messages exchanged during each conversation between your chatbot and a user. This is a key metric for understanding overall engagement.

 

How to measure the effectiveness of chatbots in 2022

 

The key performance indicators listed above are the main ones for tracking the success of any chatbot. In this article, we want to look not only at the basics but also at the more advanced chatbot metrics for each type of bot. Let’s go!

 

KPIs for a lead generation chatbot

According to a study by Harvard Business Review and InsideSales.com, if a customer doesn’t get a response within the first five minutes, the likelihood of a lead being qualified as relevant drops by 400%.

 

The effectiveness of a lead generation bot

However, with a chatbot, not a single lead will be missed. To assess the effectiveness of a chatbot as a lead generation tool, you need to compare the cost per lead and the conversion rate before and after running the bot. For fair results, compare the number of leads received from chatbots and registration forms.

X = (traffic cost + chatbot platform subscription cost) / number of leads

X — cost per lead after chatbot launch

% — conversion of traffic to lead after chatbot launch

% = (Leads / Traffic) * 100

If X is less than before the launch of the chatbot, it should be scaled to other channels such as Instagram DMs, WhatsApp, WeChat, etc.

 

If X is larger, consider increasing traffic and equalising the cost of subscribing to the chatbot platform. If additional traffic isn’t bringing in more money, it’s likely that the chatbot isn’t an effective tool for generating leads in a particular business case.

 

 

The profitability of a lead generation chatbot

To assess whether it is profitable or not, you need to compare CAC (customer acquisition cost) with LTV (Lifetime Value, the profit a customer will make over the time he or she uses a company’s products or services).

 

However, the LTV and CAC are also useful in other cases because comparing them shows the return on investment and the effectiveness of the channel.

n = (cost of subscribing to the chatbot platform + cost of traffic) / number of leads for the reporting period) / (% of conversion from lead to payment)

n – CAC

LTV = average number of payments per life cycle * average bill

If n is less than LTV, that’s great, keep using the chatbot to generate leads.

 

If n is greater than LTV, you should increase traffic to recoup the cost of subscribing to the chatbot platform. If chatbot economics don’t match the additional traffic, it may be worth finding alternative channels for additional lead generation.

 

KPIs for a chatbot to cut operating costs

Measuring time saved by agents

Any company’s hotline receives countless routine inquiries every day. But with an FAQ bot answering them, agents will have more time to deal with more complex questions. For example, The Cardiff insurance company’s bot automates 56% of incoming calls in the first line of support.

 

To calculate how much time a bot saves for agents, you need to use the following formula.

Savings = (number of correctly answered questions * agent’s average response time * agent’s average rate per hour) — (bot creation cost + service subscription cost)

If the value is positive, the chatbot can be scaled to other channels and expand the questionnaire.

 

If the value is negative, you should increase the number of questions that the chatbot answers and check the correctness of the answers.

 

Measuring savings on hiring staff

Before hiring a new customer support person, consider automating the process. A metrology service call centre that provides residents with water metre calibration and replacement services decided to take the latter route.

 

That metrology service call agents handled 35-50 inquiries a day, while the bot handled more than 200 inquiries. The average agent’s salary is £2,300 while maintaining the bot costs about £200 a month. The bot, unlike employees, can also answer questions 24/7, and it never asks for a day off or goes on vacation.

Savings can be calculated using the same formula as for time

If the cost turns out to be greater than the cost of hiring, it makes sense to use a chatbot. 

If the savings turn out to be less than the cost of hiring, you should also remember about staff turnover. Note, that the bot can pay off in 12 months.

 

KPIs for a customer support chatbot

It is crucial to establish KPIs to evaluate the performance of chatbots used for customer support. The following indicators can help identify areas for improvement and optimise the overall performance of the bot.

 

Among the most important KPIs for a customer support chatbot is the rate of successfully resolved issues without human intervention. A high issue resolution rate indicates that the chatbot is effectively functioning, and clients are satisfied with the level of support they receive.

 

Another significant indicator is the chatbot’s response time in processing customer requests. The bot must respond within a few seconds to avoid customers feeling neglected or disappointed.

 

Aside from issue resolution and response time, customer satisfaction is also a critical KPI. This metric can be measured through post-chat surveys or by analysing customer feedback and helps evaluate the overall effectiveness of the bot’s performance.

 

Key considerations

Measuring chatbot effectiveness is critical to evaluating their performance and ensuring they deliver value to both businesses and users. 

 

While there is no one-size-fits-all approach to assessing chatbot success, several key considerations must be taken into account to make informed judgments and improvements.

 

1. Set a goal in numbers and measure performance before you start the chatbot so you have data to compare.

 

2. The money saved by using a chatbot is worth spending on improving the product or user experience.

 

3. Measure UX. In Google Analytics, you can see how many users come back to use the bot.

 

4. Provide the chatbot with traffic, such as through emails and Ads. This is the only way to measure effectiveness and make your business profitable.

 

5. Don’t overlook ethical and data security considerations. Ensuring that the chatbot operates in an ethical and responsible manner, respects user privacy, and avoids bias in responses is fundamental to its long-term success and user trust.

 

Regular monitoring and continuous improvement are essential to ensure that the chatbot remains current with evolving customer needs and expectations. As technology and conversational AI models advance, it’s imperative that the chatbot’s capabilities and performance evolve accordingly.

 

Measuring chatbot effectiveness is a multifaceted process encompassing various aspects, from goal alignment and user experience to data analysis and ethical considerations. 

 

By taking these considerations into account and adopting a holistic approach to evaluation, organisations can develop chatbots that meet their objectives and build lasting relationships with their users in an ever-evolving digital landscape.

Get a demo

Please tell us about yourself and we’ll get back as soon as we can.

Name

Business email

Company name

Work phone

Message

Contact Us

Please, fill in the form and we will contact you shortly.

Name

Business email

Company name

Message

Thank you for reaching out!

We appreciate you contacting Tovie AI and will get back to you as soon as we can.

Obrigado por estender a mão!

Agradecemos o seu contato e entraremos em contato o mais rápido possível.

Thank you for reaching out!

We appreciate you contacting Tovie AI and will get back to you as soon as we can.