Skip links

Measuring Bot Performance Accurately

How well is my bot performing?

This is something all of our partners want to know, and there are a number of ways that this can be evaluated. At Ivy we are always working to be more transparent and to provide data that is accurate, relevant, meaningful, and actionable.

That is why you will be seeing some changes to your dashboard in the upcoming days.

Until recently the bot dashboard showed a metric of user star ratings. But with only 3% of users opting to give a rating, this metric did not give an accurate measurement of the bot’s performance. In fact, out of that small population, the bot ratings were highly polarized with really great -or- really bad ratings. There were almost no “average” ratings. Because of this, that graph has been removed from the dashboard.

Instead, what you’ll see is a new, improved version of the Bot Accuracy statistics. Rather than just breaking things down by answered and unanswered questions, we will be showing whether the bot responded with high, medium, or low confidence. This means that you will be able to get a better sense of whether users actually got the answer they needed, a partial answer, or if the bot was stumped. By showing how this data changes over time, you can feel confident that your bot is constantly improving and answering an increasing number of inquiries with greater accuracy.