I recently spent some time watching the Chatbase team's video on improving chatbots using Chatbase.
I will summarize the material in this post.
What is UMM?
UMM is an acronym which stands for:
U - Unsupported requests
M - Misunderstood requests
M - Missed requests
Let us take a look at some examples to show what each of these types of requests are.
For the sake of explanation, let us suppose you can assign a color to each of your intents.
So you have red, blue and green intents, and the black color represents the Fallback intent.
Suppose, we get a new user request and it is an orange. We don't (yet) have any way to handle it. So it gets mapped to the Fallback Intent.
This is an example of an unsupported request.
Suppose after seeing this, we add an orange intent into our list of intents.
We now get user phrase 5 which should be mapped to the newly added orange intent. Unfortunately, it gets mapped to the Red Intent instead. This is an example of a misunderstood request.
We get another request User Phrase 6. This should have been mapped to the Blue Intent, but instead it gets mapped to the Fallback intent. In this case, we have a missed request. It should have been mapped to the blue box.
How Chatbase can help
So we have now seen the different categories of chatbot errors.
How can Chatbase help us fix these errors?
Chatbase has a concept called "not-handled". If a user request gets mapped to the Fallback Intent, you set this flag and notify Chatbase using its API.
Unsupported and Missed Requests
You can take a look at the not handled requests and you will be able to identify your missed requests and your unsupported requests.
It is a little more challenging to identify misunderstood requests. In this case, the phrase which should have been mapped to Intent A gets mapped to Intent B instead. This means you will not be setting the not-handled flag when you send details about this request to Chatbase.
The Chatbase team suggests looking for intents with high exit rates as a sign that you have a lot of misunderstood requests. Personally, while I think this might be a good starting point to identify misunderstood requests, you need to dig a little deeper if you wish to identify most of the misunderstood requests.
Interestingly, the color analogy can be used in quite a few places as you try to reason about your chatbot. I will try and create some learning material on this topic in the future.