Recently, I wrote an article which talks about measuring the accuracy of your Dialogflow bot.
Later, I was asking a client about their bot's accuracy, and they said
"we have been getting 60% of the conversations correct (single turn), 20% wrong and 20% out of domain questions"
This made me realize that there is an important point I left out from my article on measuring your bot's accuracy.
There are certain kinds of questions you should throw out immediately and disregard from all future analysis. (And if possible, create some kind of Deleted Posts table and archive them so they don't accidentally mess up your future analyses).
Also, I thought I will have a little bit of fun and give these questions "personalities".
Note: some of these may not apply to your specific bot, or even make sense in its context. But hopefully you can take these ideas to create a list of types of messages you can ignore when calculating the accuracy of your bot.
1 The Troll
Everyone who has ever built any kind of serious bot knows The Troll.
They will come and swear at your bot, just to see what it does.
They will write personal insults.
They will suggest that they could have done a much better job at building out their bot.
My suggestion is to not spend a minute thinking about "how to handle" these kinds of questions. If someone screenshots your bot doing something very dumb as a response to what is basically a "troll comment", simply ignore those too.
These days, people have become quite savvy about spotting trolls, so don't be concerned about this.
2 The RTFMer
This is a very popular meme for those who simply refuse to "Read the F----ing Manual" (RTFM).
For example, sometime back, right at the bottom of my course list page, I created a bot. (I removed it because I didn't have enough time to maintain it).
I named it CourseBot.
Above the bot I said "Do you have any questions about my courses? You can ask my CourseBot"
Inside the description (it was a 1-click web demo integration), it said "CourseBot will help you with questions you have about my courses"
And someone came and asked the bot:
So how much do you charge for your services?
My first thought was "Why would you do that?" And then I realized, "Of course, it is so I get some new material to write about on my blog!" 😉
3 The Polymath
This is the person who started collecting a set of favorite questions to ask your bot. They are a Polymath, you see.
So they will ask:
"Tell me a joke"
"I want a pizza"
"What is the highest score of Rohit Sharma?" (cricket related question)
"Who made you?"
"Are you bot"
"Talk to a human"
And all in the same conversation session, of course.
4 The anonymous celebrity
This is the person who will ask you to get back to them with some question which is clearly aimed at the bot's creator and not to the bot itself, and don't leave any contact information.
Here is an example I received a couple of days back:
My name is (redacted) and I live in (redacted). I am interested in developing an educational chatbot app to teach (redacted) language. The idea is to provide users with the functionality of speaking in the target language. Your comments about the feasibility of this is more than welcome.
I don't have any contact information from you. How do you possibly think I can send you a response?
5 The ASAP on WhatsApp-er
Actually 4 is a little better (because I don't have any problem ignoring them) than 5.
Worst is the person who gives their phone number to the chatbot and hints that they are building this "amazing bot" that they "cannot really say much about" as you know.
"This is my number - 1234567890. Please get in touch with me via WhatsApp"
And in case I might hesitate, they add
Well, I wasn't sure at first, but now that you have uttered the magic phrase ASAP, I think I better drop everything I planned to do and call you right away!
6 The acronym specialist
This is the person who will only talk in acronyms to your bot.
"I want to build a VH type DF bot which will be deployed on FBM and hopefully WA later. Eventually I want to move on to using the same bot for both WP and AoG. Can you help?"
Maybe you are wondering why you shouldn't build your bot to actually handle this. After all, most of these will be entities in your Dialogflow bot, and you can add the acronym as one of the entity variations.
Usually the issue is the shorter these acronyms, the less likely Dialogflow can identify them properly. And that is especially true if people don't capitalize the acronym.
Second, and I cannot back this up with hard data of any kind - people who write like this very rarely make good users, customers or clients. (But you can test this for yourself. Browse through your email archive for unsolicited messages. And see what happened with those who wrote in messages full of acronyms. At some point, they will end up doing something which suggests they don't really value your time. To be clear, I am not talking about those who are already close to you who will then start writing acronyms because you now have some kind of shared vocabulary).
This is probably not a hard and fast rule though, so if you are very clear on the tradeoffs, you may want to handle these kinds of messages.
7 The lost souls
A client asked me "Someone keeps sending me the same message to my bot. Over and over. They come and do it on multiple days. What should I do?"
Obviously this is a tough one - because its hard to know why people do these things. If you knew them personally, you might reach out and ask them if they are doing OK.
But online, the only thing you can do is to ignore these messages. Let it go to the Fallback intent and let these people tire or bore themselves out.
Some of the out-of-domain questions cannot be handled by your bot. Don't include them in any accuracy calculations.
Besides making you feel better about your bot's performance, it also is an important economic decision.
You need to allocate your scarce resources (your time, your cognitive powers, your focus etc.) on the stuff which will help you build a better chatbot rather than worry about stuff which will not actually improve anything for the real users of your bot.
1 Do you have these kinds of questions in your out-of-domain list? Just ignore them.
2 Also run a quick inventory of the messages you are handling in your "negative phrases" in your Fallback intents. Do you really need to handle them?
3 Lastly, does your bot receive a similar class of messages which should be ignored? Let me know in the comments below, and I will add it to my article. It might help out other bot builders in the future.