Recently, a couple of students/clients asked me about creating scripts for testing their Dialogflow agent's conversations.
In this article, I list the things you can test for when you are writing these scripts.
Prerequisite: You need to use the explicative approach
I have written about the explicative approach to Dialogflow development in an earlier article. One of the reasons I recommend that approach is because it actually makes automation of various Dialogflow tasks a whole lot easier.
As an example, many of the items in this list will become untestable (programmatically) if you were to use slot filling.
You will be testing your agent by calling the detectIntent method using Dialogflow's REST API v2.
The test script will be quite straightforward:
1 Write a set of utterances (which form a single conversation)
2 Send each utterance to your agent's detectIntent method one by one
3 Check the response to see if all the fields are as expected
4 If not, flag the utterance for review
Obviously it is a little more involved than that, but that is the gist of how you would approach the task.
List of things you can test for
Let us look at the list of things you can test using this approach.
Please note that not all of these issues are caused by Dialogflow failures. In fact many of them are human errors.
1 Fallback intent is mapped
You expect a specific intent to be mapped, but the fallback intent is mapped.
2 Wrong intent is mapped
Again, this issue is when a wrong intent is mapped, but the wrong intent isn't the Fallback intent either.
Two things can help you when you run into this issue:
a) understand how candidate intents work
b) use the candidate explorer tool to see why the wrong intent is a candidate in the first place
3 Incorrect action is set
The action that is set by an intent will determine the logic which is used by the webhook. You can also use the intent name for deciding the webhook logic, but the action is more likely to remain unchanged.
4 Incorrect output contexts are set
When an intent fires, you can set specific output contexts to help move the conversation in a direction you want. If the wrong output context is set, it might lead to wrong intents being mapped in the future steps in your conversation.
As you are checking the output contexts, it wouldn't be a bad idea to also verify if their lifespans are set to 1. Since Dialogflow autocompletes context lifespans to use a value of 5, you might have accidentally changed the lifespan when trying to modify a context value.
5 Incorrect or incomplete parameters
You will notice something quite surprising in Dialogflow. Sometimes, even when an intent fires, it doesn't extract the parameter value correctly. You can, in theory, use required parameters to force Dialogflow to get those values, but the benefit isn't worth the cost, in my opinion.
There are actually two failure modes here: one is Dialogflow not extracting the correct value of the parameter, and the other is where Dialogflow extracts no value at all (that is, if you inspect the parameter field, it will be empty).
6 Incorrect values of parameters from previous steps
You might use something like session variables to store parameter values across long conversations.
In that case, at each step, your session-vars output context will actually hold parameter values extracted during previous steps.
You need to check if those values are still available for use as they might be used in future conversation steps.
7 Webhook call fails with timeout
You have probably experienced this, especially when you make changes to your agent and test inside your simulator.
Remember that when the webhook call returns, it also contains a response code. You can inspect the value inside this code to check if there was a timeout. The image below shows some possible error codes.
For a timeout, I believe you get the 503 error code.
8 Webhook call fails with internal server error
Usually this indicates a more serious issue with your Dialogflow webhook code.
Sometimes you might receive this error code because the format of the JSON file you have constructed might be slightly off (e.g. missing a curly brace).
This error code is usually 500.
9 Webhook call fails with 404
If you get a 404 error code, it is usually a simple fix. Your fulfillment URL is usually pointing to the wrong resource (e.g. moving the folder where you deployed the code). Also make sure that your URL doesn't redirect, because Dialogflow doesn't allow redirect URLs to be used for fulfillment.
10 Webhook call returns but sets the wrong output context
In this case, you should get a normal 200 status code for the webhook call.
However, the webhook code might be trying to set an output context and might not be setting the value correctly. If your webhook does modify or update output contexts, you need to have this test in place because it can easily change the course of your conversation and send it down the wrong path.
11 Webhook call returns but invokes the wrong followupEvent
You can jump from one intent to another from your webhook by using the followupEvent feature.
When you do this, make sure to have a test in place which checks if the correct event is being invoked. Since the event is a string value that you set inside your Dialogflow intent, it is quite possible to make a silly mistake when writing out these string values.
12 Intent settings changed
There are also some miscellaneous settings inside a Dialogflow intent such as "does it call a webhook?", "does it represent the end of conversation (actions on google)" etc.
It is very easy to accidentally change these settings (for example you might be temporary changing these values to test something inside the DF simulator).
So having some tests in place for these types of settings isn't a bad idea, since the values for these fields can be verified as they are part of the JSON response coming back.
Do you need help setting up automation test scripts for your agent?
You can get in touch here.
- Dialogflow vs RASA NLU
- Dialogflow vs Lex vs LUIS vs Watson vs Chatfuel
- Machine Learning vs non-Machine Learning algorithm
- BotFlo update
- Learn Dialogflow basics for free (till May 31)
- 10+ practical projects to learn spaCy in depth
- An Epidemiology Glossary for Programmers
- All my mini-courses are free this week
- Reader Question: What if a specific system entity isn’t available in all languages in a multi-lingual bot?
- How much can Machine Learning ACTUALLY help with answering free-form questions?