I was recently asked by a client if it is possible to bulk upload intents from a CSV file to Dialogflow.
Apparently, there was a time when this feature was already available in Dialogflow. But it isn’t straight forward anymore, and its probably for a good reason.
CSV file structure
Typically, the CSV file structure is simple. There is a column with a user says message, and then another column with the text response. What you want to do: You upload this CSV format, and you have an agent at the end of it.
Things to consider
Now, this simple format is not sufficient. For example,
- how can you handle multiple user says messages?
- what if you want to add multiple text responses?
- what should be used as the intent’s name?
An alternative file structure
Suppose we create the following file structure to accommodate the points from the previous section. You have an intent ID, followed by intent name, followed by userSays phrase, and the final column will be your text response.
While this gives you the ability to use multiple userSays phrases as well as multiple text responses, it is also more complex to process this format. But there is still a major issue when it comes to using this: the follow up intent.
Sometime around May 2017, Dialogflow (which was then API.AI) introduced the followup intents feature. These allowed you to have a hierarchy of intents, like below.
Since the number of hierarchical levels is (theoretically) unlimited, you can imagine that it is not possible to capture this very complex structure in the CSV format at all.
Doing bulk upload using Import/Export functions
Does this mean there isn’t a way to use the CSV file and upload your intents directly? Actually you still have another option.
You can do it using the Import/Export ZIP files function.
Here is the summary:
- Prepare a set of JSON files representing the CSV files you wish to upload
- Make sure the JSON files are created using a specific file and directory structure
- ZIP the top level directory
- Import or Restore from the ZIP file you created
First, let us export a ZIP file for a simple project and take a look at the directory and file structure, as well as the contents of the JSON file.
I will use an agent which has exactly two intents (taken from the prebuilt smalltalk agent).
I went to the Export and Import tab, and clicked on Export the ZIP file for this simple agent. Once you uncompress the ZIP file, the following file/folder structure is revealed:
And you can inspect the smalltalk.agent.be_clever_usersays_en.json.
You see that it is an array of JSON objects, and an individual JSON object has an id, the data, isTemplate, count and updated fields.
Next, notice the response JSON object smalltalk.agent.be_clever.json
Say, you want to add your own intent.
You simply need to follow the same pattern as the other intents, and create two new JSON files, and ZIP it up and import it into Dialogflow.
So you generate a random GUID, and get all the other stuff exactly in place. In my example, I made a copy of both the smalltalk.agent.be_clever_usersays_en.json and the smalltalk.agent.be_clever.json files and added a random GUID in the copied files, and created a ZIP file and tried to do a restore.
Make sure the filenames match:
Here is what the end result looks like after the import:
In fact, an even easier option is to simply omit all the GUIDs in the intents as well as the userSays as you create your ZIP file. Dialogflow will simply see that the ID is missing and generate its own.
But there are a couple of important things to note:
- Dialogflow will not permit two intents with the exact same name – be sure to change the name
- Your files should be of the format filename.json and filename_usersays_en.json – if you don’t follow this convention, the import doesn’t work correctly
Now, how do we use this knowledge to import a CSV file?
If you have been following along, you need to do the following:
- From the CSV file, for each intent, you should automatically generate a JSON file with the appropriate text for userSays and text responses.
- Make sure you structure the JSON files into the folder correctly
- Create a ZIP file from the folder
- Import or Restore the ZIP file
You can write scripts to do this. If you are interested in learning more about how to write the script, leave a comment on this post and I will write a future post with some pointers on how to do that.
Update Jan 10 2018:
I created a tool which implements the ZIP file generation process described above.
You can check out the tool at this link. You can watch the demo video below to understand how to use the tool.
Issues/Bugs while using the tool?
Please leave a comment on this post if you notice any bugs or issues when trying it with your CSV file.
Before you leave a bug report, please verify that there is no extra empty row or column in your CSV file as that will definitely mess up the ZIP file creation process.