Home / DialogFlow / How to bulk upload intents from a CSV file in Dialogflow
DialogFlow

How to bulk upload intents from a CSV file in Dialogflow

First published: Jan 2018 | Last update: Oct 2020

I was recently asked by a client if it is possible to bulk upload intents from a CSV file to Dialogflow.

Apparently, there was a time when this feature was already available in Dialogflow. But it isn’t straight forward anymore, and its probably for a good reason.

CSV file structure

Typically, the CSV file structure is simple. There is a column with a user says message, and then another column with the text response. What you want to do: You upload this CSV format, and you have an agent at the end of it.

Things to consider

Now, this simple format is not sufficient. For example,

  • how can you handle multiple user says messages?
  • what if you want to add multiple text responses?
  • what should be used as the intent’s name?

An alternative file structure

Suppose we create the following file structure to accommodate the points from the previous section. You have an intent ID, followed by intent name, followed by userSays phrase, and the final column will be your text response.

While this gives you the ability to use multiple userSays phrases as well as multiple text responses, it is also more complex to process this format. But there is still a major issue when it comes to using this: the follow up intent.

Followup intents

Sometime around May 2017, Dialogflow (which was then API.AI) introduced the followup intents feature. These allowed you to have a hierarchy of intents, like below.

Since the number of hierarchical levels is (theoretically) unlimited, you can imagine that it is not possible to capture this very complex structure in the CSV format at all.

Doing bulk upload using Import/Export functions

Does this mean there isn’t a way to use the CSV file and upload your intents directly? Actually you still have another option.

You can do it using the Import/Export ZIP files function.

Here is the summary:

  1. Prepare a set of JSON files representing the CSV files you wish to upload
  2. Make sure the JSON files are created using a specific file and directory structure
  3. ZIP the top level directory
  4. Import or Restore from the ZIP file you created

First, let us export a ZIP file for a simple project and take a look at the directory and file structure, as well as the contents of the JSON file.

I will use an agent which has exactly two intents (taken from the prebuilt smalltalk agent).

I went to the Export and Import tab, and clicked on Export the ZIP file for this simple agent. Once you uncompress the ZIP file, the following file/folder structure is revealed:

And you can inspect the smalltalk.agent.be_clever_usersays_en.json.

You see that it is an array of JSON objects, and an individual JSON object has an id, the data, isTemplate, count and updated fields.

Next, notice the response JSON object smalltalk.agent.be_clever.json

Say, you want to add your own intent.

You simply need to follow the same pattern as the other intents, and create two new JSON files, and ZIP it up and import it into Dialogflow. 

So you generate a random GUID, and get all the other stuff exactly in place. In my example, I made a copy of both the smalltalk.agent.be_clever_usersays_en.json and the smalltalk.agent.be_clever.json files and added a random GUID in the copied files, and created a ZIP file and tried to do a restore.

Make sure the filenames match:

Here is what the end result looks like after the import:

In fact, an even easier option is to simply omit all the GUIDs in the intents as well as the userSays as you create your ZIP file. Dialogflow will simply see that the ID is missing and generate its own.

But there are a couple of important things to note:

  • Dialogflow will not permit two intents with the exact same name – be sure to change the name
  • Your files should be of the format filename.json and filename_usersays_en.json – if you don’t follow this convention, the import doesn’t work correctly

CSV import

Now, how do we use this knowledge to import a CSV file?

If you have been following along, you need to do the following:

  • From the CSV file, for each intent, you should automatically generate a JSON file with the appropriate text for userSays and text responses.
  • Make sure you structure the JSON files into the folder correctly
  • Create a ZIP file from the folder
  • Import or Restore the ZIP file

You can write scripts to do this.

Demo App

I have created a demo app which shows how this works.

1 Choose the bot’s timezone

The timezone is important because it will affect the UTC time for any date time information which is parsed by the bot.

2 Choose the bot’s language

You can choose any language supported by Dialogflow.

3 Choose the CSV format

Right now, the app supports one of three different formats. The basic 4 column format supports only training phrases and responses. The more advanced 8 column format also supports input and output context, output context lifespan, and event. The 10 column format expands on the 8 column format and provides support for a second text response block, as well as a boolean Yes/No value for whether or not the intent calls a webhook.

Explanation of 4 Column CSV format

Sample 4 column CSV file

Sample 8 column CSV file

Sample 10 column CSV file

4 Select the CSV file from your file system

Use the Browse.. button and select an appropriate CSV file from your local computer.

5 Click on Download agent ZIP file

As soon as you select the CSV file, you should see a button appear which allows you to Download the agent ZIP file. Click on the button. You should now see that the agent ZIP file gets downloaded to your local computer.

6 Use the Restore ZIP file functionality in Dialogflow

Go to Dialogflow’s console, and use the Restore ZIP feature to upload the agent ZIP file generated by my app. If the Restore completes successfully, you should see all the stuff from your CSV file appear as intents inside your Dialogflow agent.

FREE COURSES

Related Posts

  • Great tutorials!

    Do the intent IDs need to be in sequential order, or can they be any random number?

    Can I use your tool to upload new intents? Where I select the “Import from zip” option?

    I am looking to train the agent, by uploading new intents, or even new training phrases.

    In order to train the agents, I would need to reupload a new zipfile and restore agent?

    Or would I need to:

    1- export the agent,

    2-convert the Json file back to csv (in order to get the list of intents)

    3. then add in my new intents,

    4. then convert back to the zip file using your tool?

    • >> Do the intent IDs need to be in sequential order, or can they be any random number?

      They can be any random number, but you do need to use a unique number per intent obviously.

      I think it is fair to say the rest of your questions boil down to whether or not my app can help with two way updates. Before I answer that question, can you tell me a little bit more about your use case
      a) what kind of bot are you creating?
      b) what kind of training will you be doing? (i.e. using system entities, or developer entities)?

      • Cool thanks for your reply!!

        a) I will be creating an FAQ bot in a sense. That explains which products are needed based on the user input.

        So I would need to update the bot with new intents (new products).

        b) for the training, I will be using Developer entities. Each new product is an entity.

        I would need to train the bot in various ways, the user will ask for this product.

        • I wrote a reply to your question.

          Also, is there any reason you cannot use a single entity to represent all your products, and use a webhook to fetch the answer? That might make your life much simpler.

  • Hello Aravind,
    I have a question about your application. I would like to be able to update the intents that I have on a regular basis. Is it possible to update the intents and integrate new intents into existing hierarchies and contexts with your app?

    Manuel

    • 1 If you mean update the intents from the CSV file, then yes.

      2 You cannot, however, update the intents in Dialogflow and convert the ZIP file in Dialogflow into a CSV format, because of the inherently flat structure of the CSV file.

      3 As for hierarchies, the CSV format cannot naturally support hierarchical intents (i.e. followup intents) again because of its flat structure.

      4 While in theory it is possible to modify the app to support followup intents, it will also make the UX a lot more confusing.

      5 My app does support contexts though, and you can update the contexts (and their lifespans) in the CSV and recreate the ZIP file.

    • In theory, this could happen if the language of your agent isn’t actually English, and you use the _en suffix for the user says JSON files. If you use a non-English language you should look up the corresponding language code and use it in the usersays file names.

  • Excellent tool! I had to wait 2 days before the app was unlocked on your free course, but was worth the wait. You should note: DialogFlow will not accept ZIPs with any intent names greater than 100 characters.

    • A note for those reading this comment: the app is now a paid app (with a very tiny, token payment). This stops the spammers who are using disposable email addresses to sign up.

      • Another update. I don’t offer the limited app anymore as it confused people a lot. If you want to see a demo of the app, you can check out the link mentioned in this post.

  • Hi Aravind.
    Your tool is awesome, but it doesn’t work with more than one response. I think that it should be possible if you find a way to make a comma in between responses (this should still just be in one column). For example:
    IntentID=1; IntentName=Hello; Query=Hello; Response=Hi, Hello;

    • Hi Ame,
      I have tested that it works with multiple responses, but only if you split the multiple responses into multiple lines in the original CSV file. Also the purpose of putting it into two different lines is that otherwise you will need to deal with the exact same issue you are talking about here – you need a separator to indicate the multiple lines (for example a comma like you have used here), and a) it is hard for people to write such text compared to simply putting it into multiple lines and b) you can run into parsing issues.

  • Thanks for a great documentation Aravind!
    I have a question though. Do you know if it’s possible to return with a custom payload instead of the text?

    • Hi Filip,
      This means you need to specify the custom payload JSON inside the CSV file, which I think is very hard to do. Do you mind posting your sample CSV file online where I could take a look at the custom payload’s format?