I got this question via bot.
The answer is No. Or Yes. Or Maybe.
Let me explain.
Can Google answer an unseen question correctly?
If you are aware of how search engines work under the hood you know that they build a sort of "inverted index" of web pages, which means they should have "saved" the web page in their index in the first place.
So let us do a little thought experiment. Imagine it is the year 2050. And suppose there is a famous celebrity named George Drooni.
People naturally want to find out about George Drooni, so they type the word "George Drooni" into their search engine of choice.
And nothing comes back. Apparently, Google has not yet saved his name into its inverted index.
What is an unseen question?
So does the above qualify as an unseen question? Well, theoretically yes. And not just because Google doesn't yet know about life in 2050 (well, at least we hope so!)
It is also because anything which lies outside the world wide web at this instant of time is not "seen" by Google, no matter how good or powerful its real time search capabilities. Maybe you had a quarrel with your friend this week. You are sure about that fact, but do you think if you type in a query like "your_name friend_name quarrel" it will show up on Google? Hopefully not. It was seen by you, but wasn't seen by Google.
In other words, if you had a quarrel with your friend, but there was no one around to capture it on their smartphone and post it to the internet, did the quarrel actually happen? 🙂
OK, so what about an intent you defined which had a training phrase, and someone just came in and typed out those exact words, but in a completely jumbled order?
As you can see, I exactly reversed the training phrase to generate my user query, and Dialogflow is able to identify the intent. Does this phrase qualify as an unseen question?
Even more important: suppose you do think this is an unseen question. Does it mean Dialogflow is really smart?
In reality, this is what is happening:
- You type in a training phrase
- Dialogflow matches the phrase with all the phrases in all the existing intents and assigns a score to each intent
- The intent with the highest score "wins" and is selected for mapping
A lot of things could go in to this mapping process. For example, I checked the Intent detection confidence score:
The ML threshold was set at 0.3.
What happens if I set the ML Threshold higher than 0.57? Let us try with ML Threshold of 0.7.
Not unsurprisingly, now it goes to the Default Fallback Intent.
But Dialogflow hasn't suddenly become dumber, either.
Known entity values
Suppose you declare an intent with a system entity in the training phrase, as you can see below.
Even though London doesn't appear in the training phrase (and wasn't "seen" before), Dialogflow is able to accurately map the phrase to this intent.
The essential point I am making is this: the notion of an unseen vs seen question is quite complex and nuanced.
And also, I would suggest not to get hung up on the definition of these terms during your chatbot development process. At the end of the day, Dialogflow is still software which comprises binary code full of 0s and 1s, and it is best to just treat it as such.
Even though Dialogflow is a black box, it is best to think of it as a mostly predictable pattern matching engine where you don't know the rules!
Also, if you are still reading, my Dialogflow Conversation Design course will help you dissect how intent mapping works (so the black box becomes a little less mysterious).