Developers need to provide sample utterances for each intent (task) the assistant needs to identify, to train the machine learning model. The Kore.ai XO Platform Machine Learning (ML) engine will build a model that will try to map a user utterance to one of the VA’s intents.

Kore.ai’s XO Platform allows fully unsupervised machine learning to constantly expand the language capabilities of your assistant – without human intervention. Unlike unsupervised models in which AI assistants learn from any input – good or bad – the Kore.ai XO Platform enables assistants to automatically increase their vocabulary only when the VA successfully recognizes the intent and extracts the entities of a human’s request to complete a task.

However, we recommend keeping Supervised learning enabled to monitor the VA performance and manually tune where required. Using the XO Platform, developers can evaluate all interaction logs, easily change NL settings for failed scenarios, and use the learnings to retrain the VA for better conversations.

This article discusses the processes behind the Machine Learning Engine, and how to perform training for optimum performance.

 

The Machine Learning Process

Intent Detection

The below diagram summarizes the intent detection pipeline for both training and prediction stages. For the training pipeline, the language detection and auto-correction are not run with the assumption that the trainer would be aware of the language in which training needs to be done and of the spellings to be used which might include domain-specific non-dictionary words like Kore, etc.

Entity Extraction

Entity extraction involves identifying any information provided by the user apart from the intent that can be used in the intent fulfillment. The entities are of three types

1. System entities like date, time, color, etc are provided out-of-the-box by the Platform. It includes nearly 22-24 entities and these are recognized by the ML engine automatically with no training except for string & description entity types.
2. Custom entities are defined by the bot developer and these include the list of values – enumerated, lookup, and remote, regex expressions, and composite entities. These are also mostly auto-detected by the ML engine.
3. NER or named entity recognition needs the training to identify the same entity type for different entities e.g. source & destination cities for flight booking intent, both of which are city type entities and the engine needs the training to distinguish between the two. NER can be conditional random field-based or neural network-based. CRF is preferred since it works on lesser data and has a faster training time compared to the NN-based method.

The following diagram summarizes the NER entity extraction pipeline.

ML Output

The ML Engine runs the classification against the user utterance and generates the following scores output which Ranking and Resolver uses for identifying the correct intent:

• The probability Score for each class/intent, can be interpreted as follows
• Definitive Match/Perfect Match: If the probability score >0.95 (default and adjustable) 
• Possible match: If the score is <0.95%, it becomes eligible for ranking against other intents which may have been found by other engines.
• The fuzzy score for each of the traits/intents which are greater than the Threshold score(default is 0.3) – Fuzzy logic goes through each utterance of a given intent and compares it against the user input to see how close the user input and the utterance are. The scores are usually from 0-100 and can be interpreted as follows:
• Definite Match/Perfect Match: If the score is above 95%(default  & adjustable) 
• Possible match: If the score is <95%, becomes eligible for Ranking against other intents which may have been found by other engines.
• CR Sentences – The ML engine also sends the top 5 ML utterances for each of those Intents which have qualified using the Threshold score. These 5 ML utterances are derived using the Fuzzy score. Ranking & Resolver uses these CR sentences to Rescore and choose the best of these utterances (compares each utterance against user input and chooses an utterance with topmost score)

Limitations

Though the ML model is very thorough, it has its own limitations, as follows: 

• In cases where sufficient training data is not available, the ML model tends to overfit small datasets and subsequently lead to poor generalization capability, which in turn leads to poor performance in production.
• Domain adaptation might be difficult if trained on datasets originating from some common domains like the internet or news articles.
• Controllability and interpretability are hard because, most of the time, they work like a black box, making it difficult to explain the results.
• Cost is high both in terms of resources and time.
• The above two points also result in maintenance or problem resolution being expensive (again both in terms of time & effort) and can result in regression issues.

Hence, ML engines augmented by FM engines would yield better results. One can train the assistant with a basic ML model, and any minor issues can be addressed using FM patterns and negative patterns for idiomatic sentences, command-like utterances, and quick fixes.

Training the ML Engine

Training Overview

The Machine Learning Engine builds a model, based on training through intent detection and entity extraction, as follows: 

• The intent prediction model is trained using statistical modeling and neural networks.  Intent classification tags the user utterance to a specific intent. The classification algorithm learns from the set of sample utterances that are labeled on how they should be interpreted. Training utterance preparation and curation is one of the most significant aspects of building a robust Machine learning model. 
• Entity Detection involves recognizing System Entities (Out Of the Box, Rule-based model), predicting Custom Entities (Custom-trainable Rules-based Model), and Named Entity Recognition (NER). System Entities are defined using built-in rules. However, using the NER approach, any named entity can be trained using Machine Learning by simply choosing the value from the ML training utterances and tagging them against the named entity. 

Training the ML Engine involves the following steps:

1. Choosing and gathering data that can be used as the training set,
2. Dividing the training set for evaluation and tuning (test and cross-validation sets),
3. Training a few ML models according to algorithms (feed-forward neural networks, support vector machines, and so on) and hyperparameters (for example, the number of layers and the number of neurons in each layer for neural networks),
4. Evaluating and tuning the model over test and cross-validation sets,
5. Choosing the best performing model and using it to solve the desired task.

The NLP module improves the performance by constantly validating the ML Engine and presents  actionable insights to resolve the conflicts in intent and entity training.

Optimizing Intent Recognition

 

Adding Machine Learning Utterances

To add utterances to the Machine Learning Engine, please follow the steps below:

1. Open the VA for which you want to add sample user utterances.
2. Select the Build tab from the top menu.
3. From the left menu, select the Natural Language -> Training option.
4. By default, the tab with a list of all Intents would be displayed.
5. You can use the filter option to restrict the display items to Dialog Intents, Sub Intent Dialogs or Sub-Intents. You can also choose to Include Hidden Tasks. 
6.  Click Utterances > + Utterance against the Intent for which you want to add the utterances.
   
   
7. The user utterance page opens.
8. Here is where you can enter the utterances.
   
   

   Note that utterances greater than 3,000 characters in length are not allowed.

   

Note: Utterances should be unique, but in the case of multiple intent models, the same utterance can be used across different models. Read more about multiple intent models below.

 

The negation of trained intents will be ignored by the Platform.

For example, consider a Travel Assistant trained on the Book a Flight utterance. A user might say: Your app takes me to the booking screen, but I don’t want to book a flight. In this case, the Book a Flight intent will not trigger.

 

Adding Entities for Named Entity Recognition

Apart from the intent, you can train your VA to recognize the entities, if present, in the user utterance. For example, if the user says: Book Flight from Hyderabad to Mumbai apart from recognizing the intent as “Book Flight” the source and destination of the flight should also be recognized. This can be achieved by marking the entities in the user utterance during training.

You can mark entities in your utterances, by selecting the entity value and clicking the corresponding entity name.

The Platform will also try to identify and mark the entities. You have the option to accept or discard these suggestions. The Platform will identify the entities based upon:

• System entities;
• Static List of items – either enumerated or lookup;
• NER trained entities (from above).

For each of the entities thus marked, the confidence scores identified by the ML engine are displayed. This score is available only when the Conditional Random Field is selected as the NER model.

Further, if you have enabled Entity Placeholders, the Platform will replace the entity values in the training utterance with entity name placeholders for training the ML model. Using actual entity values as well as multiple additions of an utterance with just a change in the entity value will have an adverse impact on the ML training model. The name of entities also starts contributing highly to the intent detection model.

 

Using Negative Patterns

Negative patterns can be used to eliminate intents detected by the Fundamental Meaning or Machine Learning models. Refer here to know more.

Training your Assistant

After you added user utterances, you should train the Kore.ai interpreter to recognize the utterances and the associated user intent. When you have untrained utterances in your VA, the following message is displayed:

“You have untrained utterances in your ML model. Train your VA to update with all your utterances.”

Click Train. A status bar is displayed to show progress for utterance training. When complete, the Utterances trained successfully message is displayed. The user utterances are added to the Machine Learning Database. You can further configure the ML engine, identify the dummy intents when a user utterance contains the words that are not used in the VA’s training i.e. vocabulary, refer here for more details.

Once you have trained your VA, you can test it on the newly trained data. Learn how to test your assistant.

Auto-Training

By default, machine learning is automatically trained for any defined user utterances whenever a task is:

• updated with a new
• task name or intent name,
• entity name or parameter name,
• entity type,
• VA name.
• published.
• suspended by the Admin.
• deleted by the Admin.

In the XO Platform, when auto-train is in progress, a warning message that untrained user utterances cannot be identified is displayed if you try to test the VA before auto-train is complete.

You can set up the Auto Train option as follows:

1. Open the VA for which you want to modify the settings.
2. Select the Build option from the top menu.
3. From the left navigation menu, Natural Language -> Advanced Settings.
4. Enable or Disable the Auto Training option as per your requirements.
   
   

Thresholds & Configurations

To train and improve the performance of your Assistant, Threshold and Configurations can be specified for all three NLP engines – FM, KG, and ML. You can access these settings under Build > Natural Language > Thresholds & Configurations.

Note: If your VA is multilingual, you can set the Thresholds differently for different languages. If not set, the Default Settings will be used for all languages.

 

The Threshold & Configurations for the ML engine are discussed in detail in the following sections.

Machine Learning Model Configuration Overview

The XO  Platform ver 6.3 upgraded its Machine Learning (ML) model to v3. This includes a host of improvements and also allows developers to fine-tune the model using parameters to suit business requirements. The developers can change parameters like stopword usage, synonym usage, thresholds, and n-grams, as well as opt between Deep Neural Network or Conditional Random Field-based algorithms for the Named-Entity Recognition (NER) model.

In v8.0 of the Platform, provision has been enabled to use the v5 of the ML intent model and externalize several hyperparameters. This can be achieved through the Advanced NLP Configuration, refer here for details.

When the Multiple Intents Model =is enabled, the ML Engine maintains multiple intent models for the VA as follows:

• Bot level Intent Model containing all the Primary Intents of the assistant which includes Primary Dialog Intents, and Alert Task Intents.
• Dialog Intent Models – one for every primary dialog intent and sub-dialog intent which includes the Sub-intent nodes added to the dialog definition, Sub-intents scoped as part of the Group nodes and Interruption exceptions added to the dialog definition.

You can configure the Thresholds and Configurations separately for each of the intent models. This includes:

• All the configurations under Thresholds and Configurations – ML Engine as discussed in the below section;
• All the ML Engine configurations under the Advanced NLP Configurations discussed in detail here.

 

The Multiple Intent Model

Training of “similar intents” with different purposes is usually difficult as the training given for an intent can add noise or conflict with the training given to the other intent. This is more evident in cases where the intents have a contextually different meaning or purpose.

Consider the following case: A user is in the Make a Booking task, so any query related to the Booking refund policy should be answered within this context. However, the query can also trigger FAQs from Cancel a Booking. 

Enabling the Multiple Intent Models from the Advanced NLP Configurations (see here for how) allows you to have a dedicated ML model only for the primary intents and separate ML Models for each of the dialogs with their associated sub-intents so that the intent detection of sub-intents gets preferential treatment.

Continuing with the above example, with a Multiple Intent Model, you can define a separate context-based FAQ and ensure a proper response to the user.

All the primary intents of the VA will be part of the Bot Level Intent Model. Each of the Dialog tasks will have its own ML Model consisting of all the sub-intents added to it. The Thresholds and Configurations can be individually configured for each of the models. For example, the Bot Level Intent Model can use ‘Standard’ Network Type and a specific Dialog’s intent model can use ‘LSTM’ Network Type.

 

Configuring Machine Learning Parameters

The XO Platform provides language-wise defaults for the following parameters related to the ML performance of your VA. You can customize them to suit your particular needs.

Points to note regarding ML configurations: The illustration below shows the list of all possible configurations and these are available for both single and multiple intent models. When the multiple intent model is enabled, you can configure the individual models by selecting the Configure link against the model. While there is only one VA level intent model, you can add multiple dialog intent models using the Add New button and configure each as per your requirements. Advanced ML Configurations can be applied from here or from the Advanced NLP Configurations section refer here for details.

 

Network Type

You can choose the Neural Network that you would like to use to train the intent models. This setting has been moved to Machine Learning from Advanced NLP Configurations in v8.1.

You can choose between the following types. Based on the selection additional configurations can be done from the Advanced NLP Configurations section, refer here for details.

• Standard.
• MLP-BOW – The bag-of-words model is a simplifying representation used in natural language processing and information retrieval. In this model, a text is represented as the bag of its words, disregarding grammar and even word order but keeping multiplicity.
• MLP-WordEmbeddings – Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing where words or phrases from the vocabulary are mapped to vectors of real numbers.
• LSTM (Long Short-Term Memory) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. LSTM has feedback connections and hence has the ability to capture long-term dependencies for texts of any length and is well suited for longer texts.
• CNN (convolutional neural networks) is a class of deep neural networks in deep learning most commonly applied to analyzing visual imagery. It makes use of the word order for a specific region size and has achieved remarkable results on various text classification tasks.
• Transformers use a Universal Sentence encoder in the vectorization stage of the Training pipeline. The output of the sentence encoder is fed to a Multi-Layer perceptron network for training. SentenceEncoder has an inbuilt capability of understanding the semantic similarity between sentences taking into account the synonyms and various usage patterns of the same sentence.
  The Universal Sentence Encoder encodes text into high-dimensional vectors that can be used for text classification, semantic similarity, clustering, and other natural language tasks. The model is trained and optimized for greater-than-word length text, such as sentences, phrases, or short paragraphs. It is trained on a variety of data sources and a variety of tasks with the aim of dynamically accommodating a wide variety of natural language understanding tasks. The input is the variable-length English text and the output is a 512-dimensional vector.
• KAEN (Kore Advanced Embeddings Network) – Models trained with Sentence Embeddings alone can not understand the domain-specific terminology especially if the words from training are non-dictionary words. Kore.ai provides a model which can understand the meaning of the sentence and at the same time give importance to the domain-specific terminology. There are two parallel layers in work in this model – one to optimize the weights against the sentence embeddings and the other to optimize the word importance for a given sentence.  The activation function used for these two layers is RReLU (Randomized Leaky Rectified Linear Unit, refer here for details)

 

ML Threshold

ML Threshold defines the criteria for qualifying a probability score of an intent to be a possible or definite match. The default value is set to 0.3. This means that any intent which scores >0.3 is considered as a qualified Intent. Intents scoring < 0.3 are rejected.

 

ML Definitive Score

Configure the threshold score for definite matches, which can be set to a value between 80-100%, with the following classification:

• Probability Score – If the probability score by the classification Engine is >0.95 (default which is adjustable using “ML Definitive Score” divided by 100) Intent is considered as a Definite Match/Perfect Match.
• Fuzzy logic goes through each utterance of a given Intent and compares it against the user input to see how close the user input and the utterance are (scores are usually from 0-100). If the score is above 95% (default which is adjustable using “ML Definitive Score”) Intent is considered as a Definite Match/Perfect Match.

 

Bot Synonyms

This setting is Disabled by default. Enable this option if you would like to consider intent synonyms in building the ML model.

Enabling Synonyms allows the ML model to take the synonyms defined under “Synonyms and Concepts” to be considered while training the ML model. It helps in avoiding preparing duplicate utterances.

For example: “I want to book a flight”.

If we had defined “buy”, “reserve”, “purchase” as synonyms of “book” and “ticket” or  “seat” as synonyms of “flight”, then you need not add training utterances like “I want to buy a ticket” or “I want to reserve a seat” etc.

 

Stop Words

This setting is Disabled by default. Enable this option if you would like to remove the stop words in the training utterances in building the ML model. Once enabled, stop words are used to filter out the words/phrases from the Training utterances before training the ML model and are removed from the user utterance before prediction.

This setting is not valid when Network Type is set to Transformer.

 

Feature Extraction

Using this option (introduced in ver8.0) you can associate the ML intent model with the preferred algorithm. This option is not valid when Network Type is set to MLP WordEmbeddings, LSTM, CNN, and Transformer.

The options are:

• n-gram – this is the default setting and can be used to define the contiguous sequence of words to be used from training sentences to train the model.
  For example, if Generate weather forecast report is the user utterance and if the n-gram is set to 2, then Generate weather, Weather forecast, and Forecast report are used in training the model. If n-gram is set to 3, then Generate weather forecast, and Weather forecast report will be used in training the model.
  You can set the n-gram using the n-gram Sequence Length – The minimum n-gram limit is 1 by default. You can set the maximum limit up to 4.
• skip-gram – when the corpus is very limited or when the training sentences, in general, contain fewer words then skip-gram would be a better option. For this you need to define:
• Sequence Length – the length for skip-gram sequences, with a minimum of 2 and a maximum of 4
• Maximum Skip Distance – the maximum words to skip to form the grams, with a minimum of 1 and a maximum of 3.

 

The NER Model

Choose the NER model to be used for entity detection. The XO Platform provides two entity recognition models for training using NER that follow the same approach with

• Conditional Random Fields: lightweight and is easy to use for all sizes of datasets
• Neural network: works well with medium to large datasets but training time is very high

Note: The CRF model supports all languages and the Deep Neural Network model supports English, Spanish, German, and French. This option appears on the screen only when the selected VA language is supported by the Deep Neural Network model.

 

Entity Placeholders

Enable to replace entity values present in the training utterances with the corresponding entity placeholders in the training model. Entity placeholders remove the contribution of real entity values in Intent detection. This works only when the entity training(NER) is done via ML. 

Note: Enabling this flag reduces scores contributed by entity values.

 

Example: I want to fly to London tomorrow.

In the above example, we don’t want the engine to learn that “London” and “tomorrow” are important features. Hence they are replaced with their Placeholders once NER is done and the Entity Placeholders flag is enabled. Training utterance becomes “I want to fly <to> on <date>.

This option is not valid when Network Type is set to Transformer.

Upgrading the ML Model (for old VAs)

All new VAs that are created use the new ML model by default. Developers can upgrade the ML model for old VAs or downgrade the model for the VAs using the new model.

If you are using a previous model of ML in the XO Platform, you can upgrade it as follows:

1. Open the assistant for which you want to upgrade the ML model and go to Natural Language > Thresholds & Configurations.
2. Expand Machine Learning. Under the ML Upgrade section, click the Upgrade Now button. It opens a confirmation window.
3. Click Upgrade and Train. You can see new customizable options under the Machine Learning section.

Note: If a VA is exported using the older model (V2) and imported as a new VA, it continues to be in the V2 model until you upgrade it.

 

Training Validations

The ML engine enables you to identify issues proactively in the training phase itself with the following set of recommendations: 

• Untrained Intents – notifies about intents that are not trained with any utterances so that you can add the required training. 
• Inadequate training utterances – notifies the intents that have insufficient training utterances so that you can add more utterances to them. 
• Utterance does not qualify any intent (false negative) – notifies about a utterance for which the NLP model cannot predict any intent. For example, an utterance added to Intent A is expected to predict Intent A. Whereas in some cases the model won’t be able to predict neither the trained Intent A nor any other Intents within the model. Identifying such cases proactively helps you to rectify the utterance and enhance the model for prediction. 
• Utterance predicts wrong intent (false positive) – Identifies utterances that predict intents other than the trained intent. For example, when you add an utterance that is similar to utterances from another intent, the model could predict a different intent rather than the intent to which it is trained to. Knowing this would help you to rectify the utterance and improve the model prediction
• Utterance predicts intent with low confidence – notifies about the utterances that have low confidence scores. With this recommendation, you can identify and fix such utterances to improve the confidence score during the virtual assistant creation phase itself.

Viewing NLU Training Validations

1. On the virtual assistant’s Build menu, click Natural Language -> Training.
2. In the Intents tab, you can see the set of recommendations for the Intents and ML utterances.
   Note: The errors and warnings in this screen are examples. The ML validations vary based on the error or waning recommendation as explained in the Goal-Based NLU Training Validations section above.
3. Hover over the validation options and view the following recommendations:
1. Hover on the Error icon to view the recommendations to resolve the error.
   
   

   Note: An Error is displayed when the intent has a definite problem that impacts the virtual assistant’s accuracy or intent score. Errors are high severity problems.

2. Hover on the Warning icon and follow the instructions in the warning to enhance the training for ML utterances.
   
   

   Note: A warning is displayed when the issue impacts the VA’s accuracy and it can be resolved. Warnings are less severe problems when compared to errors.

   

3. Once you click on the Intent with an error or warning, hover over the Bulb icon to view the summary of error or warning messages as illustrated below:
   

Exporting and Importing Machine Learning Utterances

You can import and export ML utterances of a VA into another in CSV and JSON formats. You can choose between ‘In-Development’ or ‘Published’ tasks to export, whereas importing utterances always replace the latest copy of the task.

How to Export or Import ML Utterances

1. On the VA’s Build menu, click Natural Language -> Training.
2. The ‘In-Development’ version of the VA’s ML utterances open by default. If you want to see the utterances in the ‘Published’ version, toggle on the top right side of the window to Published.
   

   Note: The export of ML utterances varies based on this selection as explained in the Versioning and Behavior of the Exported Utterances section below.

3. Click the options icon and select an option:
   1. Click Import Utterances and upload a CSV or JSON file with the utterances to import. -OR-
   2. Click Export Utterances and select CSV or JSON formats to export the utterances.

Versioning and Behavior of Imported Utterances

• The imported utterances in CSV/JSON entirely replace the utterances present in the latest copy of the tasks.
• If the task is in Upgrade in Progress status, the utterances related to the task get entirely replaced with the task utterances present in the imported file. The utterances in the Published copy of the task aren’t affected.
• If the task is in the Published status, an Upgrade in Progress copy of the task gets created by default and the new utterances present in the imported file will be added to the upgraded copy. The utterances in the Published copy of the task aren’t affected.

Versioning and Behavior of Exported Utterances

• When you export a VA’s utterances, all the utterances related to every task type – alert, dialog – get exported.
• When you export an In Development copy of the VA, the utterances of all tasks in the latest available copy get exported.
• When you export a Published copy of the assistant, all the utterances in the published state get exported.
• In the case of multi-language VAs, the export of utterances includes utterances added in all of the languages.
• Export of utterances to JSON includes NER tagging present in the tasks, whereas CSV export doesn’t include them.

Recommendations

General ML Training Recommendations

• Give a balanced training for all the intents that the VA needs to detect, add approximately the same number of sample utterances. A skewed model may result in skewed results.
• Provide at least 8-10 sample utterances against each intent. The model with just 1-2 utterances will not yield any machine learning benefits. Ensure that the utterances are varied and you do not provide variations that use the same words in a different order.
• Avoid training common phrases that could be applied to every intent, for example, “I want to”. Ensure that the utterances are varied for larger variety and learning.
• After every change, train the model and check the model. Ensure that all the dots in the ML model are diagonal (in the True-positive and True-negative) quadrant and you do not have scattered utterances in other quadrants. Train the model until you achieve this.
• Regularly train the assistant with new utterances.
• Regularly review the failed or abandoned utterances and add them to the utterance list against a valid task or intent.

NLP Intent Detection Training Recommendations

• If there are a good number of sample utterances, try training the VA using Machine Learning approach first, before trying to train the fundamental meaning model.
• Define bot synonyms to build a domain dictionary such as pwd for a password; SB for a savings bank account.
• After every change to the model training, run the batch testing modules. Test suites are a means to perform regression testing of your VA’s ML model.

NLP Entity Detection Training Recommendations

The best approach to train entities is based on the type of entity as explained below:

• Entity types like List of Items (enumerated, lookup), City, Date, Country do not need any training unless the same entity type is used by multiple types in the same task. If the same entity type is used in a task, use either of the training models to find the entity within the user utterances.
• When the entity type is String or Description, the recommended approach is to use Entity patterns and synonyms.
• For all other entity types, both NER and Patterns can be used in combination.

Entity Training Recommendations

• Use NER training where possible – NER coverage is higher than patterns.
• NER approach best suits detecting an entity where information is provided as unformatted data. For entities like Date and Time, the Platform has been trained with a large set of data.
• NER is a neural network-based model and will need to be trained with at least 8-10 samples to work effectively.

Suggested Reading
You might want to read on ML Model, refer here.