GETTING STARTED
Kore.ai XO Platform
Virtual Assistants Overview
Natural Language Processing (NLP)
Concepts and Terminology
Help & Learning Resources
Quick Start Guide
Accessing the Platform
Navigating the Kore.ai XO Platform
Building a Virtual Assistant
Using Workspaces
Release Notes
Current Version
Previous Versions
Deprecations
Request a Feature
CONCEPTS
Design
Storyboard
Overview
FAQs
Conversation Designer
Overview
Dialog Tasks
Mock Scenes
Dialog Tasks
Overview
Navigate Dialog Tasks
Build Dialog Tasks
Nodes & Connections
Overview
Node Types
Intent Node
Dialog Node
Entity Node
Entity Rules
Form Node
Confirmation Node
Message Nodes
Logic Node
Bot Action Node
Service Node
Webhook Node
Script Node
Process Node
Agent Transfer
Node Connections Setup
Context Object
Sub-Intent Scoping
User Prompts
Voice Call Properties
Dialog Task Management
Event Handlers
Supported Entity Types
Supported Company Names
Supported Colors
Knowledge Graph
Introduction
Knowledge Extraction
Build Knowledge Graph
Create Node Structure
Build the Graph
Add FAQs
Add FAQs from an Existing Source
Run a Task
Traits, Synonyms, and Stop Words
Manage Variable Namespaces
Update Knowledge Graph
Introduction
Move Question and Answers Between Nodes
Edit and Delete Terms
Edit Questions and Responses
Knowledge Graph Analysis
Knowledge Graph Import and Export
Prepare Data for Import
From a CSV File
From a JSON File
Importing Knowledge Graph
Exporting Knowledge Graph
Auto-Generate Knowledge Graph
Alert Tasks
Small Talk
Digital Skills
Overview
Digital Forms
Digital Views
Introduction
Widgets
Panels
Train
NLP Optimization
ML Engine
Overview
Model Validation
FM Engine
KG Engine
Traits Engine
Ranking and Resolver
Training Validations
NLP Configurations
NLP Guidelines
Intelligence
Introduction
Default Standard Responses
Contextual Memory
Contextual Intents
Interruption Management
Multi-intent Detection
Amending Entities
Default Conversations
Conversation Driven Dialog Builder
Sentinment Management
Tone Analysis
Test & Debug
Overview
Talk to Bot
Utterance Testing
Batch Testing
Conversation Testing
Health and Monitoring
Deploy
Channels
Publishing
Versioning
Analyze
Introduction
Overview Dashboard
Conversations Dashboard
Users Dashboard
Performance Dashboard
Custom Dashboards
Introduction
Custom Meta Tags
Create Custom Dashboard
NLP Insights
Conversations History
Conversation Flows
Analytics Dashboard Filters
Usage Metrics
Containment Metrics
Smart Bots
Universal Bots
Introduction
Universal Bot Definition
Universal Bot Creation
Training a Universal Bot
Universal Bot Customizations
Enabling Languages
Store
Manage Assistant
Plan & Usage
Overview
Usage Plans
Support Plans
Invoices
Authorization
Multilingual Virtual Assistants
Get Started
Supported Components & Features
Manage Languages
Manage Translation Services
Multiingual Virtual Assistant Behavior
Masking PII Details
Variables
Collections
IVR Settings
General Settings
Assistant Management
Manage Namespace
Data as Service
Data Table
Table Views
App Definitions
Sharing Data Tables or Views
HOW TOs
Build a Travel Planning Assistant
Travel Assistant Overview
Create a Travel Virtual Assistant
Design Conversation Skills
Create an ‘Update Booking’ Task
Create a Change Flight Task
Build a Knowledge Graph
Schedule a Smart Alert
Design Digital Skills
Configure Digital Forms
Configure Digital Views
Train the Assistant
Use Traits
Use Patterns
Manage Context Switching
Deploy the Assistant
Configure Agent Transfer
Use Bot Functions
Use Content Variables
Use Global Variables
Use Web SDK
Build a Banking Assistant
Design Conversation Skills
Create a Sample Banking Assistant
Create a Transfer Funds Task
Create a Update Balance Task
Create a Knowledge Graph
Set Up a Smart Alert
Design Digital Skills
Configure Digital Forms
Configure Digital Views
Add Data to Data Tables
Update Data in Data Tables
Add Data from Digital Forms
Train the Assistant
Composite Entities
Use Traits
Use Patterns for Intents & Entities
Manage Context Switching
Deploy the Assistant
Configure an Agent Transfer
Use Assistant Functions
Use Content Variables
Use Global Variables
Intent Scoping using Group Node
Analyze the Assistant
Create a Custom Dashboard
Use Custom Meta Tags in Filters
APIs & SDKs
API Reference
API Introduction
API List
API Collection
koreUtil Libraries
SDK Reference
SDK Introduction
SDK Security
SDK Registration
Web Socket Connect and RTM
Using the BotKit SDK
BotKit SDK Tutorial - Blue Prism
Widget SDK Tutorial
Web SDK Tutorial
ADMINISTRATION
Introduction to Admin Console
Administration Dashboard
User Management
Add Users
Manage Groups
Manage Roles
Assistant Management
Enrollment
Invite Users
Send Bulk Invites
Import User Data
Synchronize Users from AD
Security & Control
Using Single-Sign On
Security Settings
Cloud Connector
Analytics
Billing
  1. Home
  2. Docs
  3. Virtual Assistants
  4. Natural Language
  5. Training Validations

Training Validations

NLP models play a significant role in providing natural conversational experiences for your customers and employees. Improving the accuracy of the NLP models is a continuous journey and requires fine-tuning, as you add new use cases to your virtual assistant. The Kore.ai XO Platform proactively validates the NLP training provided to the virtual assistants and provides recommendations to improve the model. This article explains the available validations, how to view these validations, and how to validate the NLU Model.

Goal-Driven Training Validations

The ML engine enables you to identify issues proactively in the training phase itself with the following set of recommendations: 

  • Untrained Intents – notifies about intents that are not trained with any utterances so that you can add the required training. 
  • Inadequate training utterances – notifies the intents with insufficient training utterances so that you can add more utterances to them. 
  • Utterance does not qualify any intent (false negative) – notifies about an utterance for which the NLP model cannot predict any intent. For example, an utterance added to Intent A is expected to predict Intent A. Whereas in some cases the model won’t be able to predict neither the trained Intent A nor any other Intents within the model. Proactively identifying such cases helps you rectify the utterance and enhance the model for prediction. 
  • Utterance predicts wrong intent (false positive) Identifies utterances that predict intents other than the trained intent. For example, when you add an utterance similar to utterances from another intent, the model could predict a different intent rather than the intent it is trained to. Knowing this would help you to rectify the utterance and improve the model prediction.
  • Utterance predicts intent with low confidence – notifies about the utterances that have low confidence scores. With this recommendation, you can identify and fix such utterances to improve the confidence score during the virtual assistant creation phase.
  • Incorrect Patterns – notifies about the patterns that do not follow the right syntax along with the error. You can resolve such incorrect patterns to improve intent identification.
  • Wrong Entity Annotations – notifies wrongly annotated entities. For example, in the utterance ‘I want to travel to Hyderabad on Sunday 2pm’,. the Travel Date (Date type) entity is annotated with value ‘2PM’ (Time value). The platform checks for such wrong annotations and notifies the issue against the utterance, which helps to re-annotate the entity with the right values and improve entity recognition.
  • Short Utterance – notifies about the utterances whose word count is lesser than or equal to two. It helps you to follow best practices for the length of utterances, which depicts an actual end-user query and further improves the model’s accuracy.

How to View NLU Training Validations

  1. On the virtual assistant’s Build menu, click Natural Language -> Training.
  2. In the Intents tab, you can see the set of recommendations for the Intents and ML utterances.

    Note: The errors and warnings in this screen are examples. The ML validations vary based on the error or waning recommendation as explained in the Goal-Based NLU Training Validations section above.
  3. Hover over the validation options and view the following recommendations:
    • Hover on the Error icon to view the recommendations to resolve the error.
    • Note: An Error is displayed when the intent has a definite problem that impacts the virtual assistant’s accuracy or intent score. Errors are high severity problems.

    • Hover on the Warning icon and follow the instructions in the warning to enhance the training for ML utterances.

    Note: A warning is displayed when the issue impact the VA’s accuracy and it can be resolved. Warnings are less severe problems when compared to errors.

  4. Once you click on the Intent with error or warning, hover over the Bulb icon to view the summary of error or warning messages as illustrated below:

How to Use the NLU Validate Model

The NLU Validate Model provides the options to view the training module’s recommendations summary, refresh the recommendations, and view Confusions Matrix, and K-fold Cross Validation reports.

The Validate Model helps you to follow best practices and develop accurate Virtual Assistants quickly.

Recommendations Summary

In the Natural Language → Training page, click the Validate Model to see the summary of recommendations.

Note: The recommendations are listed and categorized in the preceding screenshot. Under the All tab, all errors and warnings are listed. Under the Error and Warning tabs, errors and warnings are listed, respectively.

In the recommendation, 5 intents have Patterns with invalid syntax, 5 is the count of the intents with the issue – patterns with invalid syntax. For any recommendation, the intent count is displayed first, followed by the type of the issue.

If the training model is updated with all the modifications, the platform automatically triggers a background task to refresh recommendations without explicitly informing the users.

Refreshing Recommendations

In the recommendation summary, you can refresh recommendations as explained in the previous section in some of the scenarios like:

  • Modify training data: When the training data is modified, you can check for new recommendations.
  • Implement recommendations: When a recommendation is implemented, you can verify if it is resolved.
  • Add new training data: When new intents or utterances are added, you can check for new recommendations.
  • When you click the Refresh icon, a timestamp of the last refresh is displayed along with the refreshed recommendations list. The timestamp is updated with every refresh.

    If there are untrained utterances in the training model, the platform provides you with the following options:

    Train and Regenerate

    This option allows you to train the model with untrained utterances, and then triggers a background task to generate recommendations from the latest model.

    The following steps explain the Train & Generate usage with an example.

    1. Click any intent and add a new utterance.
    2. Once the utterance is added, go to the main page and click the Refresh icon to refresh the recommendations. The following pop-up is displayed when there are untrained utterances.
    3. Click the Train & Regenerate button to train the utterances and regenerate the recommendations.
    4. A message about training initiation is displayed.
    5. Once training and regeneration of recommendations is completed, the status is displayed in the status docker.
    6. Regenerate

      Click the Regenerate button in the displayed pop-up if you want to trigger a background task that generates recommendations from the current model.

      Note: While performing the refresh, you cannot trigger the task again.
      If you click the Refresh icon when the model is being trained, recommendations from the latest Validate model are generated only once the training is completed.
    7. When you click the Refresh icon, the recommendations summary is refreshed and the count of recommendations is either increased, or decreased, or the recommendations are updated based on the latest results.

    For example, if new utterances are added to the intent, a new recommendation may get added to the summary list. Similarly, if a recommendation is implemented, the count of recommendations decreases. In a recommendation like 13 intents have very short utterances, if 2 intents are fixed, then the recommendation is updated to 11 intents have very short utterances.

    NLU Validation Options

    Next to the Validate Model, there is a drop-down with options to select either the Confusion matrix or K-fold Cross Validations.

    You can also click the Go button to access Confusion Matrix and K-fold Cross Validation reports.

    Confusion Matrix

    Confusion Matrix is useful in describing the performance of a classification model (or classifier) on a set of test data for which the true values are known. The graph generated by the confusion matrix presents an at-a-glance view of the performance of your trained utterances against the virtual assistant’s tasks. To learn more, see the Confusion Matrix section in Model Validation.

    The following screenshot shows the confusion matrix report.

  • If no data is available in the Confusion Matrix, you click the Generate button to create a report.
  • Whenever the Validate model is updated, you can click the Re-Run Model to generate the latest matrix. Once you rerun the model, the platform prompts you to either Train and Regenerate the recommendations if the model has unsaved changes or Regenerate the recommendations if the model is up to date.
  • Whenever you Train and Regenerate the recommendations, the matrix will also be regenerated. Similarly, whenever you regenerate the matrix, the recommendations are updated too.
  • K-Fold Cross Validation

    K-Fold Cross-Validation is a resampling procedure used to evaluate machine learning models on a limited data sample. The technique involves partitioning the data into subsets, training the data on a subset, and using the other subsets to evaluate the model’s performance. To learn more, see the K-Fold Cross Validation section in Model Validation.

    The following screenshot shows the K-Fold Cross-Validation report.

    Menu