GETTING STARTED
Kore.ai XO Platform
Virtual Assistants Overview
Natural Language Processing (NLP)
Concepts and Terminology
Quick Start Guide
Accessing the Platform
Navigating the Kore.ai XO Platform
Building a Virtual Assistant
Help & Learning Resources
Release Notes
Current Version
Recent Updates
Previous Versions
CONCEPTS
Design
Storyboard
Overview
FAQs
Conversation Designer
Overview
Dialog Tasks
Mock Scenes
Dialog Tasks
Overview
Navigate Dialog Tasks
Build Dialog Tasks
Node Types
Overview
Intent Node
Dialog Node
Dynamic Intent Node
GenAI Node
GenAI Prompt
Entity Node
Form Node
Confirmation Node
Message Nodes
Logic Node
Bot Action Node
Service Node
Webhook Node
Script Node
Process Node
Agent Transfer
Node Connections
Node Connections Setup
Sub-Intent Scoping
Entity Types
Entity Rules
User Prompts or Messages
Voice Call Properties
Knowledge AI
Introduction
Knowledge Graph
Introduction
Terminology
Build a Knowledge Graph
Manage FAQs
Knowledge Extraction
Import or Export Knowledge Graph
Prepare Data for Import
Importing Knowledge Graph
Exporting Knowledge Graph
Auto-Generate Knowledge Graph
Knowledge Graph Analysis
Answer from Documents
Alert Tasks
Small Talk
Digital Skills
Overview
Digital Forms
Digital Views
Introduction
Widgets
Panels
Session and Context Variables
Context Object
Intent Discovery
Train
NLP Optimization
ML Engine
Overview
Model Validation
FM Engine
KG Engine
Traits Engine
Ranking and Resolver
Training Validations
NLP Configurations
NLP Guidelines
LLM and Generative AI
Introduction
LLM Integration
Kore.ai XO GPT Module
Prompts & Requests Library
Co-Pilot Features
Dynamic Conversations Features
Intelligence
Introduction
Event Handlers
Contextual Memory
Contextual Intents
Interruption Management
Multi-intent Detection
Amending Entities
Default Conversations
Conversation Driven Dialog Builder
Sentinment Management
Tone Analysis
Default Standard Responses
Ignore Words & Field Memory
Test & Debug
Overview
Talk to Bot
Utterance Testing
Batch Testing
Conversation Testing
Conversation Testing Overview
Create a Test Suite
Test Editor
Test Case Assertion
Test Case Execution Summary
Glossary
Health and Monitoring
NLP Health
Flow Health
Integrations
Actions
Actions Overview
Asana
Configure
Templates
Azure OpenAI
Configure
Templates
BambooHR
Configure
Templates
Bitly
Configure
Templates
Confluence
Configure
Templates
DHL
Configure
Templates
Freshdesk
Configure
Templates
Freshservice
Configure
Templates
Google Maps
Configure
Templates
Here
Configure
Templates
HubSpot
Configure
Templates
JIRA
Configure
Templates
Microsoft Graph
Configure
Templates
Open AI
Configure
Templates
Salesforce
Configure
Templates
ServiceNow
Configure
Templates
Stripe
Configure
Templates
Shopify
Configure
Templates
Twilio
Configure
Templates
Zendesk
Configure
Templates
Agents
Agent Transfer Overview
Custom (BotKit)
Drift
Genesys
Intercom
NiceInContact
NiceInContact(User Hub)
Salesforce
ServiceNow
Configure Tokyo and Lower versions
Configure Utah and Higher versions
Unblu
External NLU Adapters
Overview
Dialogflow Engine
Test and Debug
Deploy
Channels
Publishing
Versioning
Analyze
Introduction
Dashboard Filters
Overview Dashboard
Conversations Dashboard
Users Dashboard
Performance Dashboard
Custom Dashboards
Introduction
Custom Meta Tags
Create Custom Dashboard
Create Custom Dashboard Filters
LLM and Generative AI Logs
NLP Insights
Task Execution Logs
Conversations History
Conversation Flows
Conversation Insights
Feedback Analytics
Usage Metrics
Containment Metrics
Universal Bots
Introduction
Universal Bot Definition
Universal Bot Creation
Training a Universal Bot
Universal Bot Customizations
Enabling Languages
Store
Manage Assistant
Team Collaboration
Plan & Usage
Overview
Usage Plans
Templates
Support Plans
Invoices
Authorization
Conversation Sessions
Multilingual Virtual Assistants
Get Started
Supported Components & Features
Manage Languages
Manage Translation Services
Multiingual Virtual Assistant Behavior
Feedback Survey
Masking PII Details
Variables
Collections
IVR Settings
General Settings
Assistant Management
Manage Namespace
Data
Overview
Data Table
Table Views
App Definitions
Data as Service
HOW TOs
Build a Travel Planning Assistant
Travel Assistant Overview
Create a Travel Virtual Assistant
Design Conversation Skills
Create an ‘Update Booking’ Task
Create a Change Flight Task
Build a Knowledge Graph
Schedule a Smart Alert
Design Digital Skills
Configure Digital Forms
Configure Digital Views
Train the Assistant
Use Traits
Use Patterns
Manage Context Switching
Deploy the Assistant
Use Bot Functions
Use Content Variables
Use Global Variables
Use Web SDK
Build a Banking Assistant
Design Conversation Skills
Create a Sample Banking Assistant
Create a Transfer Funds Task
Create a Update Balance Task
Create a Knowledge Graph
Set Up a Smart Alert
Design Digital Skills
Configure Digital Forms
Configure Digital Views
Add Data to Data Tables
Update Data in Data Tables
Add Data from Digital Forms
Train the Assistant
Composite Entities
Use Traits
Use Patterns for Intents & Entities
Manage Context Switching
Deploy the Assistant
Configure an Agent Transfer
Use Assistant Functions
Use Content Variables
Use Global Variables
Intent Scoping using Group Node
Analyze the Assistant
Create a Custom Dashboard
Use Custom Meta Tags in Filters
Migrate External Bots
Google Dialogflow Bot
APIs & SDKs
API Reference
API Introduction
Rate Limits
API List
koreUtil Libraries
SDK Reference
SDK Introduction
Web SDK
How the Web SDK Works
SDK Security
SDK Registration
Web Socket Connect and RTM
Tutorials
Widget SDK Tutorial
Web SDK Tutorial
BotKit SDK
BotKit SDK Deployment Guide
Installing the BotKit SDK
Using the BotKit SDK
SDK Events
SDK Functions
Tutorials
BotKit - Blue Prism
BotKit - Flight Search Sample VA
BotKit - Agent Transfer
  1. Home
  2. Docs
  3. Virtual Assistants
  4. Analyzing Your Bot
  5. NLP Insights

NLP Insights

The NLP Insights feature helps you gain in-depth insights into the analytics data and assess your virtual assistant’s performance in identifying and executing tasks. You can improve your VA’s performance based on the insights. The Analyze > NLP Insights page shows the specific information in the following sections:

In the latest version of the XO Platform, the NLP Insights section retains only the NLP-related analytics data for task identification. The new Task Execution Logs section displays the analytics data related to a VA’s task execution.

NLP Analytics Fields

The following fields are available for NLP Analytics:

Intent Found

An intent refers to the goal the customer has in mind when typing in a question or comment. The  phrases used to express the intent are called user utterances. The Intent Found tab includes all the user utterances identified by the platform. 

Note: You need to check for false positives in scenarios where the utterance is wrongly identified for an intent.

See the following table and Features section to know more:

The following is an example of Intent Found:

User: I want to know my order confirmation

VA: Would you like to switch to Track Order

User: Yes

VA: Let’s log you in.
      How would you like to go ahead – Log in or Guest?

In the above conversation, the user utterance of ‘knowing the order confirmation status’ is recognized by the VA and successfully mapped to the Track Order intent.

Description of the Intent Found Fields

The following table lists the fields on the Intent Fount tab with descriptions:

Fields Description
Utterances The utterances that are identified by the VA. The details on the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Intent The intent/task that is identified. The details on the tab can be grouped by intent. By default, the  Group by Intent option is turned off. To turn it on, click the Intent header and enable the Group by Intent option.

If the intent has been identified through a Dialog Task, this column displays the task’s name. If the intent is answered using the Answer from Document feature, this column mentions Answer from Documents.

Traits All the traits that are identified for the listed utterances. The details on the tab can be grouped by traits. By default, the  Group by Traits option is turned off. To turn it on, click the Traits header and enable the Group by Traits option.

This information is available for the data generated after June 1, 2021.

UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

Intent Not Found

Intent Not Found includes all the user utterances that the platform is not able to identify with a dialog task or FAQ either due to invalid training, less training data, or the intent unavailability in the virtual assistant.

See  the following table and Features section to know more:

Example of Intent Not Found: 

User: I want to know my account statement

VA: I’m sorry, I did not recognize the value you have entered. Please select a value from the list.

In the above conversation, the VA does not recognize the user utterance of ‘knowing the account statement’. It could be due to invalid training, less training, or unavailability of intents in the virtual assistant.

Description of Intent Not Found Fields

The following table lists the fields on the Intent Not Found tab with descriptions:

Fields Description
Utterances The utterances that are not identified by the VA. The details on the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Traits All the traits that are identified for the listed utterances. The details on the tab can be grouped by traits. By default, the  Group by Traits option is turned off. To turn it on, click the Traits header and enable the Group by Traits option.

This information is available for data  generated after June 1, 2021.

UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

Unhandled Utterances

The Unhandled Utterances help analyze the unidentified inputs received from the users during a task execution at an entity node, message node, or confirmation node. These insights allow you to identify the need for additional training or new intents and enhance the existing NLU model of the VAs.

In an unhandled utterance, the following new fields are available:

  • Prompt Type – A prompt type could be either an Entity node, Message node, or a Confirmation node.
  • Node Name – Name of the node in which the utterance is not handled
  • Task Name – Name of the task in which the utterances are unidentified on an entity, message, or confirmation nodes.

  • Group by functionality is available for Utterances, Traits, Prompt Type, Task Name, and Node Name
Note: The unhandled utterances are available for all conversations with product version 9.3 or higher.

See the following table and Features section to know more..

Unhandled Utterance Examples

The following examples show the conversations between the VA and user, with unhandled utterances captured at Entity, Message, and Confirmation nodes.

At an Entity Node:

For example, when the user provides an invalid input at entity or confirmation nodes as follows:

User: I want my account statement

VA: Please enter your Customer Id

User: Where do I find it?

VA: Sorry, that is an incorrect input. Please enter your Customer Id

In the above conversation, if the VA doesn’t recognize “where to find customer id” as an intent or entity, then this utterance is  categorized under Unhandled Utterances.

At a Message Node:

For example, when the user provides an invalid input at a message node as follows:

User: I want to book a flight for today

VA: Enter the flight number
            User: 12434
            VA: Enter number of seats required
            User: 3
            VA: Your flight is booked. Would you like to:
                    1. Book a Hotel
                    2. Book a sightseeing tour
            User: I want to Cancel the Flight
            VA: I’m sorry, I don’t understand. Please enter again.

In the above conversation, if the VA doesn’t recognize the intent name “cancel flight” as an input at the message node. The intent identification fails and the utterance is categorized under Unhandled Utterances.

At Confirmation Node:

For example, when the user provides an invalid input at a confirmation node as follows:

VA: How may I help you

User: I want to book a flight

VA: Enter the number of seats

User: 2

VA: Please confirm if you want two seats

User: I want to hire a cab

VA: I cannot understand it, can you rephrase it

In the above conversation, when a user enters ‘I want to hire a cab’ at the confirmation node, it is not recognized and logged under Unhandled Utterances.

Description of Unhandled Utterances Fields

The following table lists the fields on the Unhandled Utterances tab with descriptions:

Fields Description
Utterances The unhandled utterances for which the inputs received are unidentified. The details in the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Traits All the traits that are identified for the listed utterances. The details in the tab can be grouped by traits. By default, the  Group by Traits option is turned off. To turn it on, click the Traits header and enable the Group by Traits option.

This information is available for analytics generatedafter June 1, 2021.

Prompt Type A prompt type could be either an Entity node, Message node, or a Confirmation node. To turn on grouping by Prompt Type, click the Prompt Type header and enable the Group by Prompt Type option.
Task Name The task that is identified for the user utterance. To turn on grouping by task name, click the Task Name header and enable the Group by Task option.
Node Name The name of the service or script or WebHook within the task that got executed in response to the user utterance. To turn on grouping by node names to which these scripts or services belong, click the Node Name header and turn on the Group by NodeName option.
UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

Pinned

Any records from Identified and Unidentified Intents, or Unhandled Utterances tabs that are pinned are displayed in the Pinned tab. 

Description of Pinned Fields

The following table lists the fields on the Pinned tab with descriptions:

Fields Description
Utterances The pinned utterances are displayed here. The details in the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Intent The intent associated with the pinned utterance. The details in the tab can be grouped by Intents. By default, the  Group by Intent option is turned off. To turn it on, click the Intent header and enable the Group by Intent option.
Type of Issue Shows the reason for failure in case of Task Failure records.

To know the usual type of issues, see Failed Task – Type of Issues.

UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

NLP Insights Analysis

The following sections describe more about the options available on NLP Insights page and the analysis of the records captured here.

Features

The following list details the features available in NLP Insights for Intent Found, Intent Not Found, and Unhandled Utterances.

  • You can filter the information based on various criteria such as User Utterances, Intent, user id (Kore user id or channel-specific unique id), date-period, the channel of use, language, etc. You can also filter records based on multiple custom tags. See Filter Criteria to know more.
  • Complete meta information is stored for later analysis, including the original user utterance, the channel of communication, entities extracted (if any), custom tags applied, detailed NLP analysis with scores returned from each engine, and the ranking and resolver scores.
  • Ability to view the chat transcript to the point of the user utterance. This also gives the option to view the user profile and the details for that user’s conversation sessions.
  • You have an option to train the utterance.  The utterance will be marked once trained.
  • Any important record you want to mark, track later, or both can be pinned. They l appear on  the Pinned tab.
  • Sorting feature is available for Date and Time (Oldest to Newest, Newest to Oldest). You can export the insights data as a CSV file.

Note: The NLP Insights page shows the conversations from the last 24 hours by default. You can filter the insights for a selected period – use the Date drop-down to select 24 hours, last 7 days, or a custom period.

Fields Matrix

The following matrix shows the availability of fields on each tab of NLP Insights:

Fields Intent Found Intent Not Found Unhandled Utterances Pinned
Utterances
Intent X X
Traits X
UserID
Language
Date & Time
Prompt Type X X X
Task Name X X X
Node Name X X X
Failure Point X X X X
Type of Issue X X X
Type X X X X
Total Runs X X X X
Success% X X X X
2XX Responses X X X X
Non 2XX Responses X X X X
Avg Response Time X X X X
Log X X X X
Debug Point X X X X
Channel X X X X

Filter Criteria

You can filter the information on the Insights page using various filters. You can save the entered filter criteria and set it as the default filter using Save as Default Filter.

The filter criteria differ slightly between different tabs. The relevant filters are applied when you switch between the tabs on the Insights Page. See Dashboard Filter Criteria to know more details.

Detailed View

For all the user utterances listed on  the various tabs such as Intent Found, Intent Not Found, Unhandled Utterances, etc., you can open more details of the user session by clicking the respective record. The record shows the information on the following sub-tabs: Details, NLP Analysis, and Chat History.

Details

The Details tab shows the basic details of the session along with a JSON file that includes the NLP analysis for the conversation.

If the intent has been answered from a document, this section provides the following details:

  • Information on the Intent not being identified by the ML, FM, and KG engines. Hence, the user utterance is answered directly from the document;
  • The answer presented to the user;
  • The document from which the answer was provided;
  • A Similarity Score for how similar the user query is to the document content;
  • An option to add the query to the Knowledge Graph as an FAQ.

NLP Analysis

This tab provides a visual representation of the NLP Analysis, including intent scoring and selection.  See Testing and Training a Virtual Assistant and Ranking and Resolver for more information.

Chat History

On the Chat History tab, you can access the exact message or conversation for which the record is logged. It shows the entire chat history of the user session.

Chat History provides visibility into the user information by capturing the following details:

    • User Profile: Provides a 360-degree view of the user and their usage metrics.
    • User Conversation Sessions: Lists all the sessions of the user in the given period with the selected utterance section expanded.
    • Go to Selected Utterance: When you click this icon, the selected utterance is highlighted in orange (see the preceding screenshot).
    • Trace ID – A unique ID assigned to each incoming message. The Id is also included in all the logs maintained by the Platform.. When you hover over the message, the info icon appears. Click the Info icon to view the Message Id associated with the message.
    • Click the Message Id to view the Trace ID associated with a message in the Chat History.

      Note: The Trace ID is retained in the logs for 30 days. Once the Trace ID is expired, you see a tooltip message asTrace Id: Trace records for this message are not available‘.

The following user information details are displayed on the Chat History tab:

Functionality Attribute Description
User Profile Kore User ID User id assigned by the platform
Channel Data Data received from the channel, that is the information available in the User Context.
User Meta Tags The total number of meta tags associated with the user and key-value pairs for the most recent ones.
Latest Interaction Last time the user interacted with the VA.
Total Conversation Sessions The total number of interactive and non-interactive sessions registered by the user from the beginning of time.
Total Conversation Sessions in the Last 30 Days The total number of interactive and non-interactive sessions registered by the user in the last 30 days.
*The next few attributes are not displayed if there is no interaction by the user in the last 30 days
Last 30 Days’ Intent Detection Rate (Total identified intents / (Total identified intents + unidentified utterances)) * 100 for the utterances over the last 30 days
Intents Requested Total identified intents + unidentified utterances
Intents Identified Total intents identified
Last 30 Days Goal Completion Rate (Tasks success tasks / (Total success tasks + total failed tasks) ) * 100 for the tasks over the last 30 days
Tasks Initiated Total success tasks + total failed tasks
Tasks Completed Tasks successfully completed
Recent Conversation Flows Top 10 popular conversation flows executed by the user in the last 30 days. Popular flows are determined by the number of instances, for which the conversation flow is executed.
User Conversation Sessions Session Attributes
Session Start Session start date and time.
Session End Session end date and time.
Channel Channel in which the session is initiated.
Agent Transfer Tag The session where the user is transferred to an agent. Sessions should be considered even if the user returns to the VA.
Drop Off Tag The session where the user dropped off.
Total Success Tasks Count of tasks successfully completed in the session.
Total Failed Tasks Count of tasks failed in the session.
Intents Identified Count of intents successfully identified in the session.
Intents Unidentified Count of intents unidentified in the session and list of unidentified intents.
Conversation Path The series of tasks initiated by the user in the session.
Session Meta Tags Count of the session meta tags used with the details of the most recent custom meta tags displayed.
Conversation Transcript
Message Meta Tags The chat transcript is annotated with message tags for messages with meta-tags associated with them.
Agent Transfer Indicates the point of agent transfer at the last message before transfer.
Drop Off Indicate the point of drop off at the last message before dropping off.

Advanced Performance Details

Clicking a service or script or WebHook name opens the advanced details dialog for the service, which lists each instance of its run along with separate tabs for successful and failed runs. Analyzing the average response time of different runs gives you insights into any aberrations in the service or script execution. Click any row to open the JSON response associated with the service or script run.

Train the Virtual Assistant

You can train the specific intents and utterances from the Intent Found, Intent Not Found, and Unhandled Utterances tabs. To do so, hover over a row in any of these tabs, and click the Train icon. It opens the Test & Train page, where you can train the Virtual Assistant. For more information, see  Testing and Training a Virtual Assistant.

Data Export

You can export the data present on the NLP Insights page to a CSV file, by clicking the Export icon on the top right corner of the page.

Once you click the icon, the export process starts, and you can see the progress in the Status Tracker dock. The export file is downloaded to your local Downloads folder. The downloaded file has the information specific to the selected tab and the detailed analysis based on the selected filters.

These records also include the Meta Tag information.

NLP Insights

The NLP Insights feature helps you gain in-depth insights into the analytics data and assess your virtual assistant’s performance in identifying and executing tasks. You can improve your VA’s performance based on the insights. The Analyze > NLP Insights page shows the specific information in the following sections:

In the latest version of the XO Platform, the NLP Insights section retains only the NLP-related analytics data for task identification. The new Task Execution Logs section displays the analytics data related to a VA’s task execution.

NLP Analytics Fields

The following fields are available for NLP Analytics:

Intent Found

An intent refers to the goal the customer has in mind when typing in a question or comment. The  phrases used to express the intent are called user utterances. The Intent Found tab includes all the user utterances identified by the platform. 

Note: You need to check for false positives in scenarios where the utterance is wrongly identified for an intent.

See the following table and Features section to know more:

The following is an example of Intent Found:

User: I want to know my order confirmation

VA: Would you like to switch to Track Order

User: Yes

VA: Let’s log you in.
      How would you like to go ahead – Log in or Guest?

In the above conversation, the user utterance of ‘knowing the order confirmation status’ is recognized by the VA and successfully mapped to the Track Order intent.

Description of the Intent Found Fields

The following table lists the fields on the Intent Fount tab with descriptions:

Fields Description
Utterances The utterances that are identified by the VA. The details on the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Intent The intent/task that is identified. The details on the tab can be grouped by intent. By default, the  Group by Intent option is turned off. To turn it on, click the Intent header and enable the Group by Intent option.

If the intent has been identified through a Dialog Task, this column displays the task’s name. If the intent is answered using the Answer from Document feature, this column mentions Answer from Documents.

Traits All the traits that are identified for the listed utterances. The details on the tab can be grouped by traits. By default, the  Group by Traits option is turned off. To turn it on, click the Traits header and enable the Group by Traits option.

This information is available for the data generated after June 1, 2021.

UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

Intent Not Found

Intent Not Found includes all the user utterances that the platform is not able to identify with a dialog task or FAQ either due to invalid training, less training data, or the intent unavailability in the virtual assistant.

See  the following table and Features section to know more:

Example of Intent Not Found: 

User: I want to know my account statement

VA: I’m sorry, I did not recognize the value you have entered. Please select a value from the list.

In the above conversation, the VA does not recognize the user utterance of ‘knowing the account statement’. It could be due to invalid training, less training, or unavailability of intents in the virtual assistant.

Description of Intent Not Found Fields

The following table lists the fields on the Intent Not Found tab with descriptions:

Fields Description
Utterances The utterances that are not identified by the VA. The details on the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Traits All the traits that are identified for the listed utterances. The details on the tab can be grouped by traits. By default, the  Group by Traits option is turned off. To turn it on, click the Traits header and enable the Group by Traits option.

This information is available for data  generated after June 1, 2021.

UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

Unhandled Utterances

The Unhandled Utterances help analyze the unidentified inputs received from the users during a task execution at an entity node, message node, or confirmation node. These insights allow you to identify the need for additional training or new intents and enhance the existing NLU model of the VAs.

In an unhandled utterance, the following new fields are available:

  • Prompt Type – A prompt type could be either an Entity node, Message node, or a Confirmation node.
  • Node Name – Name of the node in which the utterance is not handled
  • Task Name – Name of the task in which the utterances are unidentified on an entity, message, or confirmation nodes.

  • Group by functionality is available for Utterances, Traits, Prompt Type, Task Name, and Node Name
Note: The unhandled utterances are available for all conversations with product version 9.3 or higher.

See the following table and Features section to know more..

Unhandled Utterance Examples

The following examples show the conversations between the VA and user, with unhandled utterances captured at Entity, Message, and Confirmation nodes.

At an Entity Node:

For example, when the user provides an invalid input at entity or confirmation nodes as follows:

User: I want my account statement

VA: Please enter your Customer Id

User: Where do I find it?

VA: Sorry, that is an incorrect input. Please enter your Customer Id

In the above conversation, if the VA doesn’t recognize “where to find customer id” as an intent or entity, then this utterance is  categorized under Unhandled Utterances.

At a Message Node:

For example, when the user provides an invalid input at a message node as follows:

User: I want to book a flight for today

VA: Enter the flight number
            User: 12434
            VA: Enter number of seats required
            User: 3
            VA: Your flight is booked. Would you like to:
                    1. Book a Hotel
                    2. Book a sightseeing tour
            User: I want to Cancel the Flight
            VA: I’m sorry, I don’t understand. Please enter again.

In the above conversation, if the VA doesn’t recognize the intent name “cancel flight” as an input at the message node. The intent identification fails and the utterance is categorized under Unhandled Utterances.

At Confirmation Node:

For example, when the user provides an invalid input at a confirmation node as follows:

VA: How may I help you

User: I want to book a flight

VA: Enter the number of seats

User: 2

VA: Please confirm if you want two seats

User: I want to hire a cab

VA: I cannot understand it, can you rephrase it

In the above conversation, when a user enters ‘I want to hire a cab’ at the confirmation node, it is not recognized and logged under Unhandled Utterances.

Description of Unhandled Utterances Fields

The following table lists the fields on the Unhandled Utterances tab with descriptions:

Fields Description
Utterances The unhandled utterances for which the inputs received are unidentified. The details in the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Traits All the traits that are identified for the listed utterances. The details in the tab can be grouped by traits. By default, the  Group by Traits option is turned off. To turn it on, click the Traits header and enable the Group by Traits option.

This information is available for analytics generatedafter June 1, 2021.

Prompt Type A prompt type could be either an Entity node, Message node, or a Confirmation node. To turn on grouping by Prompt Type, click the Prompt Type header and enable the Group by Prompt Type option.
Task Name The task that is identified for the user utterance. To turn on grouping by task name, click the Task Name header and enable the Group by Task option.
Node Name The name of the service or script or WebHook within the task that got executed in response to the user utterance. To turn on grouping by node names to which these scripts or services belong, click the Node Name header and turn on the Group by NodeName option.
UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

Pinned

Any records from Identified and Unidentified Intents, or Unhandled Utterances tabs that are pinned are displayed in the Pinned tab. 

Description of Pinned Fields

The following table lists the fields on the Pinned tab with descriptions:

Fields Description
Utterances The pinned utterances are displayed here. The details in the tab are grouped by utterances based on similarity by default. To turn off grouping by utterance, click the Utterances header and disable the Group by Utterances option.
Intent The intent associated with the pinned utterance. The details in the tab can be grouped by Intents. By default, the  Group by Intent option is turned off. To turn it on, click the Intent header and enable the Group by Intent option.
Type of Issue Shows the reason for failure in case of Task Failure records.

To know the usual type of issues, see Failed Task – Type of Issues.

UserID The UserID of the end user related to the conversation. You can view the metrics based on either Kore User id or Channel User Id.

Channel-specific ids are shown only for the users who have interacted with the VA during the selected period.

Language The language in which the conversation occurred.

If it is a multi-lingual VA, you can select specific languages to filter the conversation that occurred in those languages. The page shows the conversations that occurred in all enabled languages by default.
Date & Time The date and time of the chat. You can sort the data by either Newest to Oldest or Oldest to Newest.

NLP Insights Analysis

The following sections describe more about the options available on NLP Insights page and the analysis of the records captured here.

Features

The following list details the features available in NLP Insights for Intent Found, Intent Not Found, and Unhandled Utterances.

  • You can filter the information based on various criteria such as User Utterances, Intent, user id (Kore user id or channel-specific unique id), date-period, the channel of use, language, etc. You can also filter records based on multiple custom tags. See Filter Criteria to know more.
  • Complete meta information is stored for later analysis, including the original user utterance, the channel of communication, entities extracted (if any), custom tags applied, detailed NLP analysis with scores returned from each engine, and the ranking and resolver scores.
  • Ability to view the chat transcript to the point of the user utterance. This also gives the option to view the user profile and the details for that user’s conversation sessions.
  • You have an option to train the utterance.  The utterance will be marked once trained.
  • Any important record you want to mark, track later, or both can be pinned. They l appear on  the Pinned tab.
  • Sorting feature is available for Date and Time (Oldest to Newest, Newest to Oldest). You can export the insights data as a CSV file.

Note: The NLP Insights page shows the conversations from the last 24 hours by default. You can filter the insights for a selected period – use the Date drop-down to select 24 hours, last 7 days, or a custom period.

Fields Matrix

The following matrix shows the availability of fields on each tab of NLP Insights:

Fields Intent Found Intent Not Found Unhandled Utterances Pinned
Utterances
Intent X X
Traits X
UserID
Language
Date & Time
Prompt Type X X X
Task Name X X X
Node Name X X X
Failure Point X X X X
Type of Issue X X X
Type X X X X
Total Runs X X X X
Success% X X X X
2XX Responses X X X X
Non 2XX Responses X X X X
Avg Response Time X X X X
Log X X X X
Debug Point X X X X
Channel X X X X

Filter Criteria

You can filter the information on the Insights page using various filters. You can save the entered filter criteria and set it as the default filter using Save as Default Filter.

The filter criteria differ slightly between different tabs. The relevant filters are applied when you switch between the tabs on the Insights Page. See Dashboard Filter Criteria to know more details.

Detailed View

For all the user utterances listed on  the various tabs such as Intent Found, Intent Not Found, Unhandled Utterances, etc., you can open more details of the user session by clicking the respective record. The record shows the information on the following sub-tabs: Details, NLP Analysis, and Chat History.

Details

The Details tab shows the basic details of the session along with a JSON file that includes the NLP analysis for the conversation.

If the intent has been answered from a document, this section provides the following details:

  • Information on the Intent not being identified by the ML, FM, and KG engines. Hence, the user utterance is answered directly from the document;
  • The answer presented to the user;
  • The document from which the answer was provided;
  • A Similarity Score for how similar the user query is to the document content;
  • An option to add the query to the Knowledge Graph as an FAQ.

NLP Analysis

This tab provides a visual representation of the NLP Analysis, including intent scoring and selection.  See Testing and Training a Virtual Assistant and Ranking and Resolver for more information.

Chat History

On the Chat History tab, you can access the exact message or conversation for which the record is logged. It shows the entire chat history of the user session.

Chat History provides visibility into the user information by capturing the following details:

    • User Profile: Provides a 360-degree view of the user and their usage metrics.
    • User Conversation Sessions: Lists all the sessions of the user in the given period with the selected utterance section expanded.
    • Go to Selected Utterance: When you click this icon, the selected utterance is highlighted in orange (see the preceding screenshot).
    • Trace ID – A unique ID assigned to each incoming message. The Id is also included in all the logs maintained by the Platform.. When you hover over the message, the info icon appears. Click the Info icon to view the Message Id associated with the message.
    • Click the Message Id to view the Trace ID associated with a message in the Chat History.

      Note: The Trace ID is retained in the logs for 30 days. Once the Trace ID is expired, you see a tooltip message asTrace Id: Trace records for this message are not available‘.

The following user information details are displayed on the Chat History tab:

Functionality Attribute Description
User Profile Kore User ID User id assigned by the platform
Channel Data Data received from the channel, that is the information available in the User Context.
User Meta Tags The total number of meta tags associated with the user and key-value pairs for the most recent ones.
Latest Interaction Last time the user interacted with the VA.
Total Conversation Sessions The total number of interactive and non-interactive sessions registered by the user from the beginning of time.
Total Conversation Sessions in the Last 30 Days The total number of interactive and non-interactive sessions registered by the user in the last 30 days.
*The next few attributes are not displayed if there is no interaction by the user in the last 30 days
Last 30 Days’ Intent Detection Rate (Total identified intents / (Total identified intents + unidentified utterances)) * 100 for the utterances over the last 30 days
Intents Requested Total identified intents + unidentified utterances
Intents Identified Total intents identified
Last 30 Days Goal Completion Rate (Tasks success tasks / (Total success tasks + total failed tasks) ) * 100 for the tasks over the last 30 days
Tasks Initiated Total success tasks + total failed tasks
Tasks Completed Tasks successfully completed
Recent Conversation Flows Top 10 popular conversation flows executed by the user in the last 30 days. Popular flows are determined by the number of instances, for which the conversation flow is executed.
User Conversation Sessions Session Attributes
Session Start Session start date and time.
Session End Session end date and time.
Channel Channel in which the session is initiated.
Agent Transfer Tag The session where the user is transferred to an agent. Sessions should be considered even if the user returns to the VA.
Drop Off Tag The session where the user dropped off.
Total Success Tasks Count of tasks successfully completed in the session.
Total Failed Tasks Count of tasks failed in the session.
Intents Identified Count of intents successfully identified in the session.
Intents Unidentified Count of intents unidentified in the session and list of unidentified intents.
Conversation Path The series of tasks initiated by the user in the session.
Session Meta Tags Count of the session meta tags used with the details of the most recent custom meta tags displayed.
Conversation Transcript
Message Meta Tags The chat transcript is annotated with message tags for messages with meta-tags associated with them.
Agent Transfer Indicates the point of agent transfer at the last message before transfer.
Drop Off Indicate the point of drop off at the last message before dropping off.

Advanced Performance Details

Clicking a service or script or WebHook name opens the advanced details dialog for the service, which lists each instance of its run along with separate tabs for successful and failed runs. Analyzing the average response time of different runs gives you insights into any aberrations in the service or script execution. Click any row to open the JSON response associated with the service or script run.

Train the Virtual Assistant

You can train the specific intents and utterances from the Intent Found, Intent Not Found, and Unhandled Utterances tabs. To do so, hover over a row in any of these tabs, and click the Train icon. It opens the Test & Train page, where you can train the Virtual Assistant. For more information, see  Testing and Training a Virtual Assistant.

Data Export

You can export the data present on the NLP Insights page to a CSV file, by clicking the Export icon on the top right corner of the page.

Once you click the icon, the export process starts, and you can see the progress in the Status Tracker dock. The export file is downloaded to your local Downloads folder. The downloaded file has the information specific to the selected tab and the detailed analysis based on the selected filters.

These records also include the Meta Tag information.

Menu