GETTING STARTED
Kore.ai XO Platform
Virtual Assistants Overview
Natural Language Processing (NLP)
Concepts and Terminology
Quick Start Guide
Accessing the Platform
Navigating the Kore.ai XO Platform
Building a Virtual Assistant
Help & Learning Resources
Release Notes
Current Major Version
Recent Updates
Previous Versions
Deprecations
Request a Feature
CONCEPTS
Design
Storyboard
Overview
FAQs
Conversation Designer
Overview
Dialog Tasks
Mock Scenes
Dialog Tasks
Overview
Navigate Dialog Tasks
Build Dialog Tasks
Node Types
Overview
Intent Node
Dialog Node
Entity Node
Form Node
Confirmation Node
Message Nodes
Logic Node
Bot Action Node
Service Node
Webhook Node
Script Node
Process Node
Agent Transfer
Node Connections
Node Connections Setup
Sub-Intent Scoping
Entity Types
Entity Rules
User Prompts or Messages
Voice Call Properties
Knowledge AI
Introduction
Knowledge Graph
Introduction
Build a Knowledge Graph
Manage FAQs
Knowledge Extraction
Import or Export Knowledge Graph
Prepare Data for Import
Importing Knowledge Graph
Exporting Knowledge Graph
Auto-Generate Knowledge Graph
Knowledge Graph Analysis
Answer from Documents
Alert Tasks
Small Talk
Digital Skills
Overview
Digital Forms
Digital Views
Introduction
Widgets
Panels
Session and Context Variables
Context Object
Intent Discovery
Train
NLP Optimization
ML Engine
Overview
Model Validation
FM Engine
KG Engine
Traits Engine
Ranking and Resolver
Training Validations
NLP Configurations
NLP Guidelines
LLM and Generative AI
Intelligence
Introduction
Event Handlers
Contextual Memory
Contextual Intents
Interruption Management
Multi-intent Detection
Amending Entities
Default Conversations
Conversation Driven Dialog Builder
Sentinment Management
Tone Analysis
Default Standard Responses
Ignore Words & Field Memory
Test & Debug
Overview
Talk to Bot
Utterance Testing
Batch Testing
Conversation Testing
Conversation Testing Overview
Create a Test Suite
Test Editor
Test Case Assertion
Test Case Execution Summary
Glossary
Health and Monitoring
NLP Health
Flow Health
Integrations
Actions
Actions Overview
Azure OpenAI
Overview
Templates
BambooHR
Overview
Templates
Freshdesk
Overview
Templates
Freshservice
Overview
Templates
HubSpot
Overview
Templates
JIRA
Overview
Templates
Microsoft Graph
Overview
Templates
Open AI
Overview
Templates
Salesforce
Overview
Templates
ServiceNow
Overview
Templates
Stripe
Overview
Templates
Twilio
Overview
Templates
Agents
Agent Transfer Overview
Custom (BotKit)
Drift
Genesys
Intercom
NiceInContact
Salesforce
ServiceNow
Unblu
External NLU Adapters
Overview
Dialogflow Engine
Test and Debug
Deploy
Channels
Publishing
Versioning
Analyze
Introduction
Dashboard Filters
Overview Dashboard
Conversations Dashboard
Users Dashboard
Performance Dashboard
Custom Dashboards
Introduction
Custom Meta Tags
Create Custom Dashboard
Create Custom Dashboard Filters
NLP Insights
Conversations History
Conversation Flows
Conversation Insights
Feedback Analytics
Usage Metrics
Containment Metrics
Universal Bots
Introduction
Universal Bot Definition
Universal Bot Creation
Training a Universal Bot
Universal Bot Customizations
Enabling Languages
Store
Manage Assistant
Plan & Usage
Overview
Usage Plans
Support Plans
Invoices
Authorization
Multilingual Virtual Assistants
Get Started
Supported Components & Features
Manage Languages
Manage Translation Services
Multiingual Virtual Assistant Behavior
Feedback Survey
Masking PII Details
Variables
Collections
IVR Settings
General Settings
Assistant Management
Manage Namespace
Data
Overview
Data Table
Table Views
App Definitions
Data as Service
HOW TOs
Build a Travel Planning Assistant
Travel Assistant Overview
Create a Travel Virtual Assistant
Design Conversation Skills
Create an ‘Update Booking’ Task
Create a Change Flight Task
Build a Knowledge Graph
Schedule a Smart Alert
Design Digital Skills
Configure Digital Forms
Configure Digital Views
Train the Assistant
Use Traits
Use Patterns
Manage Context Switching
Deploy the Assistant
Use Bot Functions
Use Content Variables
Use Global Variables
Use Web SDK
Build a Banking Assistant
Design Conversation Skills
Create a Sample Banking Assistant
Create a Transfer Funds Task
Create a Update Balance Task
Create a Knowledge Graph
Set Up a Smart Alert
Design Digital Skills
Configure Digital Forms
Configure Digital Views
Add Data to Data Tables
Update Data in Data Tables
Add Data from Digital Forms
Train the Assistant
Composite Entities
Use Traits
Use Patterns for Intents & Entities
Manage Context Switching
Deploy the Assistant
Configure an Agent Transfer
Use Assistant Functions
Use Content Variables
Use Global Variables
Intent Scoping using Group Node
Analyze the Assistant
Create a Custom Dashboard
Use Custom Meta Tags in Filters
Migrate External Bots
Google Dialogflow Bot
APIs & SDKs
API Reference
API Introduction
API List
API Collection
koreUtil Libraries
SDK Reference
SDK Introduction
SDK Security
SDK Registration
Web Socket Connect and RTM
Installing the BotKit SDK
Using the BotKit SDK
SDK Events
Tutorials
BotKit - Blue Prism
BotKit - Flight Search Sample VA
BotKit - Agent Transfer
Widget SDK Tutorial
Web SDK Tutorial
ADMINISTRATION
Introduction to Admin Console
Administration Dashboard
User Management
Add Users
Manage Groups
Manage Roles
Data Tables and Views
Assistant Management
Enrollment
Invite Users
Send Bulk Invites
Import User Data
Synchronize Users from AD
Security & Control
Using Single-Sign On (SSO)
Two-Factor Authentication (2FA)
Security Settings
Cloud Connector
Analytics
Billing
  1. Home
  2. Docs
  3. Virtual Assistants
  4. Analyzing Your Bot
  5. Bot Summary and Usage Metrics

Bot Summary and Usage Metrics

Kore.ai Bots Platform provides in-depth insights and metrics on the performance of the virtual assistants. Following are the key features:

  • Bot Summary to get a bird’s eye view of all the bot configuration and capabilities in a place. See here for details.
  • The Overview Dashboard gives a snapshot of the Virtual Assistant’s conversations, user analytics, and performance over the period of time.
  • The Conversations Dashboard provides insights into the usage and containment of the conversation sessions.
  • The Users Dashboard displays the trends of total users interacting with the virtual assistant, with a breakup of new users and returning users.
  • The Performance Dashboard gets all the insights related to the performance of a virtual assistant. These insights will help a VA designer to improve the performance of the virtual assistant by making necessary changes to it.
  • The Custom Dashboard allows you to design your own reports and dashboards.
  • The NLP Insights helps you gain in-depth insights into the analytics data and assess your virtual assistant’s performance in identifying and executing tasks.
  • The Conversation History module allows you to review the transcripts of past conversations and label certain conversations for follow-up or further action.
  • The Conversation Flows provides a visual representation of the user journeys. The user interactions with the virtual assistants are analyzed by the platform to provide insights into the commonly used intents, paths traversed, and drop-off points.
  • The Conversation Insights groups the utterances in a cluster, based on their semantic meaning and provides a name to each of these groups, which avoids the need to analyze all the utterances of the cluster.
  • Feedback Analytics Dashboard under the Analyze section provides prebuilt dashboards to view the survey results and insights based on the duration, survey name, and type you select.
  • From the Usage Metrics section view real-time and standard dashboards on your bot’s performance. See here for details.
  • The Containment Metrics section provides insights into how effectively customers are engaging with the bot vs transferring to agents or dropping offs. . See here for details.

Bot Summary

Bot Summary can be accessed from the Build tab in the top menu. This page gives a detailed summary of the Bot configuration that includes various snippets of bot definition as widgets. Clicking on any of these widgets takes you to the corresponding module.

These snippets give an overview of the:

  • Tasks added to the bot (list of Linked Bots in case of a Universal Bot)
  • Knowledge Graph configured including the summary like the number of paths, terms, synonyms, etc.
  • Note: The Knowledge Graph widget in the Bot Summary shows the right details after training the bot. For more information on training the knowledge graph, see Knowledge Graph Training.

  • Small Talk groups along with the number of conversations added
  • Digital Skills added in terms of the number of Forms, Widgets, and Panels
  • Natural Language – utterances, patterns, and synonyms
  • Channels enabled
  • Change Logs – visible only to the owner of the bot;
  • Team working on the Bot – visible only to the owner of the bot;
  • Events Handlers, Environment Variables, and Content Variables  configured
  • Interruption Behavior, Authorization Profiles configured
  • Debug Logs and any other settings configured for this Bot.

Clicking any of the above-mentioned cards takes you to the corresponding page.

You can use this page to Import Bot and Export Bot and add New Tasks and New Group (Small Talk) too.

Usage Metrics

From the Analyze tab on the top menu, you can access the Usage Metrics page. This page is your one-stop bot activity and bot user overview. It includes both interactive and non-interactive sessions. See here for more details on sessions.

Real-time Status

The first section on the Usage Metrics page is Real-time Status. This gives an overview of your Bot usage at a given point in time. It is set to refresh every 30 secs, which can be customized from 10secs to 10 mins using the dropdown next to the Refresh button.

Please note that the Real-time Status metrics should be used only if the Conversation Session Behavior under the Manage Sessions module is set to ‘Proactively close the sessions after inactivity’ .

If your virtual assistant is using the ‘Do not proactively close the sessions after inactivity’ option, then the Real-time Metrics may not represent the correct information. This configuration keeps the conversation sessions open state as it anticipates that the user can resume the conversation post inactivity period. All such open conversation sessions are treated as active and hence they are included in the Real-time Status metrics.

The Realtime Status includes:

  • Active Users – number of users interacting with the Bot at that given moment in time.
  • Active Channels – clicking the card expands the list of active channels along with the number of users using the channel.
  • Active Agent Sessions – sessions that are in progress at a given instance.

The Refresh button can be used to get the latest data.

Other usage metrics give an idea regarding the bot’s performance in terms of the frequency of user interaction, channels of interaction, tasks performed, etc.

  • Chat Session Records: By default, Bot usage metrics based on usage of the bot during the past 24 hours are displayed. You can customize the metrics displayed by the time period and bot selection. Filter criteria can also be applied, see below for details.
  • An overview of the number of Successful Tasks being completed, the number of Conversation Sessions, and the number of Alerts Sent are displayed.
  • A graphic illustration of the Messages and Conversation Sessions. This includes
    • Messages – the number of messages exchanged between the user and Bot;
    • Conversation Sessions – the number of conversation sessions initiated with the bot ie the exchange between user and Bot without change of intent even after a specified idle time See here for details on the session definition.
  • Details of:
    • the number of Successful Tasks in the selected period of time;
    • the pie chart comparing Intent Recognition vs Failed Tasks;
    • Name and Count of successful tasks executions, with an option to export as CSV;
    • Channels of interaction with the number of messages exchanged;
    • Number sessions in Agent Transfer.

Containment Metrics

Notes:

Containment Metrics provides insights on whether the calls were successfully answered by the virtual assistant or if the calls were landed with agents.

The metrics are categorized into:

  • Self-service Conversations – Conversations that get executed successfully will be considered Self-service conversations. A conversation will be considered self-service if a task is successfully executed even without any user interaction.
  • Drop-offs – These include sessions where the user exits the chat session abruptly. This will include all the sessions where the session has ended in any of the following scenarios:
    • The user was in the middle of a task execution;
    • The last task ended as a Task Failure;
    • The last interaction/message resulted in an ‘intent not identified’ scenario;
    • Non-interactive sessions;
    • No response from the bot.
  • Agent Transfer Conversations – These include conversation sessions that reached the Agent Transfer Nodes of the Dialog Tasks.

Each category gives a bird’s eye view of:

  • The number of conversation sessions that have started in the selected time period;
  • The percentage increase or decrease when compared to the previous time period.

The following table gives the details of the metrics displayed.

Widget  Description Self Service Agent Transfer Drop Offs
Totals Percentage of conversation sessions in the selected time period – hourly if 24hr period is selected, daily for other intervals. Self Service Conversations Conversations that were transferred to an agent in the selected time period – hourly if a 24hr period is selected, daily for other intervals. Conversations where the user dropped off in the selected time period – hourly if a 24hr period is selected, daily for other intervals.
Insights

Percentage of conversations against buckets of:

  • Messages – Number of messages exchanged between the user and the bot in the session;
  • Duration – The time the user spent with the bot from the beginning of the session to the end of the sessions;
  • Tasks – The number of tasks that are successfully completed in the session

Record of all self-service conversations within the session.

The range for the duration, number of messages, and tasks can be set as per your need.

Record of all conversations within the session before Agent Transfer.

The range for the duration, number of messages, and tasks can be set as per your need.

Record of all conversations within the session before Drop Off.

The range for the duration, number of messages, and tasks can be set as per your need.

Conversation Flows The sequence of tasks and the count

Top conversation flows that were executed in Self-Service conversations.

For each flow, execution of the flow along with relevant details is displayed with the entire transcript displayed when clicked upon.

Top conversation flows that were executed in Agent Transfer

For each flow, execution of the flow along with relevant details is displayed with the entire transcript displayed when clicked upon.

Top conversation flows that resulted in user drop off.

For each flow, execution of the flow along with relevant details is displayed with the entire transcript displayed when clicked upon.

Tasks Name of the task and the count. N/A

The total number of failed tasks.

For each task, additional information like User Name, Failure Point, Failure Reason, Date & Time, and Channel would be given with the entire transcript displayed when clicked upon.

The total number of failed tasks.

For each task, additional information like User Name, Failure Point, Failure Reason, Date & Time, and Channel would be given with the entire transcript displayed when clicked upon.

Intents The cluster of similar intents/utterances and the count. N/A

The total number of times that this utterance was unidentified.

For each utterance, additional information like User Name, Utterance, Date & Time, and Channel would be given with the entire transcript displayed when clicked upon.

The total number of times that this utterance was unidentified.

For each utterance, additional information like User Name, Utterance, Date & Time, and Channel would be given with the entire transcript displayed when clicked upon.

Filter Criteria

A filter can be applied to display the required Metrics.

  • Select the Time Period:
    • 7 Days – Data aggregated over the past seven days is displayed.
    • 24 Hours – Only data aggregated during the immediately preceding 24 hours is displayed. This is the default setting.
    • Custom – Click to display the Custom Date Range dialog that you can use to define a specific date period. On the left side, select or enter a start date, and then on the right side, select or enter the end data for displaying activity data. Selection up to a maximum of 90 days preceding the current date is allowed.
  • Using the Filter option, select:
    • Metrics over a specified Channel.
    • Metrics for a specified Language.
    • The conversation session types can be either one or both (default setting) of the following:
      • interactive – the normal conversation between the user and the bot;
      • non-interactive – sessions where bot messages are presented to the user without any message from the user

      See here for details on conversation session types.

    • based upon the meta tags added at a message, user, or session level. Multiple tags can be selected wherein the ‘AND’ condition is applied.

Tag-based

You can filter the dashboard using meta tags added at a message, user, or session level. Multiple tags can be selected under the filter criteria, an ‘and‘ condition will be applied across multiple fields selected for filtering.

The following table gives the widget-wise applicability of meta tags on the dashboard.

Widget Type Session Tags User Tags Message Tags
Realtime Widgets Not Applicable Not Applicable Not Applicable
Successful Tasks Applicable Applicable Not Applicable
Alerts Sent Applicable Applicable Not Applicable
Sessions Applicable Applicable Not Applicable
Messages & Conversation Sessions Applicable Applicable Applicable for Chats
Successful Tasks Applicable Applicable Not Applicable
Intent Recognized vs. Failed Applicable Applicable Not Applicable
Top Tasks Applicable Applicable Not Applicable
Top Channels Applicable Applicable Not Applicable
Agent Transfer Applicable Applicable Not Applicable
Menu