OVERVIEW
Virtual Assistants
Kore.ai Platform
Key Concepts
Natural Language Processing (NLP)
Accessing Platform
VIRTUAL ASSISTANTS
Virtual Assistant Builder
Virtual Assistant Types
Getting Started
Creating a Simple Bot
SKILLS
Storyboard
Dialog Task
Introduction
Dialog Builder (New)
Dialog Builder (Legacy)
User Intent Node
Dialog Node
Entity Node
Supported Entity Types
Composite Entities
Supported Colors
Supported Company Names
Form Node
Logic Node
Message Nodes
Confirmation Nodes
Bot Action Node
Service Node
Custom Authentication
2-way SSL for Service nodes
Script Node
Agent Transfer Node
WebHook Node
Grouping Nodes
Connections & Transitions
Manage Dialogs
User Prompts
Knowledge Graph
Terminology
Building
Generation
Importing and Exporting
Analysis
Knowledge Extraction
Build
Alert Tasks
Introduction
Ignore Words and Field Memory
How to Schedule a Smart Alert
Small Talk
Digital Views
Overview
Configuring Digital Views
Digital Forms
Overview
How to Configure Digital Forms
NATURAL LANGUAGE
Overview
Machine Learning
Introduction
Model Validation
Fundamental Meaning
Introduction
NLP Guidelines
Knowledge Graph
Traits
Introduction
How to Use Traits
Ranking and Resolver
Advanced NLP Configurations
INTELLIGENCE
Overview
Context Management
Overview
Session and Context Variables
Context Object
How to Manage Context Switching
Manage Interruptions
Dialog Management
Sub-Intents & Follow-up Intents
Amend Entity
Multi-Intent Detection
Sentiment Management
Tone Analysis
Sentiment Management
Event Based Bot Actions
Default Conversations
Default Standard Responses
TEST & DEBUG
Talk to Bot
Utterance Testing
Batch Testing
Conversation Testing
CHANNELS
PUBLISH
ANALYZE
Overview
Dashboard
Custom Dashboard
Overview
How to Create Custom Dashboard
Conversation Flows
NLP Metrics
ADVANCED TOPICS
Universal Bots
Overview
Defining
Creating
Training
Customizing
Enabling Languages
Store
Smart Bots
Defining
koreUtil Libraries
SETTINGS
Authorization
Language Management
PII Settings
Variables
Functions
IVR Integration
General Settings
Management
Import & Export
Delete
Versioning
Collaborative Development
Plan Management
API GUIDE
API Overview
API List
API Collection
SDKs
SDK Overview
SDK Security
SDK App Registration
Web SDK Tutorial
Message Formatting and Templates
Mobile SDK Push Notification
Widget SDK Tutorial
Widget SDK – Message Formatting and Templates
Web Socket Connect & RTM
Using the BotKit SDK
Installing
Configuring
Events
Functions
BotKit SDK Tutorial – Agent Transfer
BotKit SDK Tutorial – Flight Search Sample Bot
Using an External NLP Engine
ADMINISTRATION
HOW TOs
Creating a Simple Bot
Creating a Banking Bot
Context Switching
Using Traits
Schedule a Smart Alert
Configure UI Forms
Add Form Data into Data Tables
Configuring Digital Views
Add Data to Data Tables
Update Data in Data Tables
Custom Dashboard
Custom Tags to filter Bot Metrics
Patterns for Intents & Entities
Build Knowledge Graph
Global Variables
Content Variables
Using Bot Functions
Configure Agent Transfer
Update Balance Task
Transfer Funds Task
RELEASE NOTES
  1. Home
  2. Docs
  3. Virtual Assistants
  4. Test your Bot
  5. Conversation Testing

Conversation Testing

Conversation Testing enables you to simulate end-to-end conversational flows to evaluate the dialog task or perform regression. You can create Test Cases to capture various business scenarios and run them at a later point in time to validate the performance of the bot. 

Test Cases

Test Cases consist of a sequence of Test Steps that capture the conversational flow. Each Test Step is a combination of Step Number, Test Input, Expected Output, Actual Output, and Test Result. 

Step# defines the sequence of execution. 

Test Input represents the user message part of the conversation. 

Expected Output represents the bot message that is anticipated from the bot.

Actual Output represents the response of the bot during test execution. The platform will evaluate the Actual Output against the Expected Output to determine if a test step is passed or not. 

Test Result indicates if the actual output is the same as the expected output. Test Result can be Pass, Failed, or Not Executed.

Test Case Creation 

New Test Cases can be created by recording the conversation while using ‘Talk to Bot’. Follow the following steps to create a new test case. 

Step 1: Launch ‘Talk to Bot’ by clicking on the icon in the bottom right corner of your screen. ‘Talk to Bot’ can be launched from any module on the platform. 

Step 2: Click on ‘Record Conversation’ to start recording the messages that are going to be part of the test case. 

Step 3: Interact with the bot by entering relevant user messages. 

Step 4: To create a new test case, click on ‘Stop Recording’ and then click on ‘Create Test Case’. 

Step 5: In the next screen, add the necessary details such as ‘Test Case Name’, ‘Description’ and ‘Tags’ and click on ‘Add’.

Step 6: Navigate to the Conversation Testing module to execute the added test case or add another test case by following the same steps. 

Tags

Tags help in classifying and organizing test cases. You can choose to add a tag at the time of the creation of a test case or from the ‘General Settings’ under the Test Case properties.

General Settings

 After a test case is created, you can access the ‘General Settings’ from the overflow menu. The General Settings has a few editable and few view-only properties.

You have the option to edit the following properties.

  •       Test Case Name
  •       Test Case Description
  •       Test Case Tags

Additionally, the following properties can be viewed but not edited.

  •       Date & Time of Test Case Creation
  •       Name of the User who created the Test Case
  •       Date & Time of the last update
  •       Name of the user who last updated the test case

Test Case Execution

The following steps will guide you on how to execute a test case and obtain a detailed analytical report based on the execution. 

Step 1: Select the Test Cases that you would like to execute.

Step 2: Click on ‘Run Test Case’ to start the execution. 

Step 3: Next, you are required to choose the Version of the Bot that you want to test. Based on the version selected, please ensure that you have valid authorization tokens for the bot to make the service calls that would be part of the test case. 

Step 4: You can monitor the progress of the execution from the last result column against the test case and the status docker. 

Step 5: Once the test case execution is over, you can click on the test case to view the execution history. 

Step 6: Click on any of the listed test runs to view the details of the test run. 

Test Cases are executed asynchronously i.e. even when multiple test cases are selected, the execution happens sequentially. You have the option to cancel the test execution before the execution is completed. 

Once the test case execution has started, the platform continues to execute the test case even when one or more test steps fail. However, if the test execution is stuck at a test step for more than 60s the platform will timeout and abort the execution. 

Interpreting Test Results

The following details are stored against every test execution. 

  • Date & Time of Execution completion.
  • Version Type to indicate if the test case was run against the In-development or the Published version of the bot. 
  • Execution Result to indicate the result of the execution. A test case is treated as failed if at least one of the test steps is failed. 
  • Execution Time indicates the total time duration of the execution. 
  • Test Step Details represent the total steps that were passed, failed, or not executed. 
  • Execution Error Details are shown only when the platform is unable to complete the test execution either because of the user aborting the test case or the test step timing out due to an error. 

Limitations

The following modules cannot be tested with the current version of conversation testing. 

  • Digital Forms
  • Dynamically generated bot messages where content or context variables are used.

Test Runner Tool

If you would like to set up your own test execution application, then you can download the libraries provided at the Kore.ai Github repository. You can record and download the test cases from the platform, and run these test cases on your application. Note that this tool is an open-source code that you can fork a branch and make changes as per your requirements.

Key Points

  • You can start and stop the recording anytime during your conversation with the bot.
  • The entire conversation is saved as a Test Case.
  • In case you want to split the Test Cases and generate a Test Suite, you can achieve it using “discard all” command during the conversation. The conversation before “discard all” is saved as a test case and the conversation after is another test case. Both the test cases will be saved as a single file as Test Suite. You can split the conversation into as many test cases as per your requirement.
  • The entire conversation is stored in a JSON file which will be available for download once the recording ends.

The Testing Tool will converse with the bot using the user input from the JSON file and compare the bot response with the recorded bot responses.

The results are recorded in a Test Results spreadsheet providing a comprehensive report of the test run, with the developer input, actual and expected results, and the status of the test run (passed or failed). You can use this spreadsheet to report, keep a track of, and re-test the failed test cases.

For more details on the Test Runner refer to the Kore.ai Github.

Menu