2. Home
  4. Kore.ai Conversational Platform
  6. Bot Builder Tool
  8. Test Your Bot
  10. Batch Testing
  12. Managing Test Suites

Managing Test Suites

Kore.ai provides few out-of-the box Test Suites to perform batch testing. ‘Developer defined utterances’ and ‘Successful user utterances’ are the built-in test suites that can be run to perform Batch Testing. You can also create a New Test Suite for testing custom set of utterances.

 

Developer defined utterances

This test suite validates the utterances that have been previously added and trained by the developer from Machine Learning Utterances screen. Using this test suite would mean testing collectively the entire set of utterances that a developer has added for all tasks of the bot.

 

Successful user utterances

This test suite includes all the end-user utterances that have successfully matched an intent and the corresponding task is fully executed. You can also find these utterances from ‘Intent found’ section of Analyze module.

Adding a New Test Suite

The New Test Suite enables you to import an array of test utterances collectively at once in a batch file, also known as a Dataset. The Dataset file needs to be written in CSV or JSON format and can have a maximum of 1000 utterances. You can download the sample CSV or JSON file formats as part of test suite creation using New Test Suite option.

 

JSON Format for Test Suite
JSON format for creating custom suites allows you to define an array of test cases where each test case should consist of an utterance to be tested, the intent against which the utterance to be tested and optionally define the list of expected entities to be determined from the utterance. If expected intent is a child intent, then you can also include the parent intent to be considered.

  • For Entities that have the Multi-Item enabled, values need to be given as: entity1||entity2
  • Composite Entities require the values in the following format: component1name:entityValue|component2name:entityValue2

Property Name Type Description
Test Cases Array Consists of the following:

  • input
  • intent
  • parentIntent
input String Utterance given by the end-user
intent String Determine the objective of an end-user utterance
parentIntent String [Optional] Define parent intent to be considered if the intent is a sub-intent
entities Array [Optional] Consists an array of entities to be determined from the input sentence:

  • entityValue
  • entityName
entityValue String Value of the entity expected to be determined from the utterance. You can define the expected Entity Value as a string or use a Regular Expression. For the purpose of Batch Testing, platform flattens all entity values into string formats. Refer Entity Format Conversions for more information.
entityName String Name of the entity expected to be determined from the utterance

 

CSV Format for Test Suite
CSV format for creating custom suites allows you to define test cases as records in CSV file where each test case should consist of an utterance to be tested, the intent against which the utterance to be tested and optionally define entities to be determined from the utterance. If your test case requires more than one entity to detected from a sentence, then you have to include an extra row for each of the additional entities to be detected. If expected intent is a child intent, then you can also include the parent intent to be considered.


 

  • For Entities that have the Multi-Item enabled values need to be given as: entity1||entity2
  • Composite Entities require the values in the following format: component1name:entityValue|component2name:entityValue2

Column Name Type Type Description
input String Utterance given by the end-user
intent String Determine the objective of an end-user utterance
parentIntent String [Optional] Define parent intent to be considered if the intent is a sub-intent
entityValue String [Optional] Value of the entity expected to be determined from the utterance. You can define the expected Entity Value as a string or use a Regular Expression. For the purpose of Batch Testing, platform flattens all entity values into string formats. Refer Entity Format Conversions for more information.
entityName String [Optional] Name of the entity expected to be determined from the utterance

Entity Format Conversions

Entity Type Sample Entity ValueType Value in Flat Format Order of Keys
Address P.O. Box 3700 Eureka, CA 95502 P.O. Box 3700 Eureka, CA 95502
Airport { “IATA”: “IAD”, “AirportName”: “Washington Dulles International Airport”, “City”: “Washington D.C.”, “CityLocal”: “Washington”, “ICAO”: “KIAD”, “Latitude”: “38.94”, “Longitude”: “-77.46” } Washington Dulles International Airport IAD KIAD 38.94 -77.46 Washington D.C. Washington AirportName IATA ICAO Latitude Longitude City CityLocal
City Washington Washington
Country { “alpha3”: “IND”, “alpha2”: “IN”, “localName”: “India”, “shortName”: “India”, “numericalCode”: 356} IN IND 356 India India alpha2 alpha3 numericalCode localName shortName
Company or Organization Name Kore.ai Kore.ai
Color Blue Blue
Currency [{ “code”: “USD”, “amount”: 10 }] 10 USD amount code
Date 2018-10-25 2018-10-25
Date Period { “fromDate”: “2018-11-01”, “toDate”: “2018-11-30” } 2018-11-01 2018-11-30 fromDate toDate
Date Time 2018-10-24T13:03:03+05:30 2018-10-24T13:03:03+05:30
Description Sample Description Sample Description
Email user1@emaildomain.com user1@emaildomain.com
List of Items(Enumerated) Apple Apple
List of Items(Lookup) Apple Apple
Location { “formatted_address”: “8529 Southpark Cir #100, Orlando, FL 32819, USA”, “lat”: 28.439148,”lng”: -81.423733 } 8529 Southpark Cir #100, Orlando, FL 32819, USA 28.439148 -81.423733 formatted_address lat lng
Number 100 100
Person Name Peter Pan Peter Pan
Percentage 0.25 0.25
Phone Number +914042528888 +914042528888
Quantity { “unit”: “meter”, “amount”: 16093.4, “type”: “length”, “source”: “10 miles” } 16093.4 meter length 10 miles amount unit type source
String Sample String Sample String
Time T13:15:55+05:30 T13:15:55+05:30
Time Zone -04:00 -04:00
URL https://kore.ai https://kore.ai
Zip Code 32819 32819

 

Importing a Dataset file

    1. Click New Test Suite on the batch testing page. A dialog box to import dataset appears.
    2. Enter a Name, Description and choose a Dataset Type in the respective boxes for your dataset file.
    3. To import the Dataset file, click Choose File to locate and select a JSON or CSV file containing the utterances as per the Dataset Type selected.



  1. Click Create. The dataset file will appear as an option to run the test suite on the batch testing page:
Was this article helpful to you? Yes 1 No 1