Managing Test Suites
Kore.ai provides few out-of-the box Test Suites to perform batch testing. ‘Developer defined utterances’ and ‘Successful user utterances’ are the built-in test suites that can be run to perform Batch Testing. You can also create a New Test Suite for testing custom set of utterances.
Developer defined utterances
This test suite validates the utterances that have been previously added and trained by the developer from Machine Learning Utterances screen. Using this test suite would mean testing collectively the entire set of utterances that a developer has added for all tasks of the bot.
Successful user utterances
This test suite includes all the end-user utterances that have successfully matched an intent and the corresponding task is fully executed. You can also find these utterances from ‘Intent found’ section of Analyze module.
Adding a New Test Suite
The New Test Suite enables you to import an array of test utterances collectively at once in a batch file, also known as a Dataset. The Dataset file needs to be written in CSV or JSON format and can have a maximum of 1000 utterances. You can download the sample CSV or JSON file formats as part of test suite creation using New Test Suite option.
JSON Format for Test Suite
JSON format for creating custom suites allows you to define an array of test cases where each test case should consist of an utterance to be tested, the intent against which the utterance to be tested and optionally define the list of expected entities to be determined from the utterance. If expected intent is a child intent, then you can also include the parent intent to be considered.
- For Entities that have the Multi-Item enabled, values need to be given as:
entity1||entity2
- Composite Entities require the values in the following format:
component1name:entityValue|component2name:entityValue2
Property Name | Type | Description |
---|---|---|
Test Cases | Array | Consists of the following:
|
input | String | Utterance given by the end-user |
intent | String | Determine the objective of an end-user utterance |
parentIntent | String [Optional] | Define parent intent to be considered if the intent is a sub-intent |
entities | Array [Optional] | Consists an array of entities to be determined from the input sentence:
|
entityValue | String | Value of the entity expected to be determined from the utterance. You can define the expected Entity Value as a string or use a Regular Expression. For the purpose of Batch Testing, platform flattens all entity values into string formats. Refer Entity Format Conversions for more information. |
entityName | String | Name of the entity expected to be determined from the utterance |
CSV Format for Test Suite
CSV format for creating custom suites allows you to define test cases as records in CSV file where each test case should consist of an utterance to be tested, the intent against which the utterance to be tested and optionally define entities to be determined from the utterance. If your test case requires more than one entity to detected from a sentence, then you have to include an extra row for each of the additional entities to be detected. If expected intent is a child intent, then you can also include the parent intent to be considered.
- For Entities that have the Multi-Item enabled values need to be given as:
entity1||entity2
- Composite Entities require the values in the following format:
component1name:entityValue|component2name:entityValue2
Column Name | Type | Type | Description |
---|---|---|---|
input | String | Utterance given by the end-user | |
intent | String | Determine the objective of an end-user utterance | |
parentIntent | String [Optional] | Define parent intent to be considered if the intent is a sub-intent | |
entityValue | String [Optional] | Value of the entity expected to be determined from the utterance. You can define the expected Entity Value as a string or use a Regular Expression. For the purpose of Batch Testing, platform flattens all entity values into string formats. Refer Entity Format Conversions for more information. | |
entityName | String [Optional] | Name of the entity expected to be determined from the utterance |
Entity Format Conversions
Entity Type | Sample Entity ValueType | Value in Flat Format | Order of Keys |
---|---|---|---|
Address | P.O. Box 3700 Eureka, CA 95502 | P.O. Box 3700 Eureka, CA 95502 | |
Airport | { “IATA”: “IAD”, “AirportName”: “Washington Dulles International Airport”, “City”: “Washington D.C.”, “CityLocal”: “Washington”, “ICAO”: “KIAD”, “Latitude”: “38.94”, “Longitude”: “-77.46” } | Washington Dulles International Airport IAD KIAD 38.94 -77.46 Washington D.C. Washington | AirportName IATA ICAO Latitude Longitude City CityLocal |
City | Washington | Washington | |
Country | { “alpha3”: “IND”, “alpha2”: “IN”, “localName”: “India”, “shortName”: “India”, “numericalCode”: 356} | IN IND 356 India India | alpha2 alpha3 numericalCode localName shortName |
Company or Organization Name | Kore.ai | Kore.ai | |
Color | Blue | Blue | |
Currency | [{ “code”: “USD”, “amount”: 10 }] | 10 USD | amount code |
Date | 2018-10-25 | 2018-10-25 | |
Date Period | { “fromDate”: “2018-11-01”, “toDate”: “2018-11-30” } | 2018-11-01 2018-11-30 | fromDate toDate |
Date Time | 2018-10-24T13:03:03+05:30 | 2018-10-24T13:03:03+05:30 | |
Description | Sample Description | Sample Description | |
user1@emaildomain.com | user1@emaildomain.com | ||
List of Items(Enumerated) | Apple | Apple | |
List of Items(Lookup) | Apple | Apple | |
Location | { “formatted_address”: “8529 Southpark Cir #100, Orlando, FL 32819, USA”, “lat”: 28.439148,”lng”: -81.423733 } | 8529 Southpark Cir #100, Orlando, FL 32819, USA 28.439148 -81.423733 | formatted_address lat lng |
Number | 100 | 100 | |
Person Name | Peter Pan | Peter Pan | |
Percentage | 0.25 | 0.25 | |
Phone Number | +914042528888 | +914042528888 | |
Quantity | { “unit”: “meter”, “amount”: 16093.4, “type”: “length”, “source”: “10 miles” } | 16093.4 meter length 10 miles | amount unit type source |
String | Sample String | Sample String | |
Time | T13:15:55+05:30 | T13:15:55+05:30 | |
Time Zone | -04:00 | -04:00 | |
URL | https://kore.ai | https://kore.ai | |
Zip Code | 32819 | 32819 |
Importing a Dataset file
-
- Click New Test Suite on the batch testing page. A dialog box to import dataset appears.
- Enter a Name, Description and choose a Dataset Type in the respective boxes for your dataset file.
- To import the Dataset file, click Choose File to locate and select a JSON or CSV file containing the utterances as per the Dataset Type selected.