Once you have built and trained your bot, the most important question that arises is how good is your bot’s learning model? So, evaluating your bot’s performance is important to delineate how good your bot understands the user utterances.

The Batch Testing feature helps you discern the ability of your bot to correctly recognize the expected intents and entities from a given set of utterances. This involves the execution of a series of tests to get a detailed statistical analysis and gauge the performance of your bot’s ML model.

Note: The Batch Testing dashboard displays the summary of test coverage, performance, and training recommendations for multiple test suites based on validations of the intent identification capabilities of your NLU model. The Health and Monitoring dashboard is a one-stop place that displays this information for the individual test suite(s) the user selects. Learn More.

To conduct a batch test, you can use predefined test suites available in the builder or create your own custom test suites. Based on your requirement, the test suites can be run to view the desired results. This option can be accessed from the Testing -> Batch Testing option from the left navigation menu.

Best Practices

• An optimal approach to bot NLP training is to first create a test suite of most of the use cases(user utterances) that the bot needs to identify, run it against the model and start training for the ones that failed.
• Create/update batch testing modules for high usage utterances.
• Publish the trained model only after detailed testing.
• When naming the intent, ensure that the name is relatively short (3-5 words) and does not have special characters or words from the Stop Wordlist. Try to ensure the intent name is close to what the users request in their utterance.
• Batch Test executions do not consider the context of the user. Hence you might see some False Negatives in the test results which in fact are True Positives in the actual bot when the context is taken into consideration.
• The ‘count’ in Batch Test results refers to the ‘unique assertion’ statements and not necessarily the number of the rows in the CSV file. Batch tests use a rule to validate the unique assertions based on the ‘utterance’ in consecutive rows. If two consecutive rows have same utterances and different entity values, the XO platform considers it as one assertion statement which accepts both the entity values.

Managing Test Suites

Kore.ai provides a few out-of-the-box Test Suites to perform batch testing. ‘Developer defined utterances’ and ‘Successful user utterances’ are the built-in test suites that can be run to perform Batch Testing. You can also create a New Test Suite for testing a custom set of utterances.

Developer defined utterances

This test suite validates the utterances that have been previously added and trained by the developer from Machine Learning Utterances screen. Using this test suite would mean testing collectively the entire set of utterances that a developer has added for all tasks of the bot.

Successful user utterances

This test suite includes all the end-user utterances that have successfully matched one or more intents and the corresponding task is fully executed. You can also find these utterances from the ‘Intent found’ section of the Analyze module.

Adding a New Test Suite

To add a new test suite, follow the steps below:

1. 1. Go to Build > Testing > Batch Testing, and click New Test Suite.

1. In the New test Suite window, add a name and provide a description. 
2. Next, choose how to add test cases: either manually or by uploading a test case file.

• Selecting Add Manually lets you add test cases manually or use LLM and Generative AI to generate test cases automatically. 
• Selecting Upload Test Cases File lets you upload a file containing test cases.
  

Adding Test Cases Manually

Once you create a test suite to which you have chosen to add test cases manually, you can either add them manually or generate them automatically.

In the Test Cases page, click the +Add Test Case.

In the Add Test Case window, select or enter information in the following fields:

1. The Intent that corresponds to the Dialog Task you want to test. You can add/tag one or more intents to a test utterance.
   You can add up to three intents (Dialog, FAQ, or Small Talk) for a test utterance. It helps in scenarios where ambiguity is by-design and should be considered as True Positive. For example, for the utterance “I need help to book a flight ticket,” the expected intent can be either “Book flight” (Dialog) or “How to book a flight?” (FAQ).
   
2. The Parent Intent within that task.
3. Test Utterances: You can add multiple utterances, each on a new line.
4. The Entity Order: Select one from the available options, depending on the intent you are working with. You can only select one item. The Entity Order selection is not available if you have added multiple intents.
   

   One Intent for a Test Utterance

   Multiple Intents for a Test Utterance

5. Click Save when ready.
6. Once added, your test cases are listed and ready to run. You can use the Add Manually button at the top of the list to add more test cases. The option to Generate Test cases only displays if you have enabled LLM and Generative AI for the VA.
   

Generating Test Cases Automatically

You can automatically generate batch test cases if you have enabled LLM and Generative AI for your Virtual Assistant. You can access the feature either right after creating a new test case, as shown above, or from the test cases list.

Steps to Generate Test Cases

To generate test cases automatically, follow the steps below:

1. Click Generate Test Cases from any of the two areas mentioned above.
2. In the Generate Test Case window, select the Dialog Task you want to test. 
3. Click Generate. Wait for a few moments until the generation completes.
   
4. Once test cases are generated, you can reject some of them. If required, click Generate more to get more suggestions. 
5. Click Add Test Cases when ready.
   
   

Once you add the test cases, they are listed within your test suite, along with any other cases you might have added manually or generated in the past. You can continue adding test cases to the suite using these two methods. 

Adding a Test Suite by Uploading a Test Cases File

If you Upload a Test Cases File, you can import an array of test utterances, also known as a Dataset at once in a batch file. The Dataset file must in a CSV or JSON format and can have a maximum of 1000 utterances. You can download the sample CSV or JSON file formats while creating the test suite.

{
“input”: “Utterance”,
“intent”: “Intent 1|Intent 2|Intent 3”,
“botName”: “Bot 1|Bot 2|Bot 3”
}

{
“input”: “Need to check the multibot function”,
“intent”: “Age Predictor|IVRnode|Check Rephase”,
“botName”: “Check Bot|XOP211|Check Rephase”
}

The ability to create test suites manually is not yet available for universal bots.

JSON Format for Test Suite

The JSON format for creating custom suites allows you to define an array of test cases where each test case should consist of an utterance to be tested, the intent against which the utterance to be tested, and optionally define the list of expected entities to be determined from the utterance. If expected intent is a child intent, then you can also include the parent intent to be considered.

The sample code is given below:

{
    "testCases": [
    {
      "input": "Send 200 dollars to Leonardo",
      "intent": "Transfer Funds",
      "entities": [
        {
          "entityValue": "200 USD",
          "entityName": "TransferAmount"
        },
        {
          "entityValue": "Leonardo",
          "entityName": "PayeeName"
        }
      ],
      "entityOrder":["TransferAmount", "PayeeName"]
    },
    {
      "input": "What is the balance in my checking account",
      "intent": "Show Balance",
      "parentIntent": "Transfer Funds"
    },
    {
      "input": "Repeat this transfer every month",
      "intent": "Setup Auto Pay",
      "parentIntent": "Transfer Funds"
    },
    {
      "input": "Show my past 20 transactions",
      "intent": "Show Account Statement",
      "entities": [
        {
          "entityValue": "20",
          "entityName": "HistorySize"
        }
      ]
    },
    {
      "input": "Pay my credit card dues",
      "intent": "Pay Bill"
    },
	{
      "input": "I need to pay my monthly credit card bill",
      "intent": "Pay Bill | Setup Auto Pay | Transfer Funds"
    },
	{
	
      "input": "Open an account for me",
      "intent": "Open Account | How to Open Account?"
    },
    {
      "input": "looks like something is wrong in my statement",
      "intent":"trait: Account Statement||Issue||Account Type"
    },
    {
      "input": "How are you doing today",
      "intent":"emohowzit"
    },
    {
      "input": "Thank you",
      "intent":"thank~you"
    },
    {
      "input": "I understand",
      "intent":"i_understand"
    },
    {
      "input": "That's great",
      "intent":"~positiveackwords"
    },
    {
      "input": "How many people can you speak to at once?",
      "intent":"How many people can you speak to at once?"
    },
    {
      "input": "That's sweet",
      "intent":"That's sweet",
      "parentIntent": "How many people can you speak to at once?"
    },
    {
      "input": "What else can you do?",
      "intent":" What else can you do?",
      "parentIntent": "How many people can you speak to at once? >> That's sweet "
    },
    {
      "input": "What else can you do?",
      "intent":"What else"
    },
    {
      "input": "What's your age?",
      "intent":"How old are you?"
    },
    {
      "input": "Who created you?",
      "intent":"Who made you?"
    },
    {
      "input": "How many people talk to you?",
      "intent":"How many people can you speak to at once?"
    },
    {
      "input": "What's your name?",
      "intent":"What is your name?"
    },
    {
      "input": "What are you called?",
      "intent":"What ~you called?"
    }
  ]
}

• For Entities that have the Multi-Item enabled, values need to be given as: entity1||entity2
• Composite Entities require passing values in the following format: component1name:entityValue|component2name:entityValue2The sample code is given below:

  {
    "testCases": [
      {
        "input": "Add cart to Apples , Grapes",
        "intent": "Add to cart",
        "entities": [
          {
            "entityValue": "Apples||Grapes",
            "entityName": "LOV"
          }
         ]
       },
        {
           "input": "I booked a ticket from Hyderabad on July 6th, 2018 and it costed me Rs. 1200",
           "intent": "Book a flight",
           “entities”: [
             {
              "entityValue": "City:Hyderabad|Date:2018-07-06}Curr:1200 INR",
              "entityName": "Composite5",      
             }
          ]
        }
     }
  

• The order in which the entities are to be extracted can be given as: "entityOrder":["TransferAmount", "PayeeName"]. If the order is not provided or partially provided, the platform determines the shortest route covering all the entities as the default order.

CSV Format for Test Suite

CSV format for creating custom suites allows you to define test cases as records in CSV file where each test case should consist of an utterance to be tested, the intent against which the utterance to be tested, and optionally define entities to be determined from the utterance. If your test case requires more than one entity to be detected from a sentence, then you have to include an extra row for each of the additional entities to be detected. If expected intent is a child intent, then you can also include the parent intent to be considered.

input,intent,parentIntent,entityName,entityValue,entityOrder
Send 200 dollars to Leonardo, Transfer Funds,,TransferAmount,200 USD,,
,,,PayeeName,Leonardo, TransferAmount>PayeeName
What is the balance in my checking account,Show Balance, Transfer Funds,,,
Repeat this transfer every month, Setup Auto Pay, Transfer Funds,,,
Show my past 20 transactions, Show Account Statement,,HistorySize,20,,
Pay my credit card dues,Pay Bill,,,,,

• For Entities that have the Multi-Item enabled values need to be given as: entity1||entity2
• Composite Entities require the values in the following format:component1name:entityValue|component2name:entityValue2

  input,intent,parentIntent,entityName,entityValue
  “Add cart to Apples , Grapes”,Add to cart,LOV,Apples||Grapes,,
  “I booked a ticket from hyderabad on July 6th, 2018 and it cost me Rs. 1200”,Composite Test,Composite55,City5:Hyderabad|Date5:2018-07-06”|Curr5:1200 INR
  

• The order of extraction of entity value can be mentioned in the following format: entity3>entity4>entity1. If the order is not provided or partially provided, the platform determines the shortest route covering all the entities as the default order.

Entity Format Conversions

Importing a Dataset file

1. Click New Test Suite on the batch testing page. A dialog box to import the dataset appears.
2. Enter a Name, Description, and choose a Dataset Type in the respective boxes for your dataset file.
3. To import the Dataset file, click Choose File to locate and select a JSON or CSV file containing the utterances as per the Dataset Type selected.
   
   
4. Click Create. The dataset file is displayed as an option to run the test suite on the Batch Testing page:

Running Test Suites

The following steps guide you on how to run a batch test on your bot and get a detailed analytical report on the utterances based on the test results. To get started, go to Build > Testing > Batch Testing.

To run a Test Suite, follow the steps below:

1. Click the desired test suite name in the Batch Testing window. 
   Note that Developer Defined Utterances and Successful User Utterances are default test suites to validate user utterances through Batch Testing.
2. Select In Development to run batch tests on test suites for only the in-development version of the Virtual Assistant(s). Alternatively, select Published to run batch tests on test suites for only the published version of the Virtual Assistant(s).
3. Click Run Test Suite to initiate the batch test execution.
   

Execution of Newly Created Batch Test Suites

Once you create a new test batch test suite, by default, the system automatically initiates test run execution for both In-development and Published VA versions. This makes it convenient to assess test results while the VA is in development and once a VA is published following a production release.