NLP Metrics18 min read

Kore.ai records and presents all the information as part of the Analyze section. Developers can gain in-depth insights into their bot’s performance at identifying and executing tasks. It lets you view necessary information for user utterances that matched and didn’t match with intents.

The Analyze > NLP Metrics section contains the following sections:

• Intent Found: Contains all the user utterances that were successfully mapped to a trained intent, including the dialog tasks triggered by KG intents. The utterances are grouped together based on similarity
  • You can filter the information based on various criteria such as User Utterances, Intent, user (Kore user id or channel-specific unique id), date-period, channel of use, and language. Records can also be filtered on multiple custom tags.
  • Complete meta information is stored for later analysis including the original user utterance, the channel of communication, entities extracted if any, custom tags applied, detailed NLP analysis with scores returned from each engine, and the ranking and resolver scores.
  • Ability to view the chat transcript to the point of the user utterance. This also gives the option to view the user profile and the details for that user’s conversation sessions.
  • You have an option to train the utterance and once trained the utterance will be marked.
  • Any important record you want to mark and/or track later can be pinned. These will appear in the Pinned tab.

  

• Intent not found: Contains all the user utterances that the platform was not able to map to a bot intent/FAQ. These are grouped together based on similarity for the developer to train based on occurrence count.
  • You can filter information based on various criteria such as user (Kore user id or channel-specific unique id), date-period, channel of use, and language. Records can also be filtered on Multiple custom tags.
  • Complete meta information is stored for later analysis including the original user utterance, the channel of communication, system entities extracted if any, custom tags applied, detailed NLP analysis with scores returned from each engine, and the ranking and resolver scores.
  • Ability to view the chat transcript to the point of the user utterance. This also gives the option to view the user profile and the details for that user’s conversation sessions.
  • You have an option to train the utterance and once trained the utterance will be marked. You can also filter based on trained / untrained utterances.
  • Any important record you want to mark and/or track later can be pinned. These will appear in the Pinned tab.
• Failed Task: All the user utterances that were successfully identified to intent, but the task could not be completed are listed under this section. You can group based on task and failure types to analyze and solve issues with the bot.
  • The supported platform failure types are:
    • Task aborted by user
    • Alternate task initiated
    • Chat Interface refreshed
    • Human-agent transfer
    • Authorization attempt failure – Max attempts reached
    • Incorrect entity failure – Max attempts reached
    • Script failure
    • Service failure
  • You can filter information based on various criteria such as task name, user (Kore user id or channel-specific unique id), date-period, and language.
  • Complete meta information is stored for later analysis including the original user utterance, the channel of communication, system entities extracted if any, custom tags applied, detailed NLP analysis with scores returned from each engine, and the ranking and resolver scores.
  • Ability to view the chat transcript to the point of the user utterance. This also gives the option to view the user profile and the details for that user’s conversation sessions.
  • Any important record you want to mark and/or track later can be pinned. These will appear in the Pinned tab.
• Performance: Developers can monitor all the scripts and API services across the bot tasks from a single window. The platform stores the following meta-information:
  • Node name, type and task name
  • Total number of runs
  • Success %
  • The total number of calls with 200 responses and the total number of calls with a non-200 response. The actual response code can be viewed from the details page which opens when the service row is clicked.
  • Average Response times
  • Appropriate alerts if a script or a service is failing consecutively

   

• Pinned: You can view all the records that you have marked as important or pinned from the identified/unidentified intents and failed tasks in this tab.

To open the Metrics page, from the top menu select Analyze > NLP Metrics.

To facilitate an easier review of the Bot’s performance, the end-user utterances are grouped based on similarity.

Filter Criteria

You can filter the information on the Metrics page using the following criteria. You can save the entered filter criteria and set it as a default filter using the Save as Default Filter.

Identified and Unidentified Intents

The primary details, filter criteria, and the advanced details for both the Intent Found and Intent Not Found are similar, with minor differences. You can also train the bot for any utterances directly from these tabs.

Primary Details

Training the Bot

You can train an intent from both the Intent Found and Intent Not Found tabs. To do so, hover over a row in any of these tabs, and click the Train icon. It opens the Test & Train page from where you can train the bot. For more information, read Testing and Training a Bot.

Failed Tasks

The Failed Tasks tab shows the following details related to the task that was identified but failed to execute for any reason:

Pinned

Any records from Identified and Unidentified Intents or Failed Tasks tabs that you have pinned would be displayed here in the Pinned tab. The following details related to the task/intent are displayed:

Advanced View

For all the user utterances listed under the Intent found, Intent not found, Failed Task, and Pinned tabs, you can open advanced details related to the user session with the following sub-tabs:

• Details: This shows the basic details of the session along with a JSON file that includes the NLP analysis for the conversation.
• NLP Analysis: Provides a visual representation of the NLP analysis including intent scoring and selection. For more information, read Testing and Training a Bot.
• Chat history: Directs you to the exact message or conversation for which the record is logged and shows the entire chat history of the user session.

Chat History

Chat History provides visibility into the user information with the inclusion of the following functionality:

• User Profile: Provides a 360-degree view of the user along with their usage metrics.
• User Conversation Sessions: Lists out all the sessions of the user in the given time period with the selected utterance section expanded.
• Go to Selected Utterance: Selected utterance will be highlighted in orange.

Following are the user information details provided:

Performance

The Performance tab shows the following information related to the backend performance of the bot:

Advanced Performance Details

Clicking a service or script or WebHook name opens an advanced details dialog for the service which lists each instance of its run along with separate tabs for successful and failed runs. Analyzing the average response time of different runs gives you insights into any aberrations in the service or script execution. Click any row to open the JSON response associated with the service or script run.

Debug Log

Any custom debug statements that you entered in the Script node using koreDebugger.log("<debug statement>") statements are displayed in this tab (debug statement should be in a string format).

The logs include the user conversation from across all channels. You can use them for bot analysis especially in case of failures during user interaction.

The details include:

• The actual statement that you have defined at the time of Bot definition.
• Date and time of logging
• Channel
• User ID (along with channel-specific id)
• Language of interaction
• Task name if available
• Developer flag – to indicate if the interaction was performed by a developer or end-user.

You can also view the details of the chat history associated with the session. To view more details, follow the below steps:

1. Click on any log record.
2. On the corresponding window, you can find the Details and Chat History tabs.
3. Under the Details tab, you can find the task name, channel, language, and flow.
4. Click the Chat History tab. You can find the chat transcript where the log is recorded.

   • If the debug log is generated from a bot message, you are navigated to that specific message in the chat transcript.

   • If the debug log is not part of the bot message, you are navigated to the latest message added before the debug statement.

For universal bots, the debug statements from the universal and linked bots are included in the logs.

The debug logs also include the error messages related to BotKit, like, when the platform could not reach the BotKit or when the BotKit did not acknowledge the message sent by the platform. The message includes details like the <endpoint>, <error code>, and <response time>.

Storage Limitations

The platform imposes restrictions on the number of log statements retained per bot. The limit is a combination of volume and period:

• Only the latest 700 statements per bot are stored.
• Statements older than 7 days are removed.

Exporting the Data

You can export the data present on the Bot Analyze page to a CSV file by clicking the Export icon on the top right corner of the page.

Once you click the icon, the export process starts and you can use the Status Tracker dock to track the export progress. After completion of the export, the dock shows the export status and if it’s successful provides a link to download the file.

The download includes the information present on the selected tab as well as the detailed analysis based on the selected filters.

These records will also include the Meta Tag information.