{"id":17928,"date":"2016-10-28T17:08:25","date_gmt":"2016-10-28T17:08:25","guid":{"rendered":"https:\/\/developer.kore.com\/?post_type=docs&#038;p=8558"},"modified":"2021-03-26T09:01:58","modified_gmt":"2021-03-26T09:01:58","slug":"user-utterances","status":"publish","type":"post","link":"https:\/\/developer.kore.ai\/v7-3\/docs\/bots\/nlp\/user-utterances\/","title":{"rendered":"Machine Learning"},"content":{"rendered":"<section class=\"l-section wpb_row height_auto\"><div class=\"l-section-h i-cf\"><div class=\"g-cols vc_row type_default valign_top\"><div class=\"vc_col-sm-12 wpb_column vc_column_container\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<p>Developers need to provide sample utterances for each intent (task) the bot needs to identify to train the machine learning model. The platform ML engine will build a model that will try to map a user utterance to one of the bot intents.<\/p>\n<p>Kore.ai\u2019s Bots Platform allows fully unsupervised machine learning to constantly expand the language capabilities of your chatbot \u2013 without human intervention. Unlike unsupervised models in which chatbots learn from any input \u2013 good or bad \u2013 the Kore.ai Bots Platform enables chatbots to automatically increase their vocabulary only when the chatbot successfully recognizes the intent and extracts the entities of a human\u2019s request to complete a task.<\/p>\n<p>However, we recommend keeping Supervised learning enabled to monitor the bot performance and manually tune where required. Using the bots platform, developers can evaluate all interaction logs, easily change NL settings for failed scenarios, and use the learnings to retrain the bot for better conversations.<\/p>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_small with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div>\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<h3 id=\"add-utterance\"><span class=\"ez-toc-section\" id=\"Adding_Machine_Learning_Utterances\">Adding Machine Learning Utterances<\/span><\/h3>\n<ol>\n<li>Open the bot for which you want to add sample user utterances.<\/li>\n<li>Hover over the side navigation panel and then click\u00a0<b>Natural Language -&gt;\u00a0Training<\/b>.<\/li>\n<li>Select the <b>Machine Learning Utterances\u00a0<\/b>tab.<\/li>\n<\/ol>\n<p><a ref=\"magnificPopup\" href=\"\/wp-content\/uploads\/sites\/15\/Machine-Learning-Utterances.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-14578 size-full\" src=\"\/wp-content\/uploads\/sites\/15\/Machine-Learning-Utterances.png\" alt=\"\" width=\"992\" height=\"537\" \/><\/a><\/p>\n<p>The negation of trained intents will be ignored by the platform.<br \/>\nFor example, consider a Banking Bot with trained utterance &#8211; Funds Transfer. Then a user utterance &#8220;My account is debited even without doing <span style=\"text-decoration: underline\">funds transfer<\/span>&#8221; will not trigger the &#8220;funds transfer&#8221; task.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Named_Entity_Recognition\">Named Entity Recognition<\/span><\/h3>\n<p>Apart from the intent, you can train your Bot to recognize the entities, if present, in the user utterance. For example, if the user says &#8220;Book Flight from Hyderabad to Mumbai&#8221; apart from recognizing the intent as &#8220;Book Flight&#8221; the source and destination of the flight should also be recognized. This can be achieved by marking the entities in the user utterance during training.<\/p>\n<p>You can mark entities in your utterances, by selecting the entity value and clicking the corresponding entity from the drop-down list.<br \/>\n<a ref=\"magnificPopup\" href=\"\/wp-content\/uploads\/sites\/15\/Intent-Patterns-entity-suggest1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-16912\" src=\"\/wp-content\/uploads\/sites\/15\/Intent-Patterns-entity-suggest1.png\" alt=\"\" width=\"820\" height=\"461\" \/><\/a><\/p>\n<p>The platform will also try to identify and mark the entities, you have the option to accept or discard these suggestions. The platform will identify the entities based upon:<\/p>\n<ul>\n<li>System entities;<\/li>\n<li>Static List of items either enumerated or lookup;<\/li>\n<li>NER trained entities (from above).<\/li>\n<\/ul>\n<p><a ref=\"magnificPopup\" href=\"\/wp-content\/uploads\/sites\/15\/Intent-Patterns-entity-suggest.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-16672\" src=\"\/wp-content\/uploads\/sites\/15\/Intent-Patterns-entity-suggest.png\" alt=\"\" width=\"820\" height=\"461\" \/><\/a><\/p>\n<p>Further, if you have enabled Entity Placeholders the platform will replace the entity values in the training utterance with entity name placeholders for training the ML model. Using actual entity values as well as multiple additions of an utterance with just change in the entity value will have an adverse impact on the ML training model. Name of entities also starts contributing highly to the intent detection model.<\/p>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_small with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div>\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<h3 id=\"train\"><span class=\"ez-toc-section\" id=\"Training_your_Bot\">Training your Bot<\/span><\/h3>\n<p>After you add user utterances, you should train the Kore.ai interpreter to recognize the utterances and the associated user intent. When you have untrained utterances in your bot, the following message is displayed:<\/p>\n<p><em>&#8220;You have untrained utterances in your machine learning model. Unless the engine is trained the bot will not identify intents and entities based on the un-trained utterances. Click on \u2018Train\u2019 button to update the bot with all your utterances.&#8221;<\/em><\/p>\n<p>In the <strong>User Says<\/strong> section, click <strong>Train<\/strong>. A status bar is displayed to show progress for utterance training. When complete, the\u00a0<em>Utterances trained successfully<\/em> message is displayed. The user utterances are added to the Machine Learning database.<br \/>\n<a href=\"\/v7-3\/docs\/bots\/bot-builder\/talk-to-your-bot\/testing-your-bot-with-nlp\/\">Learn how to test your bot<\/a>.<\/p>\n<h4>Auto-Train<\/h4>\n<p>By default, machine learning is automatically trained for any defined user utterances whenever a task is:<\/p>\n<ul>\n<li>changed from a status of <strong>In-Progress<\/strong> to <strong>Configured<\/strong>.<\/li>\n<li>updated with a new\n<ul>\n<li>task name or intent name,<\/li>\n<li>entity name or parameter name,<\/li>\n<li>entity type,<\/li>\n<li>bot name.<\/li>\n<\/ul>\n<\/li>\n<li>published.<\/li>\n<li>suspended by the Bots Admin.<\/li>\n<li>deleted by the Bots Admin.<\/li>\n<\/ul>\n<p>In Bot Builder when auto-train is in progress, a warning message that <em>untrained user utterances cannot be identified<\/em> is displayed if you try to test the bot before auto-train is complete.<\/p>\n<p>You can set the Auto Train option as follows:<\/p>\n<ol>\n<li>Open the bot for which you want to modify the settings.<\/li>\n<li>Hover over the side navigation panel and then click\u00a0<b>Natural Language\u00a0-&gt;\u00a0Training<\/b>.<\/li>\n<li>Select the\u00a0<b>Advanced Settings\u00a0<\/b>tab.<\/li>\n<li>Enable or Disable the <strong>Auto Training<\/strong> option as per your requirements.<\/li>\n<\/ol>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_small with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div>\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<h4>Negative Patterns<\/h4>\n<p>Negative patterns can be used to eliminate intents detected by the Fundamental Meaning or Machine Learning models. <a href=\"\/v7-3\/docs\/bots\/bot-builder-tool\/train-nlp-optimization\/fundamental-meaning\/fundamental-meaning\/#Negative_Patterns\">Refer here to know more<\/a>.<\/p>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_small with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div>\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<h3 id=\"config\"><span class=\"ez-toc-section\" id=\"Threshold_Configurations\">Threshold &amp; Configurations<\/span><\/h3>\n<p>To train and improve the performance Threshold and Configurations can be specified for all three NLP engines &#8211; FM, KG and ML. You can access these settings under <strong>Natural Language &gt; Training &gt; Thresholds &amp; Configurations.<\/strong><\/p>\n<p><strong>NOTE<\/strong>: If your Bot is multilingual, you can set the Thresholds differently for different languages. If not set, the Default Settings will be used for all languages.<\/p>\n<p>The settings for ML engine is discussed in detail in the following sections.<\/p>\n<h4>Machine Learning<\/h4>\n<p>The Bots Platform ver 6.3 upgraded its Machine Learning (ML) model to v3. This includes a host of improvements and also allows developers to fine tune the model using parameters to suit business requirement. The developers can change parameters like stopword usage, synonym usage, thresholds, and n-grams, as well as opt between <a href=\"https:\/\/en.wikipedia.org\/wiki\/Deep_learning#Deep_neural_networks\">Deep Neural Network<\/a> or <a href=\"https:\/\/en.wikipedia.org\/wiki\/Conditional_random_field\">Conditional Random Field-based<\/a> algorithm for the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Named-entity_recognition\">Named-Entity Recognition (NER) <\/a>model.<\/p>\n<h4 id=\"ml-parameters\">Configuring the Machine Learning Parameters<\/h4>\n<p>The Bots Platform provides language-wise defaults for the following parameters related to the ML performance of your bot. You can customize them to suit your particular needs.<\/p>\n<p><a ref=\"magnificPopup\" href=\"\/wp-content\/uploads\/sites\/15\/thresholds-ML.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-16594\" src=\"\/wp-content\/uploads\/sites\/15\/thresholds-ML.png\" alt=\"\" width=\"820\" height=\"461\" \/><\/a><\/p>\n<h5>Bot Synonyms<\/h5>\n<p>This setting is <b>Disabled <\/b>by default. Enable this option if you would like to consider intent synonyms in building the ML model.<\/p>\n<h5>Stop Words<\/h5>\n<p>This setting is <b>Disabled <\/b>by default. Enable this option if you would like to remove the stop words in the training utterances in building the ML model.<\/p>\n<h5>ML Definitive Score<\/h5>\n<p>Configure the threshold score for definite matches, can be set to a value between 80-100%.<\/p>\n<h5>n-gram Sequence Length<\/h5>\n<p>n-gram is the contiguous sequence of words to be used from training sentences to train the model. For example, if <i>Generate sales forecast report <\/i>is the user utterance and if n-gram is set to 2, then <i>Generate sales<\/i>, <i>Sales forecast, <\/i>and <i>Forecast report<\/i> are used in training the model.<br \/>\nThe minimum n-gram limit is 1 by default. You can set the maximum limit up to 4.<\/p>\n<h5>ML Threshold<\/h5>\n<p>Define the minimum ML score to qualify an intent as a probable match. Learn more about ML Scoring.<\/p>\n<h5>NER Model<\/h5>\n<p>Choose the NER model to be used for entity detection.<\/p>\n<div class=\"alert alert-info\"><b>Note<\/b>: The <b>CRF <\/b>model supports all languages and the<b> Deep Neural Network <\/b>model supports English, Spanish, German, and French. This option appears on the screen only when the selected bot language is supported by the Deep Neural Network model.<\/div>\n<div><\/div>\n<h5>Entity Placeholders<\/h5>\n<p>Enable to replace entity values present in the training utterances with the corresponding entity placeholders in the training model.<\/p>\n<h5 id=\"ML-upgrade\">Upgrading the ML Model<\/h5>\n<p>All new bots that are created use the new ML model by default. Developers can upgrade the ML model for old bots or downgrade the model for the bots using the new model.<br \/>\nIf you are using a previous model of ML in the bots platform, you can upgrade it as follows:<\/p>\n<ol>\n<li>Open the bot for which you want to upgrade the ML model and go to<b> Natural Language &gt; Advanced Settings<\/b>.<\/li>\n<li>Expand<b> Machine Learning<\/b>. Under ML Upgrade section, click the <b>Upgrade Now<\/b> button. It opens a confirmation window.<a ref=\"magnificPopup\" href=\"\/wp-content\/uploads\/sites\/15\/Uprade-now-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-17251\" src=\"\/wp-content\/uploads\/sites\/15\/Uprade-now-1.png\" alt=\"\" width=\"820\" height=\"461\" \/><\/a><\/li>\n<li>Click <b>Upgrade and Train<\/b>. You can see new customizable options under the Machine Learning section.<br \/>\n<img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-15479\" src=\"\/wp-content\/uploads\/sites\/15\/Upgrade.png\" alt=\"\" width=\"362\" height=\"210\" \/><\/li>\n<\/ol>\n<p>You can also downgrade the ML model for new or upgraded bots from here by clicking <b>Switch to older version<\/b>. However, note that the older version of the ML model will be deprecated soon. So, we strongly recommend staying on the latest version to receive continued support and future enhancements.<\/p>\n<p><b>Note:<\/b> If a bot is exported using the older model (V2) and imported as a new bot, it continues to be in V2 model until you upgrade it.<\/p>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_medium with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div>\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<h3 id=\"import-export\"><span class=\"ez-toc-section\" id=\"Exporting_and_Importing_Machine_Learning_Utterances\">Exporting and Importing Machine Learning Utterances<\/span><\/h3>\n<p>You can import and export ML utterances of a bot into another in CSV and JSON formats. You can choose between \u2018In-Development\u2019 or \u2018Published\u2019 tasks to export, whereas importing utterances always replace the latest copy of the task in the bot.<\/p>\n<h4>How to Export or Import ML Utterances<\/h4>\n<ol>\n<li>On the bot\u2019s menu, click <b>Natural Language &gt; Machine Learning Utterances.<\/b><\/li>\n<li>The \u2018In-Development\u2019 version of the bot\u2019s ML utterances open by default. If you want to see the utterances in the \u2018Published\u2019 version, toggle on the top right side of the window to <b>Published<\/b>.\n<div class=\"alert alert-info\"><b>Note<\/b>: The export of ML utterances vary based on this selection as explained in the Versioning and Behavior of the Exported UUtterances section below.<\/div>\n<\/li>\n<li>Click the options icon and select an option:\n<ul>\n<li>Click <b>Import <\/b>and upload a CSV or JSON file with the utterances to import. Read <a href=\"#viu\">Versioning and Behavior of the Imported Utterances\u00a0<\/a>section below for more information.-OR-<\/li>\n<li>Click <b>Export <\/b>and select CSV or JSON formats to export the utterances. Read <a href=\"#veu\">Versioning and Behavior or the Exported Utterances\u00a0<\/a>section below for more information.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h4 id=\"viu\">Versioning and Behavior of Imported Utterances<\/h4>\n<ul>\n<li>The imported utterances in CSV\/JSON entirely replace the utterances present in the <i>latest copy<\/i> of the tasks.<\/li>\n<li>If the task is in the Configured status, the utterances in the task get entirely replaced with the new utterances for the task present in the imported file.<\/li>\n<li>If the task is in <i>Upgrade in Progress <\/i>status, the utterances related to the task get entirely replaced with the task utterances present in the imported file. The utterances in the <i>Published <\/i>copy of the task aren\u2019t affected.<\/li>\n<li>If the task is in the <i>Published <\/i>status, an <i>Upgrade in Progress<\/i> copy of the task gets created by default and the new utterances present in the imported file will be added to the upgraded copy. The utterances in the Published copy of the task aren\u2019t affected.<\/li>\n<\/ul>\n<h4 id=\"veu\">Versioning and Behavior of Exported Utterances<\/h4>\n<ul>\n<li>When you export a bot\u2019s utterances, all the utterances related to every task type &#8211; alert, action, information, dialog &#8211; get exported.<\/li>\n<li>When you export an <i>In Development <\/i>copy of the bot, the utterances of all tasks in the latest available copy get exported.<\/li>\n<li>When you export a <i>Published <\/i>copy of the bot, all the utterances in the published state get exported.<\/li>\n<li>In the case of multi-language bots, the export of utterances only happens for the selected bot language.<\/li>\n<li>Export of utterances to JSON include NER tagging present in the tasks, whereas CSV export doesn\u2019t include them.<\/li>\n<\/ul>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_small with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div>\r\n\t<div class=\"wpb_text_column \">\r\n\t\t<div class=\"wpb_wrapper\">\r\n\t\t\t<h3 id=\"recommendations\"><span class=\"ez-toc-section\" id=\"ML_Training_Recommendations\">ML Training Recommendations<\/span><\/h3>\n<ul>\n<li><em>Give a balanced training<\/em>\u00a0for all the intents that the bot needs to detect, add approximately the same number of sample utterances. A skewed model may result in skewed results.<\/li>\n<li>Provide <em>at least 8-10 sample utterances<\/em> against each intent. The model with just 1-2 utterances will not yield any machine learning benefits. Ensure that the utterances\u00a0are varied and you do not provide variations that use the same words in a different order.<\/li>\n<li>Avoid training <em>common phrases<\/em> that could be applied to every intent, for example, &#8220;I want to&#8221;. Ensure that the utterances are varied for larger variety and learning.<\/li>\n<li>After every change, <em>train the model<\/em> and check the model. Ensure that all the dots in the ML model are diagonal (in the True-positive and True-negative) quadrant and you do not have scattered utterances in other quadrants. Train the model until you achieve this.<\/li>\n<li><em>Regularly train<\/em> the bot with new utterances.<\/li>\n<li>Regularly review the <em>failed or abandoned utterances<\/em> and add them to utterance list against a valid task or intent.<\/li>\n<\/ul>\n<h4>NLP Intent Detection Training Recommendations<\/h4>\n<ul>\n<li>If there are a good number of sample utterances, try training the bot using Machine Learning approach first, before trying to train the fundamental meaning model.<\/li>\n<li>Define bot synonyms to build a domain dictionary such as <i>pwd <\/i>for a password; <i>SB <\/i>for a savings bank account.<\/li>\n<li>After every change to the model training, run the batch testing modules. Test suites are a means to perform regression testing of your bot\u2019s ML model.<\/li>\n<\/ul>\n<h4>NLP Entity Detection Training Recommendations<\/h4>\n<p>The best approach to train entities is based on the type of entity as explained below:<\/p>\n<ul>\n<li>Entity type like <b>List of Items<\/b> (enumerated, lookup), <b>City<\/b>, <b>Date<\/b>, <b>Country <\/b>do not need any training unless the same entity type is used multiple types in the same task. If the same entity type is used in a bot task, use either of the training models to find the entity within the user utterances.<\/li>\n<li>When the entity type is <b>String <\/b>or <b>Description, <\/b>the recommended approach is to use Entity patterns and synonyms.<\/li>\n<li>For all other entity types, both NER and Patterns can be used in combination.<\/li>\n<\/ul>\n<div class=\"alert alert-info\">\n<h4>Entity Training Recommendations<\/h4>\n<ul>\n<li>Use NER training where possible &#8211; NER coverage is higher than patterns.<\/li>\n<li>NER approach best suits detecting an entity where information is provided as unformatted data. For entities like Date and Time, the platform has been trained with a large set of data.<\/li>\n<li>NER is a neural network-based model and will need to be trained with at least 8-10 samples to work effectively.<\/li>\n<\/ul>\n<\/div>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<div class=\"w-separator size_medium with_line width_default thick_1 style_solid color_border align_center\"><div class=\"w-separator-h\"><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/section>\n","protected":false},"excerpt":{"rendered":"Developers need to provide sample utterances for each intent (task) the bot needs to identify to train the machine learning model. The platform ML engine will build a model that will try to map a user utterance to one of the bot intents. Kore.ai\u2019s Bots Platform allows fully unsupervised machine learning to constantly expand the...","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-17928","post","type-post","status-publish","format-standard","hentry","category-nlp"],"_links":{"self":[{"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/posts\/17928","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/comments?post=17928"}],"version-history":[{"count":12,"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/posts\/17928\/revisions"}],"predecessor-version":[{"id":23616,"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/posts\/17928\/revisions\/23616"}],"wp:attachment":[{"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/media?parent=17928"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/categories?post=17928"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/developer.kore.ai\/v7-3\/wp-json\/wp\/v2\/tags?post=17928"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}