The Kore.ai XO Platform helps enhance your bot development process and enrich end-user conversational experiences by integrating pre-trained OpenAI, Azure OpenAI, or Anthropic language models in the backend. With the advancement of LLM and Generative AI technologies, this integration with OpenAI and advanced generative AI adds new capabilities to your Virtual Assistant through auto-generated suggestions.

This capability can automate dialog flow creation, user utterance testing and validation, and conversation design based on context-specific and human-like interactions.

You can find LLM and Generative AI features by going to Build > Natural Language > Advanced NLU Settings.

Key Features

The Integration of LLM and Generative AI enables the following Dynamic Conversations and Co-pilot features:

Dynamic Conversations Features

• GenAI Node: Collect Entities from end-users in a free-flowing conversation using LLM and Generative AI in the background. You can define entities to be collected as well as rules & scenarios.
• GenAI Prompt: Leverage this node to unlock the power of Generative AI with your own prompts, enabling you to build creative and custom use cases.
• Rephrase Dialog Responses: Enhance end-user experience with empathetic and contextual bot responses.
• Repeat Responses: Use LLM to reiterate the last bot responses when the Repeat Bot Response event is triggered.
• Answer from Documents: Helps answer end-user queries from unstructured PDF documents without the need to extract individual FAQs and train them.
• Zero-shot ML Model: Use Open AI LLM models for intent identification during run time based on semantic similarity.

Co-pilot Features

• Automatic Dialog Generation: This feature helps build production-ready dialog tasks automatically by briefly describing the task. A preview of the generated dialog is available and lets you modify the intent description and create multiple iterations of the dialog.
• Training Utterance Suggestions: Generate high-quality training data quickly and easily with our platform’s suggested utterances for each intent. Review and add the suggestions as needed to create a powerful training set for your bot.
• Conversation Test Cases Suggestion: The Platform suggests simulated user inputs covering various scenarios from an end-user perspective at every test step. You can use these suggestions to create test suites. 
• NLP Batch Test Cases Suggestions: The Platform generates NLP test cases for every intent, including entity checks. You only need to create test suites in the Builder using the generated testing utterances.

Benefits

All these features benefit VA developers, NLP developers, and testers as follows:

• Developers can create dialog tasks on-the-fly through the prebuilt Dialog Tasks Flow.
• Developers can unlock the power of Generative AI with their own prompts, enabling them to build creative and custom use cases.
• Mundane tasks like generating dialog tasks or training utterances are automated to help developers be more productive and focus on other important tasks like enhancing conversation design, creating complex test cases, and more.
• Testers can ensure that their intent descriptions are meaningful in the right context to generate the right content.
• The Platform provides suggestions and nudges developers in the right direction for the better design and development of Virtual Assistants. For example, it offers curated use case suggestions while creating the VA, including probable user inputs (simulating end-user behavior) in Conversation Testing. This way, the VA can simulate the end user’s behavior at every conversation step and respond more realistically by considering error scenarios, digressions, and contextual changes.

Important Considerations

The Generative AI features are supported for English and non-English NLU and Bot languages on the Kore.ai XO Platform. To learn more about managing languages for VAs, click here.

LLM and Generative AI also require sharing data with third parties: OpenAI (when using the OpenAI integration) or OpenAI and Microsoft (when using the Azure integration).

Configure LLM and Generative AI

The following steps are necessary to configure LLM and Generative AI:

1. Prerequisites
2. Integration Setup
3. Enable LLM Features

Prerequisites

To enable the LLM & Generative AI features, you must meet the following prerequisites:

1. Upgrade to NLP Version – v3 (click Upgrade when prompted within the Advanced NLU Settings section).
2. Select the desired language as the NLP Language under Configurations > Languages.

Integration Setup

LLM and Generative AI features are available by integrating with OpenAI directly or via Microsoft Azure.

To set up an integration, follow these steps:

1. Click OpenAI or Azure OpenAI under Advanced NLU Settings > LLM & Generative AI > Integration Setup.
2. Configure the OpenAI connector, Azure OpenAI connector, or the Anthropic connector.
   Note: When you configure more than one LLM Model, the one you configure first becomes the default selection for the LLM features. You can change the default model for your features if required. Between OpenAI and Azure OpenAI, we recommend the Azure integration because the Azure OpenAI Service is more reliable and provides REST API access to OpenAI’s language models.
3. Once configured, the status of your chosen integration changes from Configure Now to X/Y models configured, where X is the number of models configured, and Y is the number of models available.

Enable LLM Features

A feature is enabled by default if it supports one or more configured generative AI models. You can disable/enable a feature as required. The Platform automatically selects the default model for each enabled feature. You can select another supported model for a feature if you have configured multiple models. The selected model becomes the default model for the feature across the Platform. You can also change the selected model’s settings if required. In most cases, the default settings work fine.

Steps to change the default model, its settings, or both:

• To select another model, click the model drop-down list for the feature and select the desired model.
  
• To change the settings for the model, click the Gear icon next to its name. The Advance Settings dialog box is displayed.
  
  Adjusting the settings allows you to fine-tune the model’s behavior to meet your needs. The default settings work fine for most cases. However, if required, you can tweak the settings and find the right balance for your use case. A few settings are common across the features, and a few are feature-specific:
  • Model: The selected model for which the settings are displayed.
  • Instructions or Context: Add feature/use case-specific instructions or context to guide the model.
  • Temperature: The setting controls the randomness of the model’s output. A higher temperature, like 0.8 or above, can result in unexpected, creative, and less relevant responses. On the other hand, a lower temperature, like 0.5 or below, makes the output more focused and relevant.
  • Max Tokens: It indicates the total number of tokens used in the API call to the model. It affects the cost and the time taken to receive a response. A token can be as short as one character or as long as one word, depending on the text.
  • Number of Previous User Inputs: Indicates how many previous user messages should be sent to the model as context for rephrasing the response sent through the respective node. For example, 5 means that the previous 5 responses are sent as context.
  • Additional Instructions: Add specific instructions on how prompts should be rephrased. You can create a persona, ask it to rephrase in a particular tone, etc.
  • Similarity Threshold: The Similarity Threshold is applicable for the Answer from Docs feature. This threshold refers to the similarity between the user utterance and the document chunks. The platform shortlists all chunks that are above the threshold and sends these chunks to the LLMs to auto-generate the responses. Define a suitable threshold that works best for your use case. Setting a higher threshold limits the number of chunks and may not generate any result. Setting a lower threshold might qualify too many chunks and might dilute the response.

Dynamic Conversations Features Specifications

GenAI Node

When enabled, this feature lets you add an GenAI Node to Dialog Tasks. This node allows you to collect Entities from end-users in a free-flowing conversation (in the selected English/Non-English Bot Language) using LLM and Generative AI in the background. You can define the entities to be collected as well as rules & scenarios in English and Non-English Bot languages. You can configure node properties just like any other node. You can also use the GenAI Node across Dialog Tasks.

Usage

When creating or editing a Dialog Task that’s created manually or auto-generated, you can find a node called GenAI Node within your nodes list.
When this feature is disabled, the node is unavailable within the Dialog Builder. Learn more.

GenAI Prompt

This feature lets you define custom user prompts based on the conversation context and the response from the LLMs. You can define the subsequent conversation flow by selecting a specific AI model, tweaking its settings, and previewing the response for the prompt.

Usage

1.  When building the Dialog Flow, click the “+” button, and select the GenAI Prompt node.
   
2. Configuring the Component Properties in the following sections helps set up the node:

• General Settings: Provide Name and Display Name for the node and write your own OpenAI Prompt.
• Advanced Settings: Fine-tune the model’s behavior and tweak its settings as required for the following:
  • Model
  • System Context
  • Temperature
  • Max Tokens

3. Advanced Controls: Select the maximum wait time (Timeout) to receive a response from the LLM and the bot’s response (Timeout Error Handling) when a timeout error occurs.
4. When you add custom tags to the current message, user profile, and session under Instance Properties, you can build custom profiles for the bot conversation. .
5. Configuring node connections on an instance lets you define the connection rules for the conversation using transition conditions. This lets the conversation follow specific paths based on the user’s input.

If this feature is disabled, you cannot configure the ML model to build custom prompts using OpenAI for different use cases. Learn more.

Rephrase Dialog Responses

This feature sends all User Prompts, Error Prompts, and Bot Responses to the Generative AI along with the conversation context, which depends on the configured number of user inputs. Responses are rephrased in English or the selected Non-English Bot Language based on the context and user emotion, providing a more empathetic, natural, and contextual conversation experience to the end-user. You can give instructions (additional instructions) in English or any other bot language you select.

Usage

When configuring a Message, Entity, or Confirmation node, you can enable the Rephrase Response feature (disabled by default). This lets you set the number of user inputs sent to OpenAI/Anthropic Claude-1 based on the selected model as context for rephrasing the response sent through the node. You can choose between 0 and 5, where 0 means that no previous input is considered, while 5 means that the previous. 5 responses are sent as context.

When this feature is disabled, the Rephrase Response section is not visible within your node’s Component Properties.

Repeat Responses

This feature uses LLM to reiterate the recent bot responses when the Repeat Response event is triggered. Bot developers can enable the event and customize the trigger conditions. This empowers end-users to ask the bot to repeat its recent responses at any point during the conversation. Currently, this event is supported for IVR, Audiocodes, and Twilio Voice channels. Learn more.

Answer From Documents

This feature leverages a Large Language Model (LLM) and Generative AI models from OpenAI to generate answers for FAQs by processing uploaded documents in an unstructured PDF format and user queries.

Usage

After redacting personally identifiable information, the uploaded documents and the end-user queries are shared with OpenAI to curate the answers.

Once you meet the prerequisites and enable the feature:

1. You should upload the PDF document(s) to be shared with the third-party system (OpenAI). You can upload a maximum of 10 documents with a size of not more than 5 MB.
2. The uploaded documents are listed under the Answer from Documents section with the following details:

• Upload Name
• Uploaded by
• Uploaded on
• Status (Active/Inactive)
• Actions (View and Delete File)

3. A good practice is to test the answer generation by asking the VA a question directly related to the contents of your uploaded documents. Learn more.
4. You can view, delete, and disable the uploaded documents.
5. The VA provides answers only from uploaded documents that are active, whereas disabled documents are ignored.

If the feature is disabled, you won’t be able to send queries to LLMs as a fallback. Learn more.

Zero-shot ML Model

This feature uses a pre-trained language and Open AI LLM models to help the ML Engine identify the relevant intents from user utterances based on semantic similarity. By identifying the logical intent during run time, this feature eliminates the need for training data. The Zero-shot ML model requires well-defined intents to work well. This training approach is well-suited for virtual assistants with relatively fewer intents and distinct use cases.

Usage 

Before performing utterance testing, the user selects the Zero-Shot Learning Model with OpenAI Network Type. During utterance testing, the user provides a more descriptive input with a subject, object, and nouns. Once the test runs, the system identifies the most logical intent as the definitive match by comparing the following:

• User utterance input
• Intent names

The identified intent is then displayed as the matched intent.

If this feature is disabled, the system won’t identify and display the logical and matched intent during utterance testing. Learn more.

Co-pilot Features Specifications

Automatic Dialog Generation

This feature auto-generates conversations and dialog flows in the selected language using the VA’s purpose and intent description provided (in English or the selected Non-English Bot Language) during the creation process. The Platform uses LLM and generative AI to create suitable Dialog Tasks for Conversation Design, Logic Building & Training by including the required nodes in the flow. 

You must provide an intent description, and the Platform handles the Conversation Generation for the Dialog Flow.

You can preview the conversation flow, view the Bot Action taken, improvise the intent description, and regenerate the conversation to make it more human-like.

The nodes and the flow for the Business Logic are automatically built for your conversation, and you only need to configure the flow transition.

The Platform auto-defines the Entities, Prompts, Error Prompts, Bot Action nodes, Service Tasks, Request Definition, Connection Rules, and other parameters.

Usage

Once you enable this feature:

1. The Platform triggers the flow when you launch a task for the first time.
2. The Platform presents the intent description and an option to generate conversation.
3. You can preview the generated conversation, edit the description, and regenerate it.
4. The Platform sends the updated description to Generative AI in the background to get the new conversation.
5. Once you’re satisfied with the conversation, generate a dialog task.

If this feature is disabled, you will not have the option to auto-generate a dialog flow when first launching a Dialog Task.

Learn more.

Training Utterance Suggestions

This feature generates a list of suggested training utterances and NER annotations based on the selected NLU language for each intent description and Dialog Flow, eliminating the need for manual creation.

Usage

The Platform can generate utterances in the NLU language to train your VA.

Once you request to generate utterances for a given intent, the Platform provides utterance suggestions (in the selected NLU language) based on the following information: 

1. Intent,
2. Entities and Entity Types,
3. Probable entity values,
4. Different scenarios for training utterances based on entities, a combination of entities, structurally different utterances, etc.

You can add/delete the suggested training utterances from the list or generate more suggestions.

If this feature is disabled, you won’t see the Suggestions tab on the training page. Learn more.

Conversation Test Cases Suggestions

This feature provides a regression tool or a Playbook that creates a conversation test suite for each intent (new and old) in English or Non-English Bot language to evaluate the impact of the change on the conversation execution.

You can view input/utterance suggestions at every conversation step simulating the various input types and scenarios. This feature helps check if the task/intent is robust enough to handle random user utterances.

This feature also helps you predict and simulate the end user’s behavior and check if the VA can execute all the defined flows by generating user responses and presenting any digressions from the specified intent.

Usage

1. You can create a test suite by recording a live conversation with a VA by initiating the interaction or letting the VA initiate it.
2. An icon displays to indicate the user input suggestions from Generative AI.
3. The Platform triggers OpenAI or Anthropic Claude-1 based on your model selection to generate suggestions for user input where expected in English or a Non-English Bot Language.
4. The Platform shares the following information with OpenAI/Anthropic to generate suggestions
   1. Randomly picked intents from the VA (Dialog, FAQ),
   2. Conversation flow,
   3. Current Intent, if any,
   4. Current node type:
      • For an entity node, the system sends the following:
        • Entity name,
        • Entity type,
        • Sample entity values. 
      • Types of user input scenarios:
        • Entities,
        • Without entities,
        • Different entity combinations in the user input,
        • Digression to another intent,
        • Trigger error scenarios.
   5. You can regenerate suggestions if required.
   6. You can accept the suggestions shown or type custom input at every step.
   7. Create a Test Suite after stopping the recording and validating the model.

If the feature is disabled, the Platform doesn’t display the Generative AI suggestions icon and the suggestions themselves. Learn more.

NLP Batch Test Cases Suggestions

This feature lets you generate test cases based on the NLU Language selected, and add them to the test suite with minimum or no errors. You can give instructions in English or any Non-English bot language you’ve selected. Additionally, in case of a Multilingual NLU, the system generates utterances in the language prompted by the user. For instance, if your instructions are in Hindi, the utterances are generated in Hindi.

Usage

1. When creating a New Test Suite, select Add Manually or Upload Test Cases File to add test cases. 
2. Add Manually creates an empty test suite where you can generate test cases and do the following:
   1. Select a Dialog Task.
   2. Based on the intent context, the Generative AI generates test cases you can review and add to or remove from the Test Suite.
3. When you click Generate Test Cases, the Platform sends Generative AI (OpenAI or Anthropic Claude-1 model based on your selection) the following information to generate test cases:
   1. Intent
   2. Entities
   3. Probable entity value
   4. Different scenarios to simulate end-user utterances,
   5. Random training utterances and test cases that are generated to avoid duplicate test cases from Generative AI.

If this feature is disabled, you will not have the option to generate test cases during batch testing. Learn more.

More Like This

• XO Platform: LLM and Generative AI (Kore Academy Video)
• XO Platform Generative Language Models
• Next-Gen Conversational AI