시작
Kore.ai 대화형 플랫폼
챗봇 개요
자연어 처리(NLP)
봇 개념 및 용어들
빠른 시작 가이드
봇 빌더 접근 방법
사용 고지 사항 (영어)
Kore.ai 봇 빌더로 작업하기
봇 구축 시작하기
릴리스 정보
현재 버전 (영어)
이전 버전 (영어)

개념
디자인
스토리보드
대화 작업
개요
Using the Dialog Builder Tool
노드 유형
사용자 의도 노드
대화 노드
엔티티 노드
양식 노드
확인 노드
서비스 노드
봇 조치 노드
Service Node
WebHook 노드
스크립트 노드
노드 그룹화하기
Agent Transfer Node
사용자 프롬프트
음성 통화 속성
대화 관리
노드 및 전환
구성 요소 전환
컨텍스트 개체
이벤트 기반 봇 조치
지식 그래프
소개
지식 추출
지식 그래프 생성
봇에 지식 그래프 추가
그래프 생성
지식 그래프 작성
FAQ 추가
작업 실행
기존 소스에서 FAQ 구축
특성, 동의어 및 불용어
변수 네임스페이스 관리
수정
용어 편집 및 삭제
용어 편집 및 삭제
질문과 응답 편집
Knowledge Graph Training
지식 그래프 분석
봇 온톨로지 가져오기 및 내보내기
지식 그래프 가져오기
지식 그래프 내보내기
지식 그래프 생성
CSV 파일에서
JSON 파일
지식 그래프 생성
경고 작업
스몰 토크
Digital Skills
디지털 양식
Views
Digital Views
Panels
Widgets
기차
봇 성능 향상 – NLP 최적화
기계 학습
소개
모델 검증
기초 의미
지식 그래프 학습
특성
순위 및 해결
고급 NLP 설정
NLP 설정 및 지침
봇 인텔리전스
소개
컨텍스트 관리
컨텍스트 관리
대화 관리
다중 – 의도 탐지
엔티티 수정
기본 대화
정서 관리
어조 분석
Test & Debug
봇과 대화
발화 테스트
배치 테스트하기
대화 테스트
배포
채널 활성화
봇 게시
분석
봇 분석하기
Conversations Dashboard
Performance Dashboard
사용자 정의 대시보드
소개
맞춤형 메타 태그
사용자 정의 대시보드 생성 방법
Conversation Flows
NLP 지표
Containment Metrics
사용량 지표
스마트 봇
소개
범용 봇
소개
범용 봇 정의
범용 봇 생성
범용 봇 학습
범용 봇 커스터마이징
범용 봇용 추가 언어 활성화
스토어
Manage Assistant
플랜 및 사용량
Overview
Usage Plans
Support Plans
플랜 관리
봇 인증
다국어 봇
개인 식별 정보 삭제하기
봇 변수 사용
IVR 통합
일반 설정
봇 관리

방법
간단한 봇 생성하기
Design Conversation Skills
뱅킹 봇 생성
뱅킹 봇 – 자금 이체
뱅킹 봇 – 잔액 업데이트
Knowledge Graph (KG) 구축
스마트 경고를 예약하는 방법
Design Digital Skills
디지털 양식 설정 방법
디지털 보기 설정 방법
데이터 테이블에 데이터를 추가하는 방법
데이터 테이블 내 데이터 업데이트 방법
UI 양식에서 데이터 테이블에 데이터를 추가하는 방법
Train the Assistant
특성 사용 방법
의도와 엔티티에 대한 패턴 사용 방법
컨텍스트 전환 관리 방법
Deploy the Assistant
상담사 전환을 설정하는 방법
봇 기능 사용 방법
콘텐츠 변수 사용 방법
전역 변수 사용 방법
Kore.ai 웹 SDK 튜토리얼
Kore.ai 위젯 SDK 튜토리얼
Analyze the Assistant
사용자 정의 대시보드 생성 방법
사용자 지정 태그를 사용하여 봇 메트릭을 필터링하는 방법

API 및 SDK
API 참조
Kore.ai API 사용
API 목록
API 컬렉션
koreUtil Libraries
SDK 참조
상담사 전환을 설정하는 방법
봇 기능 사용 방법
콘텐츠 변수 사용 방법
전역 변수 사용 방법
소개
Kore.ai 웹 SDK 튜토리얼
Kore.ai 위젯 SDK 튜토리얼

관리
소개
봇 관리자 콘솔
대시보드
사용자 관리
사용자 관리
그룹 관리
역할 관리
봇 관리 모듈
등록
사용자 초대
사용자 등록을 위한 대량 초대 보내기
사용자 및 사용자 데이터 가져오기
Active Directory에서 사용자 동기화
보안 및 준수
싱글 사인 온 사용
보안 설정
Kore.ai 커넥터
봇 관리자용 분석
Billing (지원하지 않음)
  1. Docs
  2. Virtual Assistants
  3. Analyzing Your Bot
  4. Flow Health

Flow Health

Introduction 

The Flow Health dashboard summarizes how well the flows of Virtual Assistant (VA) are performing and shows the coverage of intents and transitions in the bot definition. The Flow Health dashboard displays the overall pass or fail percentage of test cases with options to view the transitions coverage. 

The Flow Health dashboard also shows the summary of user intents that are covered and missed in test cases. You can review the uncovered intents and add relevant test cases to cover missing user intents in the test suite. 

You can also use the test result analysis to identify the issues in the dialog task definitions and fix them with actionable recommendations to improve overall NLU and Flow health.

Note: The Flow Health Dashboard is available on the XO Platform version 10.0, released on Jan 21st, 2023.

Use-Case Scenario 

In this article, the Hotel Booking virtual assistant is used as an example to understand the flow of transitions and test results analysis. The Hotel Booking VA has the Search A Hotel, Book a Hotel, Get User Data, Show User Data, and Raise Complain dialog intents with various test cases in the test suites in the Conversation Testing module.

A transition shows how a user intent is transferred from one node to another node in a dialog task. A transition is a flow of intent between two nodes. It has a From node and a To node.  For example, “Book a Hotel ” is a user intent in a dialog task. A city is an entity node in the dialog flow to which the book a hotel user intent is transitioned, and then it transitions to the state node. From the state node, it transitions to country and reservemessage nodes. Therefore, in the Book a Hotel task, you have four transitions in total. For more information, see User Intent.

Once you record a conversation flow, save it and run it in the Conversation Testing module.  The conversation testing module helps in knowing how a dialog task performs. For example, if you have a Get User Data dialog task and want to see how this dialog task is performing, you can create a Show User Data test suite with various test cases. Run the Show User Data test suite and see whether it was completed successfully, failed, paused, or not executed. For more information, see the Conversation Testing article.

The test suite results are tracked in the Conversation Testing, and the analytics for test suites are displayed in the Flow Health dashboard.  

How to Use the Flow Health Dashboard

To access the Flow Health Dashboard, follow these steps:

  1. Navigate to BUILD > Testing > Health & Monitoring and select the Flow tab.

  2. Drill down to learn how to view the Dialog Intent, FAQ, and Small Talk summary sections. In these sections, you can view the intent level summaries, along with test coverage details to easily identify the transitions or intents to be covered. You can also expand to the test case results to view the execution analytics.

Note: The test suites are used to verify the Intents, Small Talks, FAQs, and Standard Responses with test cases. The Flow Health dashboard considers standard responses in test cases to calculate overall percentages. However, the dashboard has no separate summary tile for the Standard Responses. For this reason, you will notice a slight difference in the test results’ overall count or percentage.

Flow Metrics

The Flow Health dashboard extrapolates all test suite results from the conversation testing and presents insights into your test cases with performance analytics.

View the overall percentage of the test case results in the test suites with the following flow metrics:

  • Pass – Total Number of test cases passed across all test suites.
  • Fail – Total Number of Test cases failed across all test suites.
  • Not Executed – Total Number of Test cases aborted or not executed before execution across all test suites.
  1. Click the View Test Suites link to view all the test suites that are considered in the VA with the following details:
    • Test Suite  – The name of the test suite.
    • Result – The result of the test suite, such as Passed, Failed, Aborted, or Not Executed.
    • Duration – The duration for the execution of the test suite.
    • Test Cases – The number of test cases in the test suite.
    • Pass  – The number of test cases that passed.
    • Fail – The number of test cases that failed.
    • Not Executed – The number of test cases that are not executed.
    • Pass percentage – The overall pass percentage of the test cases.
  2. Click the Test Suite Name to analyze how the test suite is performing. For example, click the FAQ_TS_Pass test suite to view the test results analysis as shown below:
  3. Select the Test Coverage tab to view the Dialog Intents, FAQs, and Small Talks covered and that are not covered in the test suite. 
  4. By default, the Dialog Intents are displayed. Click the FAQs to view FAQs details.
  5. Click the Small Talks tab to view small talk details.
  6. Click the X icon to close the Intent Details window.

Flow Dashboard Components

The key components of the Flow Health dashboard include transition coverage and intent summary panels for the test suites. The details on the available panels are explained below.

Dialog Intent Summary

The Dialog Intent Summary shows the summary of test results analysis of dialog intents in the test cases from the Conversation Testing. The Dialog Intent summary also provides insights into flow transitions that are covered and ones that are missed.

In the Dialog Intent Summary region, you can view flow transitions and test result analysis-level summaries, along with the test coverage, to easily identify intents to be covered. You can drill down to view the analytics of test results.

Note: All the test cases across test suites expecting a dialog node for assertion are considered for generating the flow summary. 

Transitions Coverage 

The Flow Health dashboard calculates the number of transitions in a dialog and shows the analysis of covered and not covered transitions. The Transition Coverage section helps you quickly understand how transitions are performing and fix issues in the specific transitions with actionable recommendations. You can analyze the transitions that are missed and add relevant test cases in the test suite to cover them in multiple iterations.

The percentage of transitions coverage is displayed as follows: 

  • Total Transitions Covered – Shows all covered transitions out to total transitions in the VA definition. 
  • Covered – Shows the number of transitions that are covered in the test cases.
  • Not Covered – Shows the percentage of total transitions that are not covered in the test cases.
    1. Click the icon beside the percentages to open the Dialog Transitions page and view the following details:
      • Task Name – The name of the dialog intent. 
      • From Node – The From node where the conversation starts to transition.
      • To Node – The To node from where the conversation is transferred to.
      • Status – The Status of the transition  – covered or not covered.
    2. Click the icon beside Not Covered percentage to view missed transitions.
    3. Create a new test suite and record missed transitions to cover them. For more information, see Conversation Testing.
    4. Click the icon to search and filter the dialog intent tasks for which you want to see the transitions coverage. You can filter transitions using the From Node, To Node, and Coverage Status filters.
      For example, in the Status Coverage column, click the to select all, covered, and not covered statuses, as shown below:
    5. Click the X icon to close the Dialog Transitions window.

Conversation Test Results Analysis

The Conversation Test Results Analysis section summarizes all test suite results. You can use the test result analysis to identify the issues in the dialog task definitions and fix them.

The percentage of test results analysis is displayed as follows: 

  • Total Test Cases – Shows the total number of test cases in a test suite.
  • Pass, Fail, or Not Executed – Shows the count and percentage of test case results that have passed, failed, or not executed.

View Intent Summary

The View Intent Summary shows how well test cases are covering and performing at the intent level, making it easier to identify the most important failing and less-covered intents.

  1. Click the View Intent Summary link to view the intent details in the Dialog Intent Details view.
  2. Once you click the View Intent Summary link, the Dialog Intent Details displays all intents that are covered in a test suite.
    This summary shows the intents covered in the test suite with the total number of transitions that are covered and missed. It also shows the percentage of test cases that have passed, failed, or not executed status against the total number of test cases for an intent.

  3. In the Dialog Intent Details window, click on Covered By for any Intent Name to view the summary. For example, the Raise Complain intent is covered in a test suite with a total of Nine transitions, out of which only Eight are covered and One is missed. The total number of test cases in this intent are Eight, out of which Seven have passed and One has failed, with Zero test case not executed.

To view the intent summary and identify the issues, follow these steps:

  1. Click the > icon to open the intent details summary for the desired intent.
  2. Click the Test Suite Name to analyze how the test suite is performing. For example, click the Dialog_Test_Raise_Complain_Fail test suite to view the test results analysis as illustrated in the screenshot below:
  3. In this test suite, the test case1 has failed because it has an incorrect data type selected for the Date field as shown below:
  4. Text assertion fails because there is a mismatch between the Expected and Actual output. For more information, see Test Case Assertion.   
  5. Click the X icon to close the Intent Details window.

FAQs Summary

The FAQ Summary section displays the FAQ Coverage and Conversation Test Results Analysis generated for all the FAQ intents in all the test cases of the selected test suite, as discussed in the View Intent Summary section.

Click View Intent Summary to get a drill-down view of the FAQ Intents that are covered or not covered in the test suites on the FAQ Intent Details window.

FAQ Intent Details Window

This window gives the drill-down view of the FAQ-level details for the selected test suite. The primary objective here is to help users know the top-performing and low-performing FAQs to identify issues proactively and work on fixing them accordingly.

For more information, see View Intent Summary section.

  • Intent Name: The name of the dialog intent.
  • Path: The node path in the Knowledge Graph.
  • Covered by (Test Suites): The count of the test suites in which the FAQ intents are covered.
  • Total Test Cases: The count of the test cases in the selected test suites for that FAQ intent.
  • Pass: The count of the passed FAQ intent test cases that passed.
  • Fail: The count of the FAQ intent test cases that failed.
  • Not Executed: The count of the FAQ intent test cases that are not executed.
  • Pass %: The overall pass percentage of the test cases.

Small Talk

The Small Talk Summary section displays the Small Talks Coverage and Conversation Test Results Analysis generated for all the Small Talk intents in the selected test suite as discussed in the View Intent Summary section.

Small Talk Intent Details Window

Click View Intent Summary to view the Small Talk Details window that gives the drill-down view of the Small Talk level details for the selected test suite. The primary objective here is to help users know the top-performing and low-performing intents to identify issues proactively and fix them accordingly.

For more information, see View Intent Summary section.

  • Intent Name: The intent name captured in the Small Talk interaction.
  • Group: The group to which the Small Talk interaction is mapped.
  • Covered by (Test Suites): The count of the test suites in which the Small Talk intent is covered.
  • Total Test Cases: The count of the test cases present in the selected test suites for that Small Talk intent.
  • Pass: The count of the Small Talk intent test cases that have passed.
  • Fail: The count of the Small Talk intent test cases that failed.
  • Not Executed: The count of the Small Talk intent test cases that are not executed.
  • Pass %: The overall pass percentage of the test cases.
메뉴