{"id":61380,"date":"2024-03-08T13:00:32","date_gmt":"2024-03-08T13:00:32","guid":{"rendered":"https:\/\/developer.kore.ai\/v10-1\/?p=61380"},"modified":"2024-03-11T12:45:29","modified_gmt":"2024-03-11T12:45:29","slug":"llm-and-generative-ai-usage-logs-api","status":"publish","type":"post","link":"https:\/\/developer.kore.ai\/v10-1\/docs\/bots\/api-guide\/llm-and-generative-ai-usage-logs-api\/","title":{"rendered":"LLM and Generative AI Usage Logs API"},"content":{"rendered":"
\r\n\t
\r\n\t\t
\r\n\t\t\t

This API lets you get the LLM & Generative AI Usage Logs for both Co-Pilot and Dynamic Conversation features.<\/span><\/p>\n\n\n\n\n\n\n\n
Method<\/strong><\/td>\nPOST<\/td>\n<\/tr>\n
End Point<\/strong><\/td>\nhttps:\/\/{{host}}\/api\/1.1\/public\/bot\/{{botId}}\/getLLMUsageLogs<\/code><\/span><\/td>\n<\/tr>\n
Content Type<\/strong><\/td>\napplication\/json<\/code><\/span><\/td>\n<\/tr>\n
Authorization<\/strong><\/td>\nauth: {{JWT}}<\/code><\/span><\/p>\n

See<\/span> How to generate the JWT Token<\/span><\/a>.<\/span><\/td>\n<\/tr>\n

API Scope<\/strong><\/td>\n\n
    \n
  • Bot Builder: Fetch Gen AI and LLM Usage Logs<\/span><\/li>\n
  • Admin Console: API Scopes > Gen AI and LLM Usage Logs<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    Note<\/b>: You can access records spanning up to a 90-day timeframe with one request.<\/span><\/p>\n

    Query Parameters<\/span><\/h2>\n\n\n\n\n\n
    Parameter<\/th>\nRequired\/Optional<\/th>\nDescription<\/th>\n<\/tr>\n
    host<\/span><\/td>\nRequired<\/span><\/td>\nThe environment URL. For example, https:\/\/bots.kore.ai<\/span><\/td>\n<\/tr>\n
    botId<\/span><\/td>\nRequired<\/span><\/td>\nBot ID or Stream ID. You can access it from the General Settings page of the bot.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Sample Request<\/span><\/h2>\n
    --header 'auth: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwiYXBwSWQiOiJjcy03ZWMxNmFjZS03ZGNmLTU3MjQtYjM5NS1hYTA4YmRmYTAwMzMifQ.n_Es9ZBsiCYzpzsfN4p4I1SlHK05gewJFyqAIngr4Qg' \\\r\n--header 'Content-Type: application\/json' \\\r\n--data '{\r\n    \"dateFrom\": \"2024-03-07\",\r\n    \"dateTo\":\"2024-03-09\",\r\n    \"limit\":\"1\",\r\n    \"skip\" : \"5\",\r\n    \"isDeveloper\": true,\r\n    \"channel\" : \"msteams\",\r\n    \"featureName\" : [\"dynamicEntity\"],\r\n    \"taskId\" : \"dg-d4924db4-b325-5b4d-ae51-aa5c7be41d4f\",\r\n    \"channelUserIds\":[\"29:1gmw8z03rvk6njl6k7ohtdh2v9zubxiip7kvu1yiek_qri4grpmd0k_d1yjlpzbj40wk1am9dphqkoiwatzwttw\"],\r\n    \"userIds\" : [\"u-40b3eafc-a5aa-55f2-83e8-cf4d0fb1de07\"],\r\n    \"sort\" : {\r\n        \"field\" : \"Time Taken\",\r\n        \"order\" : \"desc\"\r\n    }\r\n}'<\/code><\/pre>\n
    Body Parameters<\/span><\/h2>\n\n\n\n\n\n\n\n\n\n\n\n\n
    Parameter<\/th>\nRequired\/Optional<\/th>\nDescription<\/th>\n<\/tr>\n
    dateFrom<\/span><\/td>\nRequired<\/span><\/td>\nThe timestamp from which the records are displayed.<\/span><\/td>\n<\/tr>\n
    dateTo<\/span><\/td>\nRequired<\/span><\/td>\nThe timestamp till which the records are displayed.<\/span><\/td>\n<\/tr>\n
    limit<\/span><\/td>\nRequired<\/span><\/td>\nThe number of records to be displayed on each page.<\/span><\/td>\n<\/tr>\n
    channel<\/span><\/td>\nOptional<\/span><\/td>\nThe channels to be considered for the metrics.<\/span><\/td>\n<\/tr>\n
    channelUIds<\/span><\/td>\nOptional<\/span><\/td>\nThe user ids (channel specified) to be included in the metrics.<\/span><\/td>\n<\/tr>\n
    isDeveloper<\/span><\/td>\nOptional<\/span><\/td>\nTo include developer metrics. Set to true or false.<\/span><\/td>\n<\/tr>\n
    taskId<\/span><\/td>\nOptional<\/span><\/td>\nTo filter records based on the task ids.<\/span><\/td>\n<\/tr>\n
    userId<\/span><\/td>\nOptional<\/span><\/td>\nTo filter records based on user id.<\/span><\/td>\n<\/tr>\n
    sort<\/span><\/td>\nOptional<\/span><\/td>\nTo sort the results.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Sample Response<\/span><\/h2>\n
    [\r\n{\r\n\"Prompt Name\": \"Default\",\r\n\"Integration\": \"OpenAI\",\r\n\"start Date\": \"2024-03-08T06:38:54.008Z\",\r\n\"End Date\": \"2024-03-08T06:38:58.386Z\",\r\n\"Time Taken\": 4378,\r\n\"statusCode\": 200,\r\n\"Bot ID\": \"st-73bfdb2f-9101-55e6-b4c8-3f568a6ea8e0\",\r\n\"User ID\": \"u-40b3eafc-a5aa-55f2-83e8-cf4d0fb1de07\",\r\n\"Feature Name \": \"GenAI Node\",\r\n\"Model Name\": \"GPT-4\",\r\n\"Channel Name\": \"msteams\",\r\n\"Description\": \"Order pizza-GenAINode0002\",\r\n\"task Id\": \"dg-200ad1ff-8db8-5219-810e-4ee3800d212c\",\r\n\"Status\": \"Success\",\r\n\"Payload Details\": {\r\n\"Prompt Name\": \"Default\",\r\n\"Request Payload\": {\r\n\"model\": \"gpt-4\",\r\n\"temperature\": 0.5,\r\n\"max_tokens\": 1500,\r\n\"top_p\": 1,\r\n\"frequency_penalty\": 0,\r\n\"presence_penalty\": 0,\r\n\"messages\": [\r\n{\r\n\"role\": \"system\",\r\n\"content\": \"You are a virtual assistant representing an enterprise business, and so you have to act professionally at all times. You do not participate or respond to any abusive language or indulge in any conversation that does not represent enterprise business.\\nAct like pizza ordering site. For the instructions that the user provides, you have to process the instructions. Here are the rules that you are supposed to follow: \\nRule 0:\\n Each pizza can only have maximum of 3 toppings \\nRule 1:\\n List of entities you need to capture from user are number of pizzas, toppings and size.. You need to capture all these entitites. \\nRule 2:\\n Capture entities from the user and the conversation history. \\nRule 3:\\n Collect the unfulfilled entities or check if the user wants to update a collected entity. \\nRule 4:\\n If user has provided the required value for any of the required entities, then do not prompt for it again. \\nRule 5:\\n Generate appropriate prompt to the end user to collect the information for the unfulfilled entities. \\nRule 6:\\n In the output return JSON must containing {\\\"bot\\\":\/\/next prompt , \\\"analysis\\\":\/\/Summarization of the entire conversation with all details.,\\\"entities\\\":\/\/array of objects,\\\"conv_status\\\":\/\/The status of the conversation. \\\"ongoing\\\" or \\\"ended\\\".} \\nRule 7:\\n When returning the result return a json, stringified and formatting removed. \\nRule 8:\\n Once all the entities details are captured generate conv_status as 'ended'. When the flow is to be continued, generate conv_status as 'ongoing'  \\nRule 9:\\n Keep the prompts and messages voice friendly. \\nRule 10:\\n If there are mutiple entities, return entitites in format of array of objects. Where each object is entity key values pairs. \\nRule 11:\\n Ignore any out of topic user\/assistant conversations and always collect ALL the mentioned entities. \\n\\n \"\r\n},\r\n{\r\n\"role\": \"user\",\r\n\"content\": \"Order pizza\"\r\n},\r\n{\r\n\"role\": \"assistant\",\r\n\"content\": \"{\\\"bot\\\":\\\"Sure, I can help with that. How many pizzas would you like to order?\\\",\\\"analysis\\\":\\\"The user initiated the pizza ordering process but didn't provide any details.\\\",\\\"entities\\\":[],\\\"conv_status\\\":\\\"ongoing\\\"}\"\r\n},\r\n{\r\n\"role\": \"user\",\r\n\"content\": \"5\"\r\n},\r\n{\r\n\"role\": \"assistant\",\r\n\"content\": \"{\\\"bot\\\":\\\"Great, 5 pizzas it is. What size would you like these pizzas to be?\\\",\\\"analysis\\\":\\\"The user has ordered 5 pizzas but has not specified the size or toppings.\\\",\\\"entities\\\":[{\\\"number_of_pizzas\\\":\\\"5\\\"}],\\\"conv_status\\\":\\\"ongoing\\\"}\"\r\n},\r\n{\r\n\"role\": \"user\",\r\n\"content\": \"Medium Size\"\r\n},\r\n{\r\n\"role\": \"assistant\",\r\n\"content\": \"{\\\"bot\\\":\\\"Perfect, 5 medium-sized pizzas. Now, could you please specify the toppings for your pizzas? Remember, each pizza can have a maximum of 3 toppings.\\\",\\\"analysis\\\":\\\"The user has ordered 5 medium-sized pizzas but has not specified the toppings.\\\",\\\"entities\\\":[{\\\"number_of_pizzas\\\":\\\"5\\\"},{\\\"size\\\":\\\"Medium\\\"}],\\\"conv_status\\\":\\\"ongoing\\\"}\"\r\n},\r\n{\r\n\"role\": \"user\",\r\n\"content\": \"okay\\n\\n{\"\r\n}\r\n]\r\n},\r\n\"Response Payload\": {\r\n\"id\": \"chatcmpl-90OAYDFFkbjgMxhMxKukwqklT6SWO\",\r\n\"object\": \"chat.completion\",\r\n\"created\": 1709879934,\r\n\"model\": \"gpt-4-0613\",\r\n\"choices\": [\r\n{\r\n\"index\": 0,\r\n\"message\": {\r\n\"role\": \"assistant\",\r\n\"content\": \"{\\\"bot\\\":\\\"I'm sorry, but your message seems incomplete. Could you please specify the toppings for your pizzas? Remember, each pizza can have a maximum of 3 toppings.\\\",\\\"analysis\\\":\\\"The user's response was incomplete. The toppings for the pizzas are still needed.\\\",\\\"entities\\\":[{\\\"number_of_pizzas\\\":\\\"5\\\"},{\\\"size\\\":\\\"Medium\\\"}],\\\"conv_status\\\":\\\"ongoing\\\"}\"\r\n},\r\n\"logprobs\": null,\r\n\"finish_reason\": \"stop\"\r\n}\r\n],\r\n\"usage\": {\r\n\"prompt_tokens\": 614,\r\n\"completion_tokens\": 74,\r\n\"total_tokens\": 688\r\n},\r\n\"system_fingerprint\": null\r\n},\r\n\"Request Tokens\": 614,\r\n\"Response Tokens\": 74\r\n}\r\n}\r\n]\r\n<\/pre>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n
    <\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/section>\n","protected":false},"excerpt":{"rendered":"This API lets you get the LLM & Generative AI Usage Logs for both Co-Pilot and Dynamic Conversation features. Method POST End Point https:\/\/{{host}}\/api\/1.1\/public\/bot\/{{botId}}\/getLLMUsageLogs Content Type application\/json Authorization auth: {{JWT}} See How to generate the JWT Token. API Scope Bot Builder: Fetch Gen AI and LLM Usage Logs Admin Console: API Scopes > Gen AI...","protected":false},"author":33,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"class_list":["post-61380","post","type-post","status-publish","format-standard","hentry","category-api-guide"],"_links":{"self":[{"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/posts\/61380","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/comments?post=61380"}],"version-history":[{"count":13,"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/posts\/61380\/revisions"}],"predecessor-version":[{"id":61505,"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/posts\/61380\/revisions\/61505"}],"wp:attachment":[{"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/media?parent=61380"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/categories?post=61380"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/developer.kore.ai\/v10-1\/wp-json\/wp\/v2\/tags?post=61380"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}