examples/flex-flows/chat-async-stream/chat-stream-with-async-flex-flow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stream chat with async flex flow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
    "\n",
    "- Write LLM application using class based flex flow.\n",
    "- Use AzureOpenAIModelConfiguration as class init parameter.\n",
    "- Use prompty to stream completions.\n",
    "- Convert the application into a async flow and batch run against multi lines of data.\n",
    "- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
    "\n",
    "## 0. Install dependent packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -r ./requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Trace your application with promptflow\n",
    "\n",
    "Assume we already have a python program, which leverage prompty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"flow.py\") as fin:\n",
    "    print(fin.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When `stream=true` is configured in the parameters of a prompt whose output format is text, promptflow sdk will return a generator type, which item is the content of each chunk.\n",
    "\n",
    "Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"chat.prompty\") as fin:\n",
    "    print(fin.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Create necessary connections\n",
    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
    "\n",
    "Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
    "\n",
    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.client import PFClient\n",
    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
    "\n",
    "# client can help manage your runs and connections.\n",
    "pf = PFClient()\n",
    "try:\n",
    "    conn_name = \"open_ai_connection\"\n",
    "    conn = pf.connections.get(name=conn_name)\n",
    "    print(\"using existing connection\")\n",
    "except:\n",
    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
    "    connection = AzureOpenAIConnection(\n",
    "        name=conn_name,\n",
    "        api_key=\"<your_AOAI_key>\",\n",
    "        api_base=\"<your_AOAI_endpoint>\",\n",
    "        api_type=\"azure\",\n",
    "    )\n",
    "\n",
    "    # use this if you have an existing OpenAI account\n",
    "    # connection = OpenAIConnection(\n",
    "    #     name=conn_name,\n",
    "    #     api_key=\"<user-input>\",\n",
    "    # )\n",
    "\n",
    "    conn = pf.connections.create_or_update(connection)\n",
    "    print(\"successfully created connection\")\n",
    "\n",
    "print(conn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize trace by using start_trace\n",
    "\n",
    "Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.tracing import start_trace\n",
    "from promptflow.core import AzureOpenAIModelConfiguration\n",
    "\n",
    "from flow import ChatFlow\n",
    "\n",
    "# create a chatFlow obj with connection\n",
    "config = AzureOpenAIModelConfiguration(\n",
    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
    ")\n",
    "chat_flow = ChatFlow(config)\n",
    "\n",
    "# start a trace session, and print a url for user to check trace\n",
    "start_trace()\n",
    "\n",
    "# run the flow as function, which will be recorded in the trace\n",
    "result = chat_flow(question=\"What is ChatGPT? Please explain with detailed statement\")\n",
    "# note the type is async generator object as we enabled stream in prompty\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "# print result in stream manner\n",
    "async for output in result:\n",
    "    print(output, end=\"\")\n",
    "    await asyncio.sleep(0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
    "\n",
    "answer = \"\"\n",
    "async for output in result:\n",
    "    answer += output\n",
    "answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eval the result "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import paths  # add the code_quality module to the path\n",
    "from check_list import EvalFlow\n",
    "\n",
    "eval_flow = EvalFlow(config)\n",
    "# evaluate answer agains a set of statement\n",
    "eval_result = eval_flow(\n",
    "    answer=answer,\n",
    "    statements={\n",
    "        \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
    "        \"consise\": \"It is a consise statement.\",\n",
    "    },\n",
    ")\n",
    "eval_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Batch run the function as flow with multi-line data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch run with a data file (with multiple lines of test data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.client import PFClient\n",
    "\n",
    "pf = PFClient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = \"./data.jsonl\"  # path to the data file\n",
    "# create run with the flow function and data\n",
    "base_run = pf.run(\n",
    "    flow=chat_flow,\n",
    "    data=data,\n",
    "    column_mapping={\n",
    "        \"question\": \"${data.question}\",\n",
    "        \"chat_history\": \"${data.chat_history}\",\n",
    "    },\n",
    "    stream=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "details = pf.get_details(base_run)\n",
    "details.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Evaluate your flow\n",
    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run evaluation on the previous batch run\n",
    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_run = pf.run(\n",
    "    flow=eval_flow,\n",
    "    data=\"./data.jsonl\",  # path to the data file\n",
    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
    "    column_mapping={\n",
    "        \"answer\": \"${run.outputs.output}\",\n",
    "        \"statements\": \"${data.statements}\",\n",
    "    },\n",
    "    stream=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "details = pf.get_details(eval_run)\n",
    "details.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "metrics = pf.get_metrics(eval_run)\n",
    "print(json.dumps(metrics, indent=4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.visualize([base_run, eval_run])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
    "\n",
    "You can check out more examples:\n",
    "- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
   ]
  }
 ],
 "metadata": {
  "build_doc": {
   "author": [
    "D-W-@github.com",
    "wangchao1230@github.com"
   ],
   "category": "local",
   "section": "Flow",
   "weight": 12
  },
  "description": "A quickstart tutorial to run a class based flex flow in stream mode and evaluate it.",
  "kernelspec": {
   "display_name": "prompt_flow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}