k8s_qna_generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/andyburgin/kubefix-llm/blob/main/k8s_qna_generation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Purpose of this notebook\n",
        "\n",
        "> The objective here is to generate a series of questions and answers from a subset of the kubernetes documentation, the resultant q and a pairs can then be used as a dataset to finetune a LLM for problem solving in [k8sgpt](https://k8sgpt.ai/)\n",
        "\n",
        "> This notebook is part of the scripts and tools needed to gnerate the dataset over at https://github.com/andyburgin/kubefix-llm\n",
        "\n"
      ],
      "metadata": {
        "id": "JpCZUN0GX7yv"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "sClVbz0x09hw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dSr8pAsC0Zqy"
      },
      "source": [
        "# Prequisites install - models, libraries and data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7h5b2HSo0iTT"
      },
      "source": [
        "## Download openchat model\n",
        "\n",
        "After much testing the openchat model is at the time of writing the best model for generating q and a pairs in json from the Kubernetes website markdown files.\n",
        "\n",
        "(It also has an apache 2 license to avoid any data generation licensing restrictions unlike a lot of populat models)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ts23SGcm0C4Z",
        "outputId": "2799e9d9-5b13-41d2-8f06-b9a72dcc2bfd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (0.20.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (3.14.0)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (2023.6.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (2.31.0)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (4.66.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (6.0.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (4.11.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub) (24.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub) (2024.2.2)\n",
            "Consider using `hf_transfer` for faster downloads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.\n",
            "downloading https://huggingface.co/TheBloke/openchat-3.5-0106-GGUF/resolve/main/openchat-3.5-0106.Q5_K_M.gguf to /root/.cache/huggingface/hub/tmpmi96821m\n",
            "openchat-3.5-0106.Q5_K_M.gguf: 100% 5.13G/5.13G [04:47<00:00, 17.9MB/s]\n",
            "./openchat-3.5-0106.Q5_K_M.gguf\n"
          ]
        }
      ],
      "source": [
        "!pip3 install huggingface-hub\n",
        "!huggingface-cli download TheBloke/openchat-3.5-0106-GGUF openchat-3.5-0106.Q5_K_M.gguf --local-dir . --local-dir-use-symlinks False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J79-Jy1-0mi0"
      },
      "source": [
        "## Install lamma-cpp-python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NQVgTB580vTM",
        "outputId": "5e5f9e06-7f65-478c-edba-101fb1a273e1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting llama-cpp-python\n",
            "  Downloading llama_cpp_python-0.2.69.tar.gz (42.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.5/42.5 MB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (4.11.0)\n",
            "Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (1.25.2)\n",
            "Collecting diskcache>=5.6.1 (from llama-cpp-python)\n",
            "  Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: jinja2>=2.11.3 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (3.1.3)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=2.11.3->llama-cpp-python) (2.1.5)\n",
            "Building wheels for collected packages: llama-cpp-python\n",
            "  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.2.69-cp310-cp310-linux_x86_64.whl size=55266130 sha256=67f058c62671b7e3609a69a6a35cf367de01eeec9fb3ca20eeaf408ab6c856c9\n",
            "  Stored in directory: /root/.cache/pip/wheels/e4/18/46/58b5c613b17c8d000d79ae650980fe871b3b490e04e6faa1c1\n",
            "Successfully built llama-cpp-python\n",
            "Installing collected packages: diskcache, llama-cpp-python\n",
            "Successfully installed diskcache-5.6.3 llama-cpp-python-0.2.69\n"
          ]
        }
      ],
      "source": [
        "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" pip install llama-cpp-python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "St-VaxXx01SW"
      },
      "source": [
        "## Install LangChain libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jkMD64Xa07Hw",
        "outputId": "b1b8bbb9-1b4e-4eed-d28f-6f7045af8aa8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting langchain\n",
            "  Downloading langchain-0.1.17-py3-none-any.whl (867 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/867.6 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.5/867.6 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m867.6/867.6 kB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.29)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.9.5)\n",
            "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
            "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain)\n",
            "  Downloading dataclasses_json-0.6.5-py3-none-any.whl (28 kB)\n",
            "Collecting jsonpatch<2.0,>=1.33 (from langchain)\n",
            "  Downloading jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
            "Collecting langchain-community<0.1,>=0.0.36 (from langchain)\n",
            "  Downloading langchain_community-0.0.36-py3-none-any.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m38.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting langchain-core<0.2.0,>=0.1.48 (from langchain)\n",
            "  Downloading langchain_core-0.1.50-py3-none-any.whl (302 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.8/302.8 kB\u001b[0m \u001b[31m21.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting langchain-text-splitters<0.1,>=0.0.1 (from langchain)\n",
            "  Downloading langchain_text_splitters-0.0.1-py3-none-any.whl (21 kB)\n",
            "Collecting langsmith<0.2.0,>=0.1.17 (from langchain)\n",
            "  Downloading langsmith-0.1.54-py3-none-any.whl (116 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.7/116.7 kB\u001b[0m \u001b[31m18.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.25.2)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.7.1)\n",
            "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.31.0)\n",
            "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.2.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)\n",
            "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain)\n",
            "  Downloading marshmallow-3.21.2-py3-none-any.whl (49 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain)\n",
            "  Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
            "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain)\n",
            "  Downloading jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)\n",
            "Collecting packaging<24.0,>=23.2 (from langchain-core<0.2.0,>=0.1.48->langchain)\n",
            "  Downloading packaging-23.2-py3-none-any.whl (53 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting orjson<4.0.0,>=3.9.14 (from langsmith<0.2.0,>=0.1.17->langchain)\n",
            "  Downloading orjson-3.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.5/142.5 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (0.6.0)\n",
            "Requirement already satisfied: pydantic-core==2.18.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (2.18.2)\n",
            "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (4.11.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2024.2.2)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)\n",
            "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain)\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "Installing collected packages: packaging, orjson, mypy-extensions, jsonpointer, typing-inspect, marshmallow, jsonpatch, langsmith, dataclasses-json, langchain-core, langchain-text-splitters, langchain-community, langchain\n",
            "  Attempting uninstall: packaging\n",
            "    Found existing installation: packaging 24.0\n",
            "    Uninstalling packaging-24.0:\n",
            "      Successfully uninstalled packaging-24.0\n",
            "Successfully installed dataclasses-json-0.6.5 jsonpatch-1.33 jsonpointer-2.4 langchain-0.1.17 langchain-community-0.0.36 langchain-core-0.1.50 langchain-text-splitters-0.0.1 langsmith-0.1.54 marshmallow-3.21.2 mypy-extensions-1.0.0 orjson-3.10.3 packaging-23.2 typing-inspect-0.9.0\n"
          ]
        }
      ],
      "source": [
        "!pip install langchain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s2yLOIGy1NF_"
      },
      "source": [
        "## Clone k8s website"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_xpNLkqD1Av6",
        "outputId": "5ca5fcbe-8d55-4ce3-8c90-b2bca716ebad"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'website'...\n",
            "remote: Enumerating objects: 403235, done.\u001b[K\n",
            "remote: Counting objects: 100% (728/728), done.\u001b[K\n",
            "remote: Compressing objects: 100% (458/458), done.\u001b[K\n",
            "remote: Total 403235 (delta 437), reused 477 (delta 265), pack-reused 402507\u001b[K\n",
            "Receiving objects: 100% (403235/403235), 438.54 MiB | 14.47 MiB/s, done.\n",
            "Resolving deltas: 100% (296073/296073), done.\n",
            "Updating files: 100% (9451/9451), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/kubernetes/website.git"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Ka-UITw17AG"
      },
      "source": [
        "## Create folder structure"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-_PvUMX82BYP"
      },
      "outputs": [],
      "source": [
        "!mkdir out"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqu1IKyI9DOB"
      },
      "source": [
        "## IMPORTANT - upload kubedoclist.txt !!!!\n",
        "\n",
        "This is available over at https://github.com/andyburgin/kubefix-llm it's basically a list of paths to the md files in /content/en/docs folder of the kubernetes website (minus the `.md` extension).\n",
        "\n",
        "Any lines starting with `#` will be ignored - we don't need to process every doc for our purposes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qSC_iDOY4pa-"
      },
      "outputs": [],
      "source": [
        "# noop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Li4g02yO2EPX"
      },
      "source": [
        "# Run Q and A generation\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dMLPW1Dq4XDI"
      },
      "source": [
        "## Run model against each file in `/content/website/content/en/docs/` listed in `kubedoclist.txt` not prefixed with`#` creating json output in `out` folder\n",
        "\n",
        "You're probably gonna hit some problems with some of the bigger markdown files, they might go into an infite generation loop or start producing nonsense - if/when that happens...\n",
        "\n",
        "* Stop the script, tar and download the files you have (see next step).\n",
        "* Make a note of the problem file and add it to your `kubedoclist-fix.txt`.\n",
        "* Comment out the file in your `kubedoclist.txt` file and re upload.\n",
        "* Run this cell again.\n",
        "\n",
        "\n",
        "Note that some of the files generated won't be valid json and will require manual fixing up - more below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7vSIB6ea4qgu",
        "outputId": "c7c8f8bf-5a36-438e-d0cd-842d943b2687"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /content/openchat-3.5-0106.Q5_K_M.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = openchat_openchat-3.5-0106\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 8192\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
            "llama_model_loader: - kv  11:                          general.file_type u32              = 17\n",
            "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32002]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 32000\n",
            "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
            "llama_model_loader: - kv  22:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type q5_K:  193 tensors\n",
            "llama_model_loader: - type q6_K:   33 tensors\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Processing started\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "llm_load_vocab: special tokens definition check successful ( 261/32002 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32002\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 8192\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_embd_head_k    = 128\n",
            "llm_load_print_meta: n_embd_head_v    = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
            "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 0\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 10000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 8192\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: model type       = 8B\n",
            "llm_load_print_meta: model ftype      = Q5_K - Medium\n",
            "llm_load_print_meta: model params     = 7.24 B\n",
            "llm_load_print_meta: model size       = 4.78 GiB (5.67 BPW) \n",
            "llm_load_print_meta: general.name     = openchat_openchat-3.5-0106\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 32000 '<|end_of_turn|>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.30 MiB\n",
            "llm_load_tensors: offloading 32 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 33/33 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =    85.94 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  4807.06 MiB\n",
            "...................................................................................................\n",
            "llama_new_context_with_model: n_batch is less than GGML_KQ_MASK_PAD - increasing to 32\n",
            "llama_new_context_with_model: n_ctx      = 32768\n",
            "llama_new_context_with_model: n_batch    = 32\n",
            "llama_new_context_with_model: n_ubatch   = 32\n",
            "llama_new_context_with_model: flash_attn = 0\n",
            "llama_new_context_with_model: freq_base  = 10000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =  4096.00 MiB\n",
            "llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   134.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =     4.50 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1030\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
            "Model metadata: {'tokenizer.chat_template': \"{{ bos_token }}{% for message in messages %}{{ 'GPT4 Correct ' + message['role'].title() + ': ' + message['content'] + '<|end_of_turn|>'}}{% endfor %}{% if add_generation_prompt %}{{ 'GPT4 Correct Assistant:' }}{% endif %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '32000', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '8192', 'general.name': 'openchat_openchat-3.5-0106', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '17'}\n",
            "Using gguf chat template: {{ bos_token }}{% for message in messages %}{{ 'GPT4 Correct ' + message['role'].title() + ': ' + message['content'] + '<|end_of_turn|>'}}{% endfor %}{% if add_generation_prompt %}{{ 'GPT4 Correct Assistant:' }}{% endif %}\n",
            "Using chat eos_token: <|end_of_turn|>\n",
            "Using chat bos_token: <s>\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Skipping doc - file exists reference/access-authn-authz/authorization.json\n",
            "Skipping doc - file exists reference/access-authn-authz/bootstrap-tokens.json\n",
            "Skipping doc - file exists reference/access-authn-authz/certificate-signing-requests.json\n",
            "Skipping doc - file exists reference/access-authn-authz/kubelet-authn-authz.json\n",
            "Skipping doc - file exists reference/access-authn-authz/node.json\n",
            "Skipping doc - file exists reference/access-authn-authz/psp-to-pod-security-standards.json\n",
            "Processing doc reference/access-authn-authz/service-accounts-admin.md\n",
            " [\n",
            "  {\n",
            "    \"question\": \"What is a ServiceAccount in Kubernetes?\",\n",
            "    \"answer\": \"A ServiceAccount provides an identity for processes that run in a Pod. A process inside a Pod can use the identity of its associated service account to authenticate to the cluster's API server.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between user accounts and service accounts?\",\n",
            "    \"answer\": \"User accounts are for humans, while service accounts are for application processes. User accounts are intended to be global: names must be unique across all namespaces of a cluster. Service accounts are namespaced: two different namespaces can contain ServiceAccounts that have identical names.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the service account token in Kubernetes?\",\n",
            "    \"answer\": \"The service account token is used to authenticate a process running inside a Pod to the cluster's API server. It can be bound to an object like a Pod or Secret, and its metadata is included as extra 'private claims' in the JWT.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are projected volumes?\",\n",
            "    \"answer\": \"Projected volumes are a type of volume that contains data from multiple sources. They can include data like a token for Kubernetes API access, certificate authority data, and information about the namespace in which the Pod is running.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the TokenRequest API?\",\n",
            "    \"answer\": \"The TokenRequest API is used to obtain time-bound tokens for Kubernetes API access. It is used by the kubelet to fetch time-bound tokens using the TokenRequest API. A token served for a TokenRequest expires either when the pod is deleted or after a defined lifespan (by default, that is 1 hour).\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What happens to long-lived ServiceAccount tokens?\",\n",
            "    \"answer\": \"Long-lived ServiceAccount tokens can be manually created. However, it is recommended to use the TokenRequest API to obtain short-lived API access tokens instead.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"How do you create a token for a ServiceAccount?\",\n",
            "    \"answer\": \"You can create a token for a ServiceAccount by using the `kubectl create token` command with the `--bind-service-account` flag, specifying the name of the service account.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between auto-generated and manually created ServiceAccount tokens?\",\n",
            "    \"answer\": \"Auto-generated ServiceAccount tokens are generated by the Kubernetes control plane and have a bounded lifetime. They can be bound to Pods or Secrets, and their metadata is included as extra 'private claims' in the JWT. Manually created ServiceAccount tokens are long-lived and not bound to any specific object. It is recommended to use the TokenRequest API to obtain short-lived API access tokens instead.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What happens to unused auto-generated legacy ServiceAccount tokens?\",\n",
            "    \"answer\": \"Beginning from version 1.29, legacy ServiceAccount tokens that were automatically generated will be marked as invalid if they remain unused for a certain period of time (set to default at one year). Tokens that continue to be unused for this defined period will subsequently be purged by the control plane.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the ServiceAccount controller?\",\n",
            "    \"answer\": \"The ServiceAccount controller manages the ServiceAccounts inside namespaces, and ensures a ServiceAccount named 'default' exists in every active namespace. It also watches for ServiceAccount deletion and deletes all corresponding ServiceAccount Secret.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the token controller?\",\n",
            "    \"answer\": \"The token controller acts asynchronously. It watches for ServiceAccount deletion and deletes all corresponding ServiceAccount Secret. It also watches for ServiceAccount token Secret addition, and ensures the referenced ServiceAccount exists, and adds a token to the Secret if needed. It also watches for Secret deletion and removes a reference from the corresponding ServiceAccount if needed.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the Legacy ServiceAccount token tracking controller?\",\n",
            "    \"answer\": \"The Legacy ServiceAccount token tracking controller generates a ConfigMap called `kube-system/kube-apiserver-legacy-service-account-token-tracking` in the `kube-system` namespace. The ConfigMap records the timestamp when legacy service account tokens began to be monitored by the system.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the Legacy ServiceAccount token cleaner?\",\n",
            "    \"answer\": \"The Legacy ServiceAccount token cleaner checks every 24 hours to see if any auto-generated legacy ServiceAccount token has not been used in a specified amount of time. If so, the cleaner marks it as invalid.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the TokenRequest API?\",\n",
            "    \"answer\": \"The TokenRequest API is used to obtain time-bound tokens for Kubernetes API access. It is used by the kubelet to fetch time-bound tokens using the TokenRequest API. A token served for a TokenRequest expires either when the pod is deleted or after a defined lifespan (by default, that is 1 hour).\"\n",
            "  }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     192.19 ms\n",
            "llama_print_timings:      sample time =     714.68 ms /  1214 runs   (    0.59 ms per token,  1698.67 tokens per second)\n",
            "llama_print_timings: prompt eval time =   80454.50 ms /  6061 tokens (   13.27 ms per token,    75.33 tokens per second)\n",
            "llama_print_timings:        eval time =   52661.70 ms /  1213 runs   (   43.41 ms per token,    23.03 tokens per second)\n",
            "llama_print_timings:       total time =  143967.27 ms /  7274 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a ServiceAccount in Kubernetes?\",\n",
            "    \"answer\": \"A ServiceAccount provides an identity for processes that run in a Pod. A process inside a Pod can use the identity of its associated service account to authenticate to the cluster's API server.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between user accounts and service accounts?\",\n",
            "    \"answer\": \"User accounts are for humans, while service accounts are for application processes. User accounts are intended to be global: names must be unique across all namespaces of a cluster. Service accounts are namespaced: two different namespaces can contain ServiceAccounts that have identical names.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the service account token in Kubernetes?\",\n",
            "    \"answer\": \"The service account token is used to authenticate a process running inside a Pod to the cluster's API server. It can be bound to an object like a Pod or Secret, and its metadata is included as extra 'private claims' in the JWT.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are projected volumes?\",\n",
            "    \"answer\": \"Projected volumes are a type of volume that contains data from multiple sources. They can include data like a token for Kubernetes API access, certificate authority data, and information about the namespace in which the Pod is running.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the TokenRequest API?\",\n",
            "    \"answer\": \"The TokenRequest API is used to obtain time-bound tokens for Kubernetes API access. It is used by the kubelet to fetch time-bound tokens using the TokenRequest API. A token served for a TokenRequest expires either when the pod is deleted or after a defined lifespan (by default, that is 1 hour).\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What happens to long-lived ServiceAccount tokens?\",\n",
            "    \"answer\": \"Long-lived ServiceAccount tokens can be manually created. However, it is recommended to use the TokenRequest API to obtain short-lived API access tokens instead.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"How do you create a token for a ServiceAccount?\",\n",
            "    \"answer\": \"You can create a token for a ServiceAccount by using the `kubectl create token` command with the `--bind-service-account` flag, specifying the name of the service account.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between auto-generated and manually created ServiceAccount tokens?\",\n",
            "    \"answer\": \"Auto-generated ServiceAccount tokens are generated by the Kubernetes control plane and have a bounded lifetime. They can be bound to Pods or Secrets, and their metadata is included as extra 'private claims' in the JWT. Manually created ServiceAccount tokens are long-lived and not bound to any specific object. It is recommended to use the TokenRequest API to obtain short-lived API access tokens instead.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What happens to unused auto-generated legacy ServiceAccount tokens?\",\n",
            "    \"answer\": \"Beginning from version 1.29, legacy ServiceAccount tokens that were automatically generated will be marked as invalid if they remain unused for a certain period of time (set to default at one year). Tokens that continue to be unused for this defined period will subsequently be purged by the control plane.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the ServiceAccount controller?\",\n",
            "    \"answer\": \"The ServiceAccount controller manages the ServiceAccounts inside namespaces, and ensures a ServiceAccount named 'default' exists in every active namespace. It also watches for ServiceAccount deletion and deletes all corresponding ServiceAccount Secret.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the token controller?\",\n",
            "    \"answer\": \"The token controller acts asynchronously. It watches for ServiceAccount deletion and deletes all corresponding ServiceAccount Secret. It also watches for ServiceAccount token Secret addition, and ensures the referenced ServiceAccount exists, and adds a token to the Secret if needed. It also watches for Secret deletion and removes a reference from the corresponding ServiceAccount if needed.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the Legacy ServiceAccount token tracking controller?\",\n",
            "    \"answer\": \"The Legacy ServiceAccount token tracking controller generates a ConfigMap called `kube-system/kube-apiserver-legacy-service-account-token-tracking` in the `kube-system` namespace. The ConfigMap records the timestamp when legacy service account tokens began to be monitored by the system.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the Legacy ServiceAccount token cleaner?\",\n",
            "    \"answer\": \"The Legacy ServiceAccount token cleaner checks every 24 hours to see if any auto-generated legacy ServiceAccount token has not been used in a specified amount of time. If so, the cleaner marks it as invalid.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the role of the TokenRequest API?\",\n",
            "    \"answer\": \"The TokenRequest API is used to obtain time-bound tokens for Kubernetes API access. It is used by the kubelet to fetch time-bound tokens using the TokenRequest API. A token served for a TokenRequest expires either when the pod is deleted or after a defined lifespan (by default, that is 1 hour).\"\n",
            "  }\n",
            "]\n",
            "Processing doc reference/access-authn-authz/validating-admission-policy.md\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is Validating Admission Policy?\",\n",
            "    \"answer\": \"Validating Admission Policy is a declarative, in-process alternative to validating admission webhooks. It uses the Common Expression Language (CEL) to declare the validation rules of a policy.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What resources make a policy?\",\n",
            "    \"answer\": \"A policy is generally made up of three resources: The `ValidatingAdmissionPolicy`, a `ValidatingAdmissionPolicyBinding` and a parameter resource if configuration is needed. At least a `ValidatingAdmissionPolicy` and a corresponding  `ValidatingAdmissionPolicyBinding` must be defined for a policy to have an effect.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the validation actions?\",\n",
            "    \"answer\": \"The supported validation actions are: `Deny`, `Warn`, `Audit`. For example, to both warn clients about a validation failure and to audit the failure, use: `validationActions: [Warn, Audit]`.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a parameter resource?\",\n",
            "    \"answer\": \"Parameter resources allow a policy configuration to be separate from its definition. A policy can define paramKind, which outlines GVK of the parameter resource, and then a policy binding ties a policy by name (via policyName) to a particular parameter resource via paramRef.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a failure policy?\",\n",
            "    \"answer\": \"Failure Policy defines how mis-configurations and CEL expressions evaluating to error from the admission policy are handled. Allowed values are `Ignore` or `Fail`. If Fail, an error calling the ValidatingAdmissionPolicy causes the admission to fail and the API request to be rejected.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a validation expression?\",\n",
            "    \"answer\": \"Validation Expression represents the expression which will be evaluated by CEL. It has access to the contents of the Admission request/response, organized into CEL variables as well as some other useful variables: `object`, `oldObject`, `request`, `params`, `namespaceObject`.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a message expression?\",\n",
            "    \"answer\": \"Message Expression is used to return a more friendly message when the policy rejects a request. It has access to `object`, `oldObject`, `request`, `params`, and `namespaceObject`. The message expression must evaluate to a string.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is type checking?\",\n",
            "    \"answer\": \"Type Checking is the process of parsing the expressions in policy definitions, reporting any syntax errors, rejecting the definition if any errors are found. Afterward, the referred variables are checked for type errors, including missing fields and type confusion, against the matched types of `spec.matchConstraints`.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is variable composition?\",\n",
            "    \"answer\": \"Variable Composition is a way to extract some part of the expressions into variables. A variable is a named expression that can be referred later in `variables` in other expressions. A variable is lazily evaluated when it is first referred. Any error that occurs during the evaluation will be reported during the evaluation of the referring expression.\"\n",
            "  }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     192.19 ms\n",
            "llama_print_timings:      sample time =     425.58 ms /   734 runs   (    0.58 ms per token,  1724.70 tokens per second)\n",
            "llama_print_timings: prompt eval time =   88555.76 ms /  6219 tokens (   14.24 ms per token,    70.23 tokens per second)\n",
            "llama_print_timings:        eval time =   31988.71 ms /   733 runs   (   43.64 ms per token,    22.91 tokens per second)\n",
            "llama_print_timings:       total time =  127109.34 ms /  6952 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is Validating Admission Policy?\",\n",
            "    \"answer\": \"Validating Admission Policy is a declarative, in-process alternative to validating admission webhooks. It uses the Common Expression Language (CEL) to declare the validation rules of a policy.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What resources make a policy?\",\n",
            "    \"answer\": \"A policy is generally made up of three resources: The `ValidatingAdmissionPolicy`, a `ValidatingAdmissionPolicyBinding` and a parameter resource if configuration is needed. At least a `ValidatingAdmissionPolicy` and a corresponding  `ValidatingAdmissionPolicyBinding` must be defined for a policy to have an effect.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the validation actions?\",\n",
            "    \"answer\": \"The supported validation actions are: `Deny`, `Warn`, `Audit`. For example, to both warn clients about a validation failure and to audit the failure, use: `validationActions: [Warn, Audit]`.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a parameter resource?\",\n",
            "    \"answer\": \"Parameter resources allow a policy configuration to be separate from its definition. A policy can define paramKind, which outlines GVK of the parameter resource, and then a policy binding ties a policy by name (via policyName) to a particular parameter resource via paramRef.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a failure policy?\",\n",
            "    \"answer\": \"Failure Policy defines how mis-configurations and CEL expressions evaluating to error from the admission policy are handled. Allowed values are `Ignore` or `Fail`. If Fail, an error calling the ValidatingAdmissionPolicy causes the admission to fail and the API request to be rejected.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a validation expression?\",\n",
            "    \"answer\": \"Validation Expression represents the expression which will be evaluated by CEL. It has access to the contents of the Admission request/response, organized into CEL variables as well as some other useful variables: `object`, `oldObject`, `request`, `params`, `namespaceObject`.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a message expression?\",\n",
            "    \"answer\": \"Message Expression is used to return a more friendly message when the policy rejects a request. It has access to `object`, `oldObject`, `request`, `params`, and `namespaceObject`. The message expression must evaluate to a string.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is type checking?\",\n",
            "    \"answer\": \"Type Checking is the process of parsing the expressions in policy definitions, reporting any syntax errors, rejecting the definition if any errors are found. Afterward, the referred variables are checked for type errors, including missing fields and type confusion, against the matched types of `spec.matchConstraints`.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is variable composition?\",\n",
            "    \"answer\": \"Variable Composition is a way to extract some part of the expressions into variables. A variable is a named expression that can be referred later in `variables` in other expressions. A variable is lazily evaluated when it is first referred. Any error that occurs during the evaluation will be reported during the evaluation of the referring expression.\"\n",
            "  }\n",
            "]\n",
            "Processing doc reference/access-authn-authz/webhook.md\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a WebHook?\",\n",
            "    \"answer\": \"A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. A web application implementing WebHooks will POST a message to a URL when certain things happen.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What does the mode `Webhook` cause Kubernetes to do?\",\n",
            "    \"answer\": \"When specified, mode `Webhook` causes Kubernetes to query an outside REST service when determining user privileges.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the configuration file format for mode `Webhook`?\",\n",
            "    \"answer\": \"The configuration file uses the [kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) file format.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What does a SubjectAccessReview object describe?\",\n",
            "    \"answer\": \"A SubjectAccessReview object describes the action, user attempting to make the request, and either details about the resource being accessed or request attributes.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the two methods for disallowing access in a webhook response?\",\n",
            "    \"answer\": \"The first method indicates that the authorization webhook does not allow, or has 'no opinion' about the request, but if other authorizers are configured, they are given a chance to allow the request. The second method denies immediately, short-circuiting evaluation by other configured authorizers.\"\n",
            "  }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     192.19 ms\n",
            "llama_print_timings:      sample time =     200.30 ms /   351 runs   (    0.57 ms per token,  1752.35 tokens per second)\n",
            "llama_print_timings: prompt eval time =   20261.55 ms /  1608 tokens (   12.60 ms per token,    79.36 tokens per second)\n",
            "llama_print_timings:        eval time =   11718.42 ms /   350 runs   (   33.48 ms per token,    29.87 tokens per second)\n",
            "llama_print_timings:       total time =   34501.65 ms /  1958 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a WebHook?\",\n",
            "    \"answer\": \"A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. A web application implementing WebHooks will POST a message to a URL when certain things happen.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What does the mode `Webhook` cause Kubernetes to do?\",\n",
            "    \"answer\": \"When specified, mode `Webhook` causes Kubernetes to query an outside REST service when determining user privileges.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the configuration file format for mode `Webhook`?\",\n",
            "    \"answer\": \"The configuration file uses the [kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) file format.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What does a SubjectAccessReview object describe?\",\n",
            "    \"answer\": \"A SubjectAccessReview object describes the action, user attempting to make the request, and either details about the resource being accessed or request attributes.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the two methods for disallowing access in a webhook response?\",\n",
            "    \"answer\": \"The first method indicates that the authorization webhook does not allow, or has 'no opinion' about the request, but if other authorizers are configured, they are given a chance to allow the request. The second method denies immediately, short-circuiting evaluation by other configured authorizers.\"\n",
            "  }\n",
            "]\n",
            "\n",
            "\n",
            "(took 308.34 s.):\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "import os\n",
        "import re\n",
        "import gc\n",
        "\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "from langchain_core.prompts import PromptTemplate\n",
        "\n",
        "from langchain.callbacks.manager import CallbackManager\n",
        "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
        "CallbackManager = CallbackManager([StreamingStdOutCallbackHandler()])\n",
        "\n",
        "from langchain_community.llms import LlamaCpp\n",
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "start = time.time()\n",
        "print(\"Processing started\")\n",
        "\n",
        "# paths to input and output\n",
        "inputPath = \"/content/website/content/en/docs/\"\n",
        "outputPath = \"/content/out/\"\n",
        "documentList = \"/content/kubedoclist.txt\"\n",
        "\n",
        "# tidy up from previous run in collab\n",
        "rag_chain = None\n",
        "llm = None\n",
        "gc.collect()\n",
        "\n",
        "\n",
        "def create_llm():\n",
        "  return LlamaCpp(model_path=\"/content/openchat-3.5-0106.Q5_K_M.gguf\",  n_gpu_layers=128, temperature=0.2, max_tokens=32768, n_ctx=32768, n_batch=8, n_threads=8, verbose=True, callback_manager=CallbackManager)\n",
        "\n",
        "\n",
        "def create_prompt():\n",
        "  ## prompt inspired by https://helixml.substack.com/p/how-we-got-fine-tuning-mistral-7b?utm_source=pocket_saves\n",
        "  sys_prompt = \"\"\"\n",
        "  You are an intelligent professor. You create at least 5 question and answer pairs from the given document for your students. Respond with an array of strict JSON 'question' & 'answer' pairs.\n",
        "\n",
        "  For example:\n",
        "  ```json\n",
        "    [\n",
        "      {{\n",
        "        \"question\": \"…\",\n",
        "        \"answer\": \"…\"\n",
        "      }},\n",
        "      {{\n",
        "        \"question\": \"…\",\n",
        "        \"answer\": \"…\"\n",
        "      }}\n",
        "    ]\n",
        "  ```\n",
        "  \"\"\"\n",
        "  instruction = \"\"\"\n",
        "  Here is the document:\n",
        "\n",
        "  {content}\n",
        "  \"\"\"\n",
        "\n",
        "  prompt = \"GPT4 Correct User: \" + sys_prompt + \"\\n\\n\" + instruction + \"<|end_of_turn|>GPT4 Correct Assistant:\"\n",
        "  return PromptTemplate.from_template(template=prompt)\n",
        "\n",
        "\n",
        "def call_model(content):\n",
        "  response = rag_chain.invoke(content)\n",
        "  print(response)\n",
        "  return(response)\n",
        "\n",
        "\n",
        "# main\n",
        "fileDocs = open(documentList,\"r\")\n",
        "docList = fileDocs.read().splitlines()\n",
        "fileDocs.close()\n",
        "\n",
        "custom_rag_prompt = create_prompt()\n",
        "\n",
        "llm = create_llm()\n",
        "\n",
        "rag_chain = (\n",
        "{\"content\": RunnablePassthrough()}\n",
        "| custom_rag_prompt\n",
        "| llm\n",
        "| StrOutputParser()\n",
        ")\n",
        "\n",
        "for filenamepre in docList:\n",
        "  if not filenamepre.startswith('#') :\n",
        "    if os.path.exists(outputPath + filenamepre + \".json\"):\n",
        "      print(\"Skipping doc - file exists \" + filenamepre + \".json\")\n",
        "    else:\n",
        "      print(\"Processing doc \" + filenamepre + \".md\")\n",
        "\n",
        "      # open input file from documentList with appended \".md\"\n",
        "      fileInput = open(inputPath + filenamepre + \".md\", \"r\")\n",
        "      content = fileInput.read()\n",
        "      fileInput.close()\n",
        "\n",
        "      # remove hugo template stuff\n",
        "      pattern =  r'{{.*?}}'\n",
        "      content = re.sub(pattern, \"\", content, flags=re.DOTALL)\n",
        "      pattern =  r'<!--.*?-->'\n",
        "      content = re.sub(pattern, \"\", content, flags=re.DOTALL)\n",
        "\n",
        "      response = call_model(content)\n",
        "      response = bytes(response, 'utf-8').decode('unicode_escape')\n",
        "\n",
        "      # make outupt folder if it doesn't exist\n",
        "      if not os.path.exists(os.path.dirname(outputPath+filenamepre)):\n",
        "        os.makedirs(os.path.dirname(outputPath+filenamepre))\n",
        "\n",
        "      fileOutput = open(outputPath + filenamepre + \".json\", \"w\")\n",
        "      fileOutput.write(response)\n",
        "      fileOutput.close()\n",
        "\n",
        "end = time.time()\n",
        "print(f\"\\n\\n(took {round(end - start, 2)} s.):\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_R_PgFyzSbXh"
      },
      "source": [
        "## Archive files and download `qanda.tgz`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c6UFhaotAm8C",
        "outputId": "4556625c-ab18-4014-99e5-3001e5238d5f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tar: Removing leading `/' from member names\n",
            "/content/out/concepts/\n",
            "/content/out/concepts/configuration/\n",
            "/content/out/concepts/configuration/secret.3.json\n",
            "/content/out/concepts/configuration/secret.4.json\n",
            "/content/out/concepts/configuration/secret.2.json\n",
            "/content/out/concepts/configuration/secret.1.json\n"
          ]
        }
      ],
      "source": [
        "!tar -zcvf qanda.tgz /content/out/*\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Fixing the failed generations\n",
        "\n"
      ],
      "metadata": {
        "id": "q6RF7xOkVAx7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create Q and A in raw text for the docs that the original script just refused to output as json or got confused and started generating endless nonsense\n",
        "\n",
        "Create a file `kubedoclist-fix.txt` with list of file names to fix and output to json:\n",
        "```\n",
        "concepts/cluster-administration/flow-control\n",
        "concepts/configuration/manage-resources-containers\n",
        "concepts/extend-kubernetes/compute-storage-net/device-plugins\n",
        "concepts/security/security-checklist\n",
        "concepts/security/service-accounts\n",
        "concepts/services-networking/service\n",
        "concepts/storage/projected-volumes\n",
        "concepts/storage/volumes\n",
        "concepts/storage/persistent-volumes\n",
        "concepts/storage/storage-classes\n",
        "concepts/windows/intro\n",
        "concepts/workloads/controllers/deployment\n",
        "concepts/workloads/controllers/job\n",
        "reference/access-authn-authz/admission-controllers\n",
        "reference/access-authn-authz/authentication\n",
        "reference/access-authn-authz/certificate-signing-requests\n",
        "reference/access-authn-authz/extensible-admission-controllers\n",
        "reference/access-authn-authz/kubelet-tls-bootstrapping\n",
        "reference/access-authn-authz/rbac\n",
        "setup/best-practices/certificates\n",
        "setup/best-practices/cluster-large\n",
        "tasks/access-application-cluster/ingress-minikube\n",
        "tasks/administer-cluster/encrypt-data\n",
        "tasks/administer-cluster/kms-provider\n",
        "tasks/administer-cluster/memory-manager\n",
        "tasks/configure-pod-container/configure-service-account\n",
        "tasks/debug/debug-application/debug-service\n",
        "tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning\n",
        "tasks/extend-kubernetes/custom-resources/custom-resource-definitions\n",
        "tasks/manage-kubernetes-objects/declarative-config\n",
        "tasks/manage-kubernetes-objects/kustomization\n",
        "tasks/run-application/horizontal-pod-autoscale\n",
        "```\n",
        "Download each of the source documents and break into `flow-control.md.1`, `flow-control.md.2` and `flow-control.md.3` and place in the fix folder, the below script will produce q and a for the partial files and place them in the `out` folder as a json file, download these and merge into the fixed up json files - sadly you're going to have to fix these up by hand afterwards with the dataset gen script"
      ],
      "metadata": {
        "id": "hGAUGmhRVS6H"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## generate q and a pairs from fixed documents"
      ],
      "metadata": {
        "id": "fZbLkE9z7TVS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "import os\n",
        "import re\n",
        "import gc\n",
        "\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "from langchain_core.prompts import PromptTemplate\n",
        "\n",
        "from langchain.callbacks.manager import CallbackManager\n",
        "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
        "CallbackManager = CallbackManager([StreamingStdOutCallbackHandler()])\n",
        "\n",
        "from langchain_community.llms import LlamaCpp\n",
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "start = time.time()\n",
        "print(\"Processing started\")\n",
        "\n",
        "# paths to input and output\n",
        "inputPath = \"/content/website/content/en/docs/\"\n",
        "outputPath = \"/content/out/\"\n",
        "documentList = \"/content/kubedoclist-fix.txt\"\n",
        "\n",
        "# tidy up from previous run in collab\n",
        "rag_chain = None\n",
        "llm = None\n",
        "gc.collect()\n",
        "\n",
        "\n",
        "def create_llm():\n",
        "  return LlamaCpp(model_path=\"/content/openchat-3.5-0106.Q5_K_M.gguf\",  n_gpu_layers=128, temperature=0.2, max_tokens=32768, n_ctx=32768, n_batch=8, n_threads=8, verbose=True, callback_manager=CallbackManager)\n",
        "\n",
        "\n",
        "def create_prompt():\n",
        "  ## prompt inspired by https://helixml.substack.com/p/how-we-got-fine-tuning-mistral-7b?utm_source=pocket_saves\n",
        "  sys_prompt = \"\"\"\n",
        "  You are an intelligent professor. You create at least 5 question and answer pairs from the given document for your students. Respond with an array of strict JSON 'question' & 'answer' pairs.\n",
        "\n",
        "  For example:\n",
        "  ```json\n",
        "    [\n",
        "      {{\n",
        "        \"question\": \"…\",\n",
        "        \"answer\": \"…\"\n",
        "      }},\n",
        "      {{\n",
        "        \"question\": \"…\",\n",
        "        \"answer\": \"…\"\n",
        "      }}\n",
        "    ]\n",
        "  ```\n",
        "  \"\"\"\n",
        "  instruction = \"\"\"\n",
        "  Here is the document:\n",
        "\n",
        "  {content}\n",
        "  \"\"\"\n",
        "\n",
        "  prompt = \"GPT4 Correct User: \" + sys_prompt + \"\\n\\n\" + instruction + \"<|end_of_turn|>GPT4 Correct Assistant:\"\n",
        "  return PromptTemplate.from_template(template=prompt)\n",
        "\n",
        "\n",
        "def call_model(content):\n",
        "  response = rag_chain.invoke(content)\n",
        "  print(response)\n",
        "  return(response)\n",
        "\n",
        "\n",
        "# main\n",
        "fileDocs = open(documentList,\"r\")\n",
        "docList = fileDocs.read().splitlines()\n",
        "fileDocs.close()\n",
        "\n",
        "custom_rag_prompt = create_prompt()\n",
        "\n",
        "llm = create_llm()\n",
        "\n",
        "rag_chain = (\n",
        "{\"content\": RunnablePassthrough()}\n",
        "| custom_rag_prompt\n",
        "| llm\n",
        "| StrOutputParser()\n",
        ")\n",
        "\n",
        "for filenamepre in docList:\n",
        "  if not filenamepre.startswith('#') :\n",
        "    if os.path.exists(outputPath + filenamepre + \".json\"):\n",
        "      print(\"Skipping doc - file exists \" + filenamepre + \".json\")\n",
        "    else:\n",
        "      print(\"Processing doc \" + filenamepre + \".md\")\n",
        "\n",
        "      # open input file from documentList with appended \".md\"\n",
        "      fileInput = open(inputPath + filenamepre + \".md\", \"r\")\n",
        "      content = fileInput.read()\n",
        "      fileInput.close()\n",
        "\n",
        "      # remove hugo template stuff\n",
        "      pattern =  r'{{.*?}}'\n",
        "      content = re.sub(pattern, \"\", content, flags=re.DOTALL)\n",
        "      pattern =  r'<!--.*?-->'\n",
        "      content = re.sub(pattern, \"\", content, flags=re.DOTALL)\n",
        "\n",
        "      response = call_model(content)\n",
        "      response = bytes(response, 'utf-8').decode('unicode_escape')\n",
        "\n",
        "      #resplines = response.split(\"\\n\")\n",
        "      #response = \"\\n\".join(resplines)\n",
        "\n",
        "      # make outupt folder if it doesn't exist\n",
        "      if not os.path.exists(os.path.dirname(outputPath+filenamepre)):\n",
        "        os.makedirs(os.path.dirname(outputPath+filenamepre))\n",
        "\n",
        "      fileOutput = open(outputPath + filenamepre + \".json\", \"w\")\n",
        "      fileOutput.write(response)\n",
        "      fileOutput.close()\n",
        "\n",
        "end = time.time()\n",
        "print(f\"\\n\\n(took {round(end - start, 2)} s.):\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wjS8yUcWVrKh",
        "outputId": "a9e1b300-a9e3-4620-9d4c-81bc6c7d9859"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /content/openchat-3.5-0106.Q5_K_M.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = openchat_openchat-3.5-0106\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 8192\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
            "llama_model_loader: - kv  11:                          general.file_type u32              = 17\n",
            "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32002]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 32000\n",
            "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
            "llama_model_loader: - kv  22:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type q5_K:  193 tensors\n",
            "llama_model_loader: - type q6_K:   33 tensors\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Processing started\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "llm_load_vocab: special tokens definition check successful ( 261/32002 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32002\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 8192\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_embd_head_k    = 128\n",
            "llm_load_print_meta: n_embd_head_v    = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
            "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 0\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 10000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 8192\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: model type       = 8B\n",
            "llm_load_print_meta: model ftype      = Q5_K - Medium\n",
            "llm_load_print_meta: model params     = 7.24 B\n",
            "llm_load_print_meta: model size       = 4.78 GiB (5.67 BPW) \n",
            "llm_load_print_meta: general.name     = openchat_openchat-3.5-0106\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 32000 '<|end_of_turn|>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.30 MiB\n",
            "llm_load_tensors: offloading 32 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 33/33 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =    85.94 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  4807.06 MiB\n",
            "...................................................................................................\n",
            "llama_new_context_with_model: n_batch is less than GGML_KQ_MASK_PAD - increasing to 32\n",
            "llama_new_context_with_model: n_ctx      = 32768\n",
            "llama_new_context_with_model: n_batch    = 32\n",
            "llama_new_context_with_model: n_ubatch   = 32\n",
            "llama_new_context_with_model: flash_attn = 0\n",
            "llama_new_context_with_model: freq_base  = 10000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =  4096.00 MiB\n",
            "llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   134.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =     4.50 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1030\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
            "Model metadata: {'tokenizer.chat_template': \"{{ bos_token }}{% for message in messages %}{{ 'GPT4 Correct ' + message['role'].title() + ': ' + message['content'] + '<|end_of_turn|>'}}{% endfor %}{% if add_generation_prompt %}{{ 'GPT4 Correct Assistant:' }}{% endif %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '32000', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '8192', 'general.name': 'openchat_openchat-3.5-0106', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '17'}\n",
            "Using gguf chat template: {{ bos_token }}{% for message in messages %}{{ 'GPT4 Correct ' + message['role'].title() + ': ' + message['content'] + '<|end_of_turn|>'}}{% endfor %}{% if add_generation_prompt %}{{ 'GPT4 Correct Assistant:' }}{% endif %}\n",
            "Using chat eos_token: <|end_of_turn|>\n",
            "Using chat bos_token: <s>\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Processing doc concepts/configuration/secret.1.md\n",
            " [\n",
            "  {\n",
            "    \"question\": \"What is a Secret in Kubernetes?\",\n",
            "    \"answer\": \"A Secret in Kubernetes is an object that contains sensitive data such as passwords, tokens, or keys. It is used to store and manage confidential information without including it directly in the application code.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are some uses for Secrets?\",\n",
            "    \"answer\": \"Secrets can be used for purposes such as setting environment variables for a container, providing credentials like SSH keys or passwords to Pods, and allowing the kubelet to pull container images from private registries.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are some alternatives to using Secrets?\",\n",
            "    \"answer\": \"Alternatives to using Secrets include ServiceAccount tokens, third-party tools for managing sensitive data, custom signers for X.509 certificates, and device plugins for exposing node-local encryption hardware.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the default type of Secret in Kubernetes?\",\n",
            "    \"answer\": \"The default type of Secret in Kubernetes is Opaque if no type is explicitly specified. When creating a Secret using kubectl, you must use the generic subcommand to indicate an Opaque Secret type.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a ServiceAccount token Secret?\",\n",
            "    \"answer\": \"A ServiceAccount token Secret is used to store a token credential that identifies a ServiceAccount. This is a legacy mechanism for providing long-lived ServiceAccount credentials to Pods.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"How can you create a ServiceAccount token Secret?\",\n",
            "    \"answer\": \"You should only create a ServiceAccount token Secret if you can't use the TokenRequest API to obtain a token, and the security exposure of persisting a non-expiring token credential in a readable API object is acceptable to you. For instructions, see Manually create a long-lived API token for a ServiceAccount.\"\n",
            "  }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     316.11 ms\n",
            "llama_print_timings:      sample time =     264.78 ms /   454 runs   (    0.58 ms per token,  1714.66 tokens per second)\n",
            "llama_print_timings: prompt eval time =   34362.56 ms /  3008 tokens (   11.42 ms per token,    87.54 tokens per second)\n",
            "llama_print_timings:        eval time =   15573.85 ms /   454 runs   (   34.30 ms per token,    29.15 tokens per second)\n",
            "llama_print_timings:       total time =   53780.09 ms /  3462 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a Secret in Kubernetes?\",\n",
            "    \"answer\": \"A Secret in Kubernetes is an object that contains sensitive data such as passwords, tokens, or keys. It is used to store and manage confidential information without including it directly in the application code.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are some uses for Secrets?\",\n",
            "    \"answer\": \"Secrets can be used for purposes such as setting environment variables for a container, providing credentials like SSH keys or passwords to Pods, and allowing the kubelet to pull container images from private registries.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are some alternatives to using Secrets?\",\n",
            "    \"answer\": \"Alternatives to using Secrets include ServiceAccount tokens, third-party tools for managing sensitive data, custom signers for X.509 certificates, and device plugins for exposing node-local encryption hardware.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the default type of Secret in Kubernetes?\",\n",
            "    \"answer\": \"The default type of Secret in Kubernetes is Opaque if no type is explicitly specified. When creating a Secret using kubectl, you must use the generic subcommand to indicate an Opaque Secret type.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a ServiceAccount token Secret?\",\n",
            "    \"answer\": \"A ServiceAccount token Secret is used to store a token credential that identifies a ServiceAccount. This is a legacy mechanism for providing long-lived ServiceAccount credentials to Pods.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"How can you create a ServiceAccount token Secret?\",\n",
            "    \"answer\": \"You should only create a ServiceAccount token Secret if you can't use the TokenRequest API to obtain a token, and the security exposure of persisting a non-expiring token credential in a readable API object is acceptable to you. For instructions, see Manually create a long-lived API token for a ServiceAccount.\"\n",
            "  }\n",
            "]\n",
            "Processing doc concepts/configuration/secret.2.md\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a Secret in Kubernetes?\",\n",
            "    \"answer\": \"A Secret in Kubernetes is an object that contains a small amount of sensitive data such as passwords, tokens, or keys. It helps to keep confidential data out of your application code and provides additional security measures.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the steps to safely use Secrets?\",\n",
            "    \"answer\": \"To safely use Secrets, you can enable encryption at rest, configure RBAC rules with least-privilege access, restrict Secret access to specific containers, and consider using external Secret store providers.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between Docker config Secrets and basic authentication Secrets?\",\n",
            "    \"answer\": \"Docker config Secrets are used for storing credentials needed for accessing a container image registry, while basic authentication Secrets are used for storing data needed for basic authentication.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between SSH authentication Secrets and Opaque type Secrets?\",\n",
            "    \"answer\": \"SSH authentication Secrets are used for storing data used in SSH authentication, while Opaque type Secrets can be used for any kind of sensitive data.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the purpose of using a defined and public Secret type?\",\n",
            "    \"answer\": \"Using a defined and public Secret type helps others understand the purpose of your Secret, sets a convention for what key names to expect, and allows the Kubernetes API to verify that the required keys are set for a Secret of that type.\"\n",
            "  }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     316.11 ms\n",
            "llama_print_timings:      sample time =     210.98 ms /   365 runs   (    0.58 ms per token,  1730.01 tokens per second)\n",
            "llama_print_timings: prompt eval time =   15553.94 ms /  1296 tokens (   12.00 ms per token,    83.32 tokens per second)\n",
            "llama_print_timings:        eval time =   11984.08 ms /   365 runs   (   32.83 ms per token,    30.46 tokens per second)\n",
            "llama_print_timings:       total time =   30088.13 ms /  1661 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a Secret in Kubernetes?\",\n",
            "    \"answer\": \"A Secret in Kubernetes is an object that contains a small amount of sensitive data such as passwords, tokens, or keys. It helps to keep confidential data out of your application code and provides additional security measures.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the steps to safely use Secrets?\",\n",
            "    \"answer\": \"To safely use Secrets, you can enable encryption at rest, configure RBAC rules with least-privilege access, restrict Secret access to specific containers, and consider using external Secret store providers.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between Docker config Secrets and basic authentication Secrets?\",\n",
            "    \"answer\": \"Docker config Secrets are used for storing credentials needed for accessing a container image registry, while basic authentication Secrets are used for storing data needed for basic authentication.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the difference between SSH authentication Secrets and Opaque type Secrets?\",\n",
            "    \"answer\": \"SSH authentication Secrets are used for storing data used in SSH authentication, while Opaque type Secrets can be used for any kind of sensitive data.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is the purpose of using a defined and public Secret type?\",\n",
            "    \"answer\": \"Using a defined and public Secret type helps others understand the purpose of your Secret, sets a convention for what key names to expect, and allows the Kubernetes API to verify that the required keys are set for a Secret of that type.\"\n",
            "  }\n",
            "]\n",
            "Processing doc concepts/configuration/secret.3.md\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a Kubernetes Secret?\",\n",
            "    \"answer\": \"A Kubernetes Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. It helps to keep confidential data out of your application code and provides additional security measures.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the default storage locations for Kubernetes Secrets?\",\n",
            "    \"answer\": \"By default, Kubernetes Secrets are stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are some best practices for managing Kubernetes Secrets?\",\n",
            "    \"answer\": \"Some best practices for managing Kubernetes Secrets include enabling encryption at rest, configuring RBAC rules with least-privilege access, restricting Secret access to specific containers, and considering using external Secret store providers.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a TLS Secret?\",\n",
            "    \"answer\": \"A TLS Secret is a type of Kubernetes Secret used for storing a certificate and its associated key that are typically used for TLS. It is provided for convenience and helps ensure the consistency of Secret format in your project.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a bootstrap token Secret?\",\n",
            "    \"answer\": \"A bootstrap token Secret is a type of Kubernetes Secret used during the node bootstrap process. It stores tokens used to sign well-known ConfigMaps and is usually created in the kube-system namespace.\"\n",
            "  }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     316.11 ms\n",
            "llama_print_timings:      sample time =     219.18 ms /   370 runs   (    0.59 ms per token,  1688.15 tokens per second)\n",
            "llama_print_timings: prompt eval time =   24233.41 ms /  1885 tokens (   12.86 ms per token,    77.79 tokens per second)\n",
            "llama_print_timings:        eval time =   12674.49 ms /   369 runs   (   34.35 ms per token,    29.11 tokens per second)\n",
            "llama_print_timings:       total time =   39669.35 ms /  2254 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "  {\n",
            "    \"question\": \"What is a Kubernetes Secret?\",\n",
            "    \"answer\": \"A Kubernetes Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. It helps to keep confidential data out of your application code and provides additional security measures.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are the default storage locations for Kubernetes Secrets?\",\n",
            "    \"answer\": \"By default, Kubernetes Secrets are stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What are some best practices for managing Kubernetes Secrets?\",\n",
            "    \"answer\": \"Some best practices for managing Kubernetes Secrets include enabling encryption at rest, configuring RBAC rules with least-privilege access, restricting Secret access to specific containers, and considering using external Secret store providers.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a TLS Secret?\",\n",
            "    \"answer\": \"A TLS Secret is a type of Kubernetes Secret used for storing a certificate and its associated key that are typically used for TLS. It is provided for convenience and helps ensure the consistency of Secret format in your project.\"\n",
            "  },\n",
            "  {\n",
            "    \"question\": \"What is a bootstrap token Secret?\",\n",
            "    \"answer\": \"A bootstrap token Secret is a type of Kubernetes Secret used during the node bootstrap process. It stores tokens used to sign well-known ConfigMaps and is usually created in the kube-system namespace.\"\n",
            "  }\n",
            "]\n",
            "Processing doc concepts/configuration/secret.4.md\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "   {\n",
            "     \"question\": \"What is a Secret in Kubernetes?\",\n",
            "     \"answer\": \"A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key.\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"How are Secrets stored by default in Kubernetes?\",\n",
            "     \"answer\": \"By default, Secrets are stored unencrypted in the API server's underlying data store (etcd).\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"What is one way to access data from a Secret in a Pod?\",\n",
            "     \"answer\": \"One way to access data from a Secret in a Pod is to have Kubernetes make the value of that Secret be available as a file inside the filesystem of one or more of the Pod's containers.\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"What is an immutable Secret?\",\n",
            "     \"answer\": \"An immutable Secret is a Secret that has its data field marked as read-only, preventing accidental updates and improving cluster performance.\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"How can you configure least-privilege access to Secrets in Kubernetes?\",\n",
            "     \"answer\": \"You can annotate a ServiceAccount with the `kubernetes.io/enforce-mountable-secrets: \\\"true\\\"` annotation to enforce least-privilege access to Secrets.\"\n",
            "   }\n",
            "]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "llama_print_timings:        load time =     316.11 ms\n",
            "llama_print_timings:      sample time =     182.65 ms /   318 runs   (    0.57 ms per token,  1741.02 tokens per second)\n",
            "llama_print_timings: prompt eval time =   24004.24 ms /  1927 tokens (   12.46 ms per token,    80.28 tokens per second)\n",
            "llama_print_timings:        eval time =   11007.40 ms /   317 runs   (   34.72 ms per token,    28.80 tokens per second)\n",
            "llama_print_timings:       total time =   37437.07 ms /  2244 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " [\n",
            "   {\n",
            "     \"question\": \"What is a Secret in Kubernetes?\",\n",
            "     \"answer\": \"A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key.\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"How are Secrets stored by default in Kubernetes?\",\n",
            "     \"answer\": \"By default, Secrets are stored unencrypted in the API server's underlying data store (etcd).\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"What is one way to access data from a Secret in a Pod?\",\n",
            "     \"answer\": \"One way to access data from a Secret in a Pod is to have Kubernetes make the value of that Secret be available as a file inside the filesystem of one or more of the Pod's containers.\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"What is an immutable Secret?\",\n",
            "     \"answer\": \"An immutable Secret is a Secret that has its data field marked as read-only, preventing accidental updates and improving cluster performance.\"\n",
            "   },\n",
            "   {\n",
            "     \"question\": \"How can you configure least-privilege access to Secrets in Kubernetes?\",\n",
            "     \"answer\": \"You can annotate a ServiceAccount with the `kubernetes.io/enforce-mountable-secrets: \\\"true\\\"` annotation to enforce least-privilege access to Secrets.\"\n",
            "   }\n",
            "]\n",
            "\n",
            "\n",
            "(took 163.37 s.):\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Finally\n",
        "\n",
        "OK so you've managed to get all of the markdown files converted into valid json - you'll now appreciate the amount of manual work needed to get this working.\n",
        "\n",
        "Next step is to finetune the model, head back to https://github.com/andyburgin/kubefix-llm and see what's next"
      ],
      "metadata": {
        "id": "B1_Bexz1ioJb"
      }
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "authorship_tag": "ABX9TyNZSYKCJuMg+sLsXwPip/wN",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}