KnowledgeGraphInterface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb675d76-55cf-49b6-8705-e726a8f9980a",
   "metadata": {},
   "source": [
    "# Smart flash cards using LLMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3b2c89f7-5bcc-4229-9cf6-1410b5f8924a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-11T12:55:01.524881Z",
     "start_time": "2024-06-11T12:54:36.496075600Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import openai # make sure to pip install openai\n",
    "import re\n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from collections import Counter\n",
    "from typing import Any, Optional, Tuple, Dict, List, NamedTuple, Set\n",
    "import scipy\n",
    "import time\n",
    "\n",
    "from pprint import pprint as pprint\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['figure.dpi'] = 100\n",
    "\n",
    "import umap.umap_ as umap\n",
    "import hdbscan\n",
    "import plotly.express as px\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display\n",
    "import tkinter as tk\n",
    "from tkinter import ttk\n",
    "\n",
    "from basic_utils import *\n",
    "from basic_user_interface import *\n",
    "from atomic_card_processing import *\n",
    "from knowledge_graph import *\n",
    "from knowledge_graph_querying import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "# process my basic flashcards from online\n",
    "csv_title = 'my_flash_cards_general' \n",
    "verbose=True\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-06-11T12:55:07.045643100Z",
     "start_time": "2024-06-11T12:55:07.030485700Z"
    }
   },
   "id": "ae896bf5ea32fafa"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Process initial flashcards to extract info \n",
    "Load flashcards from CSV and create json file with saved meta data.\n",
    "\n",
    "This does not need to be rerun if 'my_flash_cards_general_cards_df_abstraction_groups' file exists."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "913eb68e-ee30-4db3-b609-398ff57e88e6"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c22d45a6-93ce-4e1a-9506-250e29720301",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_cards_df_abstraction_groups_from_front_and_back_csv(csv_title, verbose=verbose, start_ind=0)\n",
    "\n",
    "# save_cards_df_to_json(cards_df_abstraction_groups, csv_title + '_cards_df_abstraction_groups')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "merge_and_save(csv_title)\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3af6ca4732acc0b2"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Build a knowledge graph"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d9c36a71-1c2b-4b57-a070-7779b04e17e6"
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              Question  \\\n",
      "0    What is the definition of sorting in data stru...   \n",
      "1         What is the stability of sorting algorithms?   \n",
      "2    What are the main criteria for evaluating a so...   \n",
      "3    What is the difference between internal sortin...   \n",
      "4    What is the main difference between comparison...   \n",
      "..                                                 ...   \n",
      "404  How are the front and rear of a linked queue r...   \n",
      "405  What are the steps to implement enqueuing in a...   \n",
      "406  What are the steps to implement the dequeue op...   \n",
      "407  What is the special case of dequeue in a linke...   \n",
      "408  How to implement the destructor of a linked qu...   \n",
      "\n",
      "                                                Answer  \\\n",
      "0    Reorder the records according to the key code ...   \n",
      "1    If records with the same key code maintain the...   \n",
      "2    Time complexity, space complexity, and the com...   \n",
      "3    Internal sorting is the sorting completed in m...   \n",
      "4    Comparison sorting sorts by comparing key code...   \n",
      "..                                                 ...   \n",
      "404  The head node is used for dequeuing and deleti...   \n",
      "405  Create a new node s, s->data = x, s->next = NU...   \n",
      "406  Node<T> *p = front->next, front->next = p->nex...   \n",
      "407  If front->next == NULL, then rear points to fr...   \n",
      "408  Use a while loop to delete each node until fro...   \n",
      "\n",
      "                                             Key ideas  \\\n",
      "0    1. Sorting in data structures refers to the pr...   \n",
      "1    1. Sorting algorithms are methods used to arra...   \n",
      "2    1. Time complexity is the measure of the amoun...   \n",
      "3    1. Internal sorting is the process of sorting ...   \n",
      "4    1. Comparison sorting involves sorting element...   \n",
      "..                                                 ...   \n",
      "404  1. The front and rear of a linked queue are re...   \n",
      "405  1. Enqueuing is the process of adding an eleme...   \n",
      "406  1. The dequeue operation removes the front ele...   \n",
      "407  1. The special case of dequeue in a linked que...   \n",
      "408  1. Understanding of what a destructor is in ob...   \n",
      "\n",
      "                                    Abstraction groups Revision_history  \n",
      "0    {'-1': ['Sorting', 'Data Structure', 'Record',...      {'EF': 2.5}  \n",
      "1    {'-1': ['Sorting Algorithm', 'Stability', 'Rec...      {'EF': 2.5}  \n",
      "2    {'-1': ['Sorting Algorithm', 'Time Complexity'...      {'EF': 2.5}  \n",
      "3    {'-1': ['Internal Sorting', 'External Sorting'...      {'EF': 2.5}  \n",
      "4    {'-1': ['Comparison Sorting', 'Non-comparison ...      {'EF': 2.5}  \n",
      "..                                                 ...              ...  \n",
      "404  {'-1': ['Front', 'Rear', 'Linked Queue', 'Head...      {'EF': 2.5}  \n",
      "405  {'-1': ['Enqueuing', 'Linked Queue', 'New Node...      {'EF': 2.5}  \n",
      "406  {'-1': ['Dequeue', 'Linked Queue', 'Pointer', ...      {'EF': 2.5}  \n",
      "407  {'-1': ['Dequeue', 'Linked Queue', 'Front', 'R...      {'EF': 2.5}  \n",
      "408  {'-1': ['Destructor', 'Linked Queue', 'While L...      {'EF': 2.5}  \n",
      "\n",
      "[409 rows x 5 columns]\n",
      "Cards loaded: 409\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\L\\SR-KnowledgeGraph\\atomic_card_processing.py:86: FutureWarning: Passing literal json to 'read_json' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.\n",
      "  card_df_reloaded = pd.read_json(json.load(f), orient=\"index\")\n"
     ]
    }
   ],
   "source": [
    "# Reload basic cards to start to make knowledge graph \n",
    "cards_df = read_cards_df_from_json('34567_full_cards_df_abstraction_groups')\n",
    "print(cards_df)\n",
    "print(\"Cards loaded:\", len(cards_df))"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-06-11T12:55:36.204301600Z",
     "start_time": "2024-06-11T12:55:36.168012100Z"
    }
   },
   "id": "0f4f533e-5426-4334-b296-715903625cc9"
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0b914c04-00f8-44fc-9092-52aec22a5109",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-11T13:54:56.597465Z",
     "start_time": "2024-06-11T13:54:56.583700700Z"
    }
   },
   "outputs": [],
   "source": [
    "# Build a graph from real flashcard data \n",
    "kGraph = KnowledgeGraph(lower_bound_epsilon=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6830c4b5d74d2cc9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-11T13:59:33.974026100Z",
     "start_time": "2024-06-11T13:59:25.788929500Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding 409 cards\n",
      "Recalculating relative abstraction\n",
      "   Node number:  0 , Title: \"Recursive Function\" at time  0.0\n",
      "   Node number:  200 , Title: \"Single-Pass Sequential Search\" at time  0.56\n",
      "   Node number:  400 , Title: \"Logarithmic\" at time  1.05\n",
      "   Node number:  600 , Title: \"Threaded\" at time  1.58\n",
      "   Node number:  800 , Title: \"String Length\" at time  2.08\n",
      "   Node number:  1000 , Title: \"Open Flame\" at time  2.46\n",
      "   Node number:  1200 , Title: \"Dataset\" at time  3.04\n",
      "Updating raw embedding vectors\n",
      "   Node number:  0 , Title: \"Recursive Function\" at time  0.0\n",
      "   Node number:  200 , Title: \"Single-Pass Sequential Search\" at time  0.59\n",
      "   Node number:  400 , Title: \"Logarithmic\" at time  1.15\n",
      "   Node number:  600 , Title: \"Threaded\" at time  1.79\n",
      "   Node number:  800 , Title: \"String Length\" at time  2.35\n",
      "   Node number:  1000 , Title: \"Open Flame\" at time  2.86\n",
      "   Node number:  1200 , Title: \"Dataset\" at time  3.47\n"
     ]
    }
   ],
   "source": [
    "# Add card deck to kGraph\n",
    "card_deck = create_card_deck_from_dataframe_of_abstraction_groups(cards_df)\n",
    "title_list = kGraph.add_card_deck(card_deck, verbose=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updating all node embeddings:\n",
      "   Node number:  0 , Title: \"Science\" at time  0.0\n",
      "   Node number:  200 , Title: \"Frequency\" at time  7.26\n",
      "   Node number:  400 , Title: \"Representation\" at time  8.41\n",
      "   Node number:  600 , Title: \"Substring Index\" at time  8.82\n",
      "   Node number:  800 , Title: \"Circular queue\" at time  9.16\n",
      "   Node number:  1000 , Title: \"Complete Directed Graph\" at time  9.55\n",
      "   Node number:  1200 , Title: \"Unicode\" at time  9.98\n"
     ]
    }
   ],
   "source": [
    "kGraph.update_all_embeddings(verbose=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-06-11T13:59:49.984661600Z",
     "start_time": "2024-06-11T13:59:39.035045700Z"
    }
   },
   "id": "d71d4169-00f6-4330-a6a7-1cdd633f69c3"
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [
    "from knowledge_graph import *\n",
    "kGraph.save_final_embedding()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-06-11T13:24:37.033037700Z",
     "start_time": "2024-06-11T13:24:36.984271800Z"
    }
   },
   "id": "7fc1312847bc5d89"
  },
  {
   "cell_type": "markdown",
   "id": "025b3fe6-65ff-41d2-9b78-23ca05fa637c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Visualize graph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70b810ea-a8da-42f0-b26a-3f6cab12be91",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Build similarity matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "631637d2-d4fc-4880-b21c-f1f537a8f07b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the card-card overlap. This is a bit slow to run (20-30 seconds)\n",
    "\n",
    "cardIDs = np.array(list(kGraph.cards.keys()))\n",
    "cardIDs.sort()  # low to high\n",
    "similarity_metric = np.zeros((len(cardIDs), len(cardIDs)))\n",
    "\n",
    "name_labels = [kGraph.cards[cardID].topic for cardID in range(len(similarity_metric))]\n",
    "\n",
    "for cardID1 in cardIDs:\n",
    "    emb_vec1 = kGraph.cards[cardID1].embedding_vector_trimmed\n",
    "    similarity_metric[cardID1,cardID1] = 1.0  # diag is 1 by definition\n",
    "    for cardID2 in cardIDs:\n",
    "        if cardID2 > cardID1:\n",
    "            emb_vec2 = kGraph.cards[cardID2].embedding_vector_trimmed\n",
    "            inner_prod = emb_vec_inner_product(emb_vec1, emb_vec2)\n",
    "            similarity_metric[cardID1, cardID2] = (inner_prod + 1e-13)/(1.0 + 1e-13)\n",
    "            similarity_metric[cardID2, cardID1] = (inner_prod + 1e-13)/(1.0 + 1e-13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cdf3662-35ba-4bdc-9ed2-7b86a754ff9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,2, figsize=(6, 3))\n",
    "ax[0].imshow(similarity_metric, vmin=0, vmax=1, cmap='gnuplot2')\n",
    "ax[0].set_xlabel('Card index')\n",
    "ax[0].set_ylabel('Card index')\n",
    "ax[0].set_title('Similarity Metric')\n",
    "ax[1].hist(similarity_metric.flatten(), bins=100)\n",
    "ax[1].set_xlabel('Card overlap')\n",
    "ax[1].set_ylabel('Counts')\n",
    "ax[1].set_title('Similarity Histogram')\n",
    "ax[1].set_yscale('log')\n",
    "plt.subplots_adjust(wspace=0.4)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4f82186-391c-43e4-b0d9-03c77630d7eb",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Interactive clustering visualization\n",
    "Create a low dimensional clustered representation. Scan cursor over clusters to see what is in them. Adjust cluster size with slider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52e3d321-1428-41da-8da1-0fad8595641a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make and adjustable clustering algorithm based on desired size after UMAP\n",
    "\n",
    "# Convert similarity to distance\n",
    "distance_metric = 1 - similarity_metric\n",
    "\n",
    "# Apply UMAP dimensionality reduction\n",
    "reducer = umap.UMAP(metric=\"precomputed\", random_state=42)\n",
    "embedding = reducer.fit_transform(distance_metric)\n",
    "\n",
    "def plot_clusters(min_cluster_size):\n",
    "    # Perform HDBSCAN clustering\n",
    "    clusterer = hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, gen_min_span_tree=True)\n",
    "    cluster_labels = clusterer.fit_predict(embedding)\n",
    "\n",
    "    # Create an interactive plot\n",
    "    fig = px.scatter(x=embedding[:, 0], y=embedding[:, 1], color=cluster_labels, hover_name=name_labels,\n",
    "                     color_continuous_scale=px.colors.qualitative.Dark24, width=600, height=600)\n",
    "    fig.update_traces(marker=dict(size=8))\n",
    "    fig.show()\n",
    "\n",
    "\n",
    "\n",
    "# Create an interactive slider widget to control the min_cluster_size parameter\n",
    "slider = widgets.IntSlider(value=5, min=2, max=50, step=1, description='Min Cluster Size:')\n",
    "widgets.interact(plot_clusters, min_cluster_size=slider)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "739ce39e-dd0d-4525-a236-462892d36f08",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Display a few nodes/cards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7f77e30-b1e9-4e24-982d-14f4413a18c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "node_title = 'Quicksort'\n",
    "node = kGraph.nodes[node_title]\n",
    "kGraph.display_object_overlaps(node)\n",
    "\n",
    "card = kGraph.cards[4]\n",
    "card.display(verbose=True)     \n",
    "kGraph.display_object_overlaps(card)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e3cc12-d54c-45d6-b952-0eb965d30122",
   "metadata": {},
   "source": [
    "## User interface using ChatGPT\n",
    "Added a chatGPT user interface to generate, select, answer, and save new questions. \n",
    "For details see basic_user_interface.py files\n",
    "\n",
    "#### Generating new questions (flexible configuration):\n",
    "1. Generate further questions based on desired topic, and desired goal. This generates questions based on clusters and style of existing flashcards in graph.\n",
    "2. Generate further questions based on existing new questions\n",
    "3. Generate further questions based on similar questions already in graph (viewed in a separate window). \n",
    "\n",
    "#### Triage new questions:\n",
    "1. Choose which new questions to keep and explore further.\n",
    "2. Generate new questions about similar topics.\n",
    "3. Ultimately select which questions to answer.\n",
    "\n",
    "#### Answering questions:\n",
    "1. Use ChatGPT to answer the desired new questions. \n",
    "2. View and edit these answers and questions to desired accuracy.\n",
    "3. Save them to the knowledge graph and update embeddings, and also save to original long term JSON file storage. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f37e6c4d-648c-428f-8db0-059405f5c750",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Launch explorer and follow instructions within\n",
    "exploration_data = launch_explorer(kGraph, cards_df) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565368b7-8b21-4500-9fd4-2b31f124cd63",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## OLD method: Query knowledge graph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00f48204-c6e4-4443-a5cc-6809f42660d2",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### Question examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04589314-4912-4e2c-9417-a042754b5692",
   "metadata": {},
   "outputs": [],
   "source": [
    "################## Example questions ##################\n",
    "\n",
    "# Actual flashcards, or very close. It seems to usually solve these.\n",
    "# \"What is the most basic model of how natural environments determine major animal types in a given biome?\"\n",
    "# \"What are good practices for a function input and output when coding?\"\n",
    "# \"What are two major problems with existing LLMs like ChatGPT?\"\n",
    "# \"In the stratego paper, what is the regularized nash dynamics algorithm most similar to in other reinforcement learning algorithms?\"\n",
    "\n",
    "# Hard\n",
    "# \"If I am trying to fit a function, and it isn't very accurate, what method can I use to improve accuracy?\"\n",
    "\n",
    "# Easier\n",
    "# \"Why is boosting useful?\"\n",
    "# \"Why do language models sometimes give wildly wrong answers when predicting text?\"\n",
    "# \"What is the best way to learn new information about a new field of science?\"\n",
    "# \"Why should I care about determinants?\"\n",
    "# \"What happens in the drug approval process?\"\n",
    "# \"What major event happened in Texas in 2021?\"\n",
    "# \"How can I find the minimum of a function with a computer program?\"\n",
    "# \"What probability distribution deals with repeated draws from a set with replacement?\"\n",
    "# \"How hard is it to calculate the determinant of a matrix?\"\n",
    "# \"What are ways to impove the scientific funding process?\"\n",
    "# \"What is a multimodal neuron in a neural network?\"\n",
    "# \"What will the probability distribution be if I count how many times something with a fixed rate happens over some period of time?\"\n",
    "# \"When did the modern plants come into existence in evolutionary history?\"\n",
    "# \"What is gradient descent?\"\n",
    "# \"What is the sticky engine of growth for a startup?\"\n",
    "# \"What is Jim Allison famous for?\"\n",
    "# \"What modern encryption system works pretty well?\"\n",
    "# \"How can I sort a list in python?\"\n",
    "# \"How does a 3D metals printer work?\"\n",
    "\n",
    "# Not in distribution\n",
    "# \"When did the modern coral come into existence in evolutionary history?\" # testing a case that wasn't present \n",
    "# \"What causes the band structure of atoms in a periodic lattice to be sinusoidal?\"\n",
    "# \"What remains to be determined about the structure of fermion pairs in the spin-imbalanced Hubbard model?\"\n",
    "\n",
    "# Fake information\n",
    "# \"A fleegdorf is stuck in my garage, what do I do?\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd6f485-24d4-401f-bf01-fc14ccb166a8",
   "metadata": {},
   "source": [
    "##### Exploring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c695ab47-a985-4f77-b488-efbddef53791",
   "metadata": {},
   "outputs": [],
   "source": [
    "flashcardQuestion = \"What happens in the drug approval process?\"\n",
    "num_cards_to_show = 10\n",
    "verbose = True\n",
    "extra_verbose = True\n",
    "outside_knowledge_allowed=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa2eaad6-48ce-4ee6-a73e-86d7831b6065",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gather the initial subject list \n",
    "question_subject_list = get_refined_subject_list_from_question(flashcardQuestion, kGraph,\n",
    "                                                               num_cards_to_show=num_cards_to_show, verbose=verbose,\n",
    "                                                              extra_verbose=extra_verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b03dbd0a-d0da-42c1-9461-cd32f7994310",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer the question\n",
    "flashcardAnswer = get_answer_from_question_with_subject_list(flashcardQuestion, question_subject_list, kGraph,\n",
    "                                          num_cards_to_show=num_cards_to_show, \n",
    "                                                             outside_knowledge_allowed=outside_knowledge_allowed,\n",
    "                                                             verbose=verbose,\n",
    "                                                              extra_verbose=extra_verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49640c1d-2fdd-434c-ba3e-a053f3ffb6fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "generated_further_question_list = get_suggested_further_questions_from_question_and_subject_list(flashcardQuestion, question_subject_list, kGraph,  \n",
    "                                                               num_seed_cards_to_show=4,\n",
    "                                                               num_related_cards_to_show=5,\n",
    "                                                               num_questions_to_generate=4,\n",
    "                                                                                         increasing_abstraction=True,\n",
    "                                                                                                 temperature=0.25,\n",
    "                                                               verbose=verbose,\n",
    "                                                              extra_verbose=extra_verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0034c1c7-7d4e-42e7-8207-dce93eda78ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enhance the detail of the question\n",
    "flashcardQuestion_enhanced = get_enhanced_question_from_question_and_subject_list(flashcardQuestion, question_subject_list, kGraph,\n",
    "                                                        num_cards_to_show=num_cards_to_show, verbose=verbose,\n",
    "                                                              extra_verbose=extra_verbose)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adcb9625-d0d4-48dc-8800-122b6bbb09c2",
   "metadata": {},
   "source": [
    "#### Make entire flashcard better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72376565-cce9-4b8b-ae49-3643c582dbb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enhance the detail of the question\n",
    "cardID = 401\n",
    "flashcardQuestion = kGraph.cards[cardID].question\n",
    "flashcardAnswer = kGraph.cards[cardID].answer\n",
    "# flashcardConceptList = kGraph.cards[cardID].concepts.get_concepts_list()\n",
    "print(flashcardQuestion)\n",
    "# print(flashcardConceptList)\n",
    "print(flashcardAnswer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f05d7328-730d-4a79-8416-5dc3b410b1f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gather the initial subject list \n",
    "question_subject_list = get_refined_subject_list_from_question(flashcardQuestion, kGraph,\n",
    "                                                               num_cards_to_show=num_cards_to_show, verbose=verbose,\n",
    "                                                              extra_verbose=extra_verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "673fdd76-f2ae-4cc8-a445-df43e1026675",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make updated phrasing\n",
    "flashcard_enhanced = get_enhanced_flashcard_from_question_and_answer_and_subject_list(flashcardQuestion, flashcardAnswer, \n",
    "                                                                                      question_subject_list, kGraph,\n",
    "                                                        num_cards_to_show=num_cards_to_show, verbose=verbose,\n",
    "                                                              extra_verbose=extra_verbose)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2be83daf-8848-4079-ad70-5512ca7273a5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## OLD method: Cluster cards into a hierarchy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f56a80d4-cfbc-4103-93c6-3a805986d69a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Functions for first clustering within a similarity metric\n",
    "\n",
    "def get_ordered_IDs_by_similarity_to_ID(similarity_metric, cluster_IDs_list, target_ID):\n",
    "    similarity_vals_presort = similarity_metric[target_ID][cluster_IDs_list]\n",
    "    ordered_ID_inds = np.flip(np.argsort(similarity_vals_presort))\n",
    "    ordered_IDs = np.array(cluster_IDs_list)[ordered_ID_inds]\n",
    "    return ordered_IDs\n",
    "\n",
    "def get_sub_similarity_metric(similarity_metric, sub_inds_list):\n",
    "    return np.copy(similarity_metric[sub_inds_list][:,sub_inds_list])\n",
    "\n",
    "def get_cluster_quality(similarity_metric, cluster_IDs_list, target_ID, cluster_size_scaling=4.0):\n",
    "    \n",
    "    def get_log_ratio_vals(submat_self_similarities):\n",
    "\n",
    "        def get_array_without_diag(A):\n",
    "            return A[~np.eye(A.shape[0],dtype=bool)].reshape(A.shape[0],-1)\n",
    "        # for mean values, may want to remove diagonal\n",
    "        submat_self_similarities_offdiag = get_array_without_diag(submat_self_similarities)\n",
    "        off_diag_mean_vals = submat_self_similarities_offdiag.mean(axis=0)\n",
    "\n",
    "        min_vals = submat_self_similarities.min(axis=0)\n",
    "        # mean_vals = submat_self_similarities.mean(axis=0)\n",
    "\n",
    "        zip_min_mean_vals = zip(min_vals, off_diag_mean_vals)\n",
    "        log_ratio_vals = np.array([np.log(min_val/mean_val) \n",
    "                                   for min_val, mean_val in zip_min_mean_vals])\n",
    "        return log_ratio_vals\n",
    "    \n",
    "    similarities_to_target = similarity_metric[target_ID][cluster_IDs_list]\n",
    "    avg_similarity_to_target = np.average(similarities_to_target)\n",
    "    \n",
    "    submat_of_cluster = get_sub_similarity_metric(similarity_metric, cluster_IDs_list)\n",
    "    log_ratio_vals = get_log_ratio_vals(submat_of_cluster)\n",
    "    log_scaling = 1.0/cluster_size_scaling\n",
    "    return avg_similarity_to_target * np.log(len(cluster_IDs_list)) + log_scaling * np.sum(log_ratio_vals)\n",
    "\n",
    "def get_best_approximate_cluster_relative_to_ID_with_desired_size(similarity_metric, target_ID, \n",
    "                                                         cluster_size_scaling=4.0, \n",
    "                                                         verbose=False):\n",
    "    max_IDs_to_check = 50\n",
    "    ordered_IDs = get_ordered_IDs_by_similarity_to_ID(similarity_metric, range(len(similarity_metric)), target_ID)\n",
    "\n",
    "    # Get approximate best first order cluster \n",
    "    quality_vals = []\n",
    "    ordered_ind_max_vals = []\n",
    "    ordered_ind_sets = []\n",
    "    for ordered_ind_max in range(max_IDs_to_check):\n",
    "        cluster_IDs_list = ordered_IDs[0:ordered_ind_max+1]\n",
    "        quality_metric = get_cluster_quality(similarity_metric, cluster_IDs_list, target_ID, \n",
    "                                                               cluster_size_scaling=cluster_size_scaling)\n",
    "        # Append measurement\n",
    "        quality_vals.append(quality_metric)\n",
    "        ordered_ind_max_vals.append(ordered_ind_max)\n",
    "        ordered_ind_sets.append(cluster_IDs_list)\n",
    "        \n",
    "        if quality_metric < 0: # Then we will probably just get worse from here. Stop\n",
    "            break\n",
    "    \n",
    "    # Find argmax \n",
    "    best_quality_ind = np.argmax(quality_vals)\n",
    "    best_set_initial_pass = ordered_ind_sets[best_quality_ind]\n",
    "    current_best_quality_val = quality_vals[best_quality_ind]\n",
    "    \n",
    "    if verbose:\n",
    "        print('Initial results, with best quality value:', current_best_quality_val, \n",
    "              ' and set:', best_set_initial_pass)\n",
    "        plt.plot(ordered_ind_max_vals, quality_vals)\n",
    "        plt.show()\n",
    "        \n",
    "    return best_set_initial_pass\n",
    "\n",
    "def get_best_revised_cluster_relative_to_ID_with_desired_size(similarity_metric, target_ID, best_set_initial_guess, \n",
    "                                             num_max_revisit_passes=50,\n",
    "                                             cluster_size_scaling=4.0, \n",
    "                                             verbose=False):\n",
    "    how_much_larger_set_to_check = 2\n",
    "    \n",
    "    best_set = best_set_initial_guess\n",
    "    ordered_IDs = get_ordered_IDs_by_similarity_to_ID(similarity_metric, range(len(similarity_metric)), target_ID)\n",
    "    max_IDs_to_check = len(best_set) * how_much_larger_set_to_check\n",
    "\n",
    "    # Loop through and check if adding or removing single cards is worse or better. \n",
    "    for _iteration in range(num_max_revisit_passes):       \n",
    "        \n",
    "        current_best_quality_val = get_cluster_quality(similarity_metric, list(best_set), target_ID,\n",
    "                                                                       cluster_size_scaling=cluster_size_scaling)\n",
    "        if verbose:\n",
    "            print('iteration number', _iteration, ' with current best set: ', best_set, ' and best quality val', current_best_quality_val)\n",
    "        \n",
    "        quality_vals = []\n",
    "        testID_vals = []\n",
    "        new_test_sets = []\n",
    "        for ordered_ind, testID in enumerate(ordered_IDs[0:max_IDs_to_check]):\n",
    "            new_test_set = np.copy(best_set)\n",
    "            if testID in best_set:\n",
    "                new_test_set = np.setdiff1d(new_test_set, testID)\n",
    "            else:\n",
    "                new_test_set = np.append(new_test_set, testID)\n",
    "\n",
    "            if len(new_test_set) > 0: # only consider new sets that have nonzero size\n",
    "                cluster_IDs_list = list(new_test_set)\n",
    "                quality_metric = get_cluster_quality(similarity_metric, cluster_IDs_list, target_ID,\n",
    "                                                                       cluster_size_scaling=cluster_size_scaling)\n",
    "\n",
    "                quality_vals.append(quality_metric)\n",
    "                testID_vals.append(testID)\n",
    "                new_test_sets.append(new_test_set)\n",
    "\n",
    "        \n",
    "        if np.max(quality_vals - current_best_quality_val) > 0:\n",
    "            changed_ID = testID_vals[np.argmax(quality_vals)]\n",
    "            if verbose:\n",
    "                if changed_ID in best_set:\n",
    "                    print('  removed card ',changed_ID)\n",
    "                else:\n",
    "                    print('  added card ',changed_ID)\n",
    "            best_set = new_test_sets[np.argmax(quality_vals)]\n",
    "            current_best_quality_val = np.max(quality_vals)\n",
    "        else:\n",
    "            # There was no improvement for any proposed change, so stop iterating \n",
    "            break\n",
    "        \n",
    "    ordered_IDs_list_result = get_ordered_IDs_by_similarity_to_ID(similarity_metric, list(best_set), target_ID)\n",
    "\n",
    "    return ordered_IDs_list_result\n",
    "\n",
    "\n",
    "def get_best_first_order_cluster_relative_to_ID_with_desired_size(similarity_metric, target_ID, \n",
    "                                             num_max_revisit_passes=50,\n",
    "                                             cluster_size_scaling=4.0, \n",
    "                                             verbose=False):\n",
    "\n",
    "    best_set_initial_pass = get_best_approximate_cluster_relative_to_ID_with_desired_size(similarity_metric, target_ID, \n",
    "                                                                     cluster_size_scaling=cluster_size_scaling,\n",
    "                                                                     verbose=verbose)\n",
    "    \n",
    "    ordered_IDs_result = get_best_revised_cluster_relative_to_ID_with_desired_size(similarity_metric, target_ID, best_set_initial_pass, \n",
    "                                                 num_max_revisit_passes=num_max_revisit_passes,\n",
    "                                                 cluster_size_scaling=cluster_size_scaling, \n",
    "                                                 verbose=verbose)\n",
    "    \n",
    "    return ordered_IDs_result\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe546b21-636f-4855-8ddf-2b4e2b54f976",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for high level clustering of a similarity metric\n",
    "\n",
    "def remove_complete_subsets(clusters_IDgroups_list):\n",
    "    # Removes clusters that are a complete subset of another cluster.\n",
    "    # Returns clusters in order of decreasing size\n",
    "    \n",
    "    # Get clusters in order of increasing size. \n",
    "    cluster_inds_increasing_size = np.argsort(np.array([len(cluster_IDgroup) for cluster_IDgroup in clusters_IDgroups_list]))\n",
    "    ordered_clusters_IDgroups_list = [clusters_IDgroups_list[cluster_ind] for cluster_ind in cluster_inds_increasing_size]\n",
    "\n",
    "    # For each cluster, loop through other clusters and check if its a strict subset\n",
    "    # If it is not a strict subset of any, then keep it in the final list. \n",
    "    final_clusters_IDgroups_list = []\n",
    "    for cluster_ind_test, cluster_IDgroup_test in enumerate(ordered_clusters_IDgroups_list):\n",
    "        is_subset = False\n",
    "        for cluster_IDgroup_potential_superset in ordered_clusters_IDgroups_list[cluster_ind_test + 1:]: # only loop over larger ones\n",
    "            if set(cluster_IDgroup_test).issubset(set(cluster_IDgroup_potential_superset)):\n",
    "                is_subset = True \n",
    "        if not is_subset:\n",
    "            final_clusters_IDgroups_list.append(cluster_IDgroup_test)\n",
    "    \n",
    "    final_clusters_IDgroups_list.reverse() # report in order of decreasing size at end \n",
    "    \n",
    "    return final_clusters_IDgroups_list\n",
    "\n",
    "def get_best_clusters_with_desired_size(similarity_metric,\n",
    "                                             meta_cluster_size_scaling=0.75, # sets the overall precision of clusters. Less precise = larger clusters\n",
    "                                             verbose=False):\n",
    "    \n",
    "    # Finds best clusters by scanning the sensitivity of the first order clustering metric\n",
    "    # and finding point where it is best quality. \n",
    "\n",
    "    # Set up some basic parameters \n",
    "    cluster_score_tolerance = 0.02\n",
    "    cluster_size_scaling_pow_min = -5\n",
    "    cluster_size_scaling_pow_max = 6\n",
    "    cluster_size_scaling_arr = 2 ** np.linspace(cluster_size_scaling_pow_min, cluster_size_scaling_pow_max, \n",
    "                                                2*(cluster_size_scaling_pow_max - cluster_size_scaling_pow_min + 1))\n",
    "\n",
    "    significant_clusters_unique_tuple_id_list = []\n",
    "    significant_clusters_IDgroups_list = []\n",
    "    significant_cluster_score_list = []\n",
    "\n",
    "    for target_ID in range(len(similarity_metric)):\n",
    "\n",
    "        cluster_score_vs_cluster_size = []\n",
    "        cluster_ID_list_vs_cluster_size = []\n",
    "        prev_scaled_cluster_score = -1 # for reference \n",
    "\n",
    "        # Loop through various low level clustering target sizes to find one with best exclusion around this ID\n",
    "        for cluster_size_scaling in cluster_size_scaling_arr:\n",
    "            proposed_ID_list_this_cluster_size = get_best_first_order_cluster_relative_to_ID_with_desired_size(similarity_metric, target_ID, \n",
    "                                                             cluster_size_scaling=cluster_size_scaling, \n",
    "                                                             verbose=verbose)\n",
    "\n",
    "            # Get clusters around each of the proposed items in this new cluster \n",
    "            proposed_reverse_ID_sets_this_cluster_size = []\n",
    "            for proposed_ID in proposed_ID_list_this_cluster_size:\n",
    "                proposed_reverse_ID_list_this_cluster_size = get_best_first_order_cluster_relative_to_ID_with_desired_size(similarity_metric, proposed_ID, \n",
    "                                                                 cluster_size_scaling=cluster_size_scaling, \n",
    "                                                                 verbose=verbose)\n",
    "                proposed_reverse_ID_sets_this_cluster_size.append(proposed_reverse_ID_list_this_cluster_size)\n",
    "\n",
    "            # Gather metric for success rate\n",
    "            proposed_num_IDs = len(proposed_ID_list_this_cluster_size)\n",
    "            inclusion_success_rate_list = []\n",
    "            exclusion_success_rate_list = []\n",
    "\n",
    "            for proposed_reverse_ID_set in proposed_reverse_ID_sets_this_cluster_size:\n",
    "                reference_set = set(proposed_ID_list_this_cluster_size)\n",
    "                test_set = set(proposed_reverse_ID_set)\n",
    "                extra_IDs = test_set.difference(reference_set)\n",
    "                missing_IDs = reference_set.difference(test_set)\n",
    "\n",
    "                min_success_val_cutoff = 1e-10\n",
    "                exclusion_success_rate_list.append(max(1.0 -len(extra_IDs) / proposed_num_IDs, min_success_val_cutoff))\n",
    "                inclusion_success_rate_list.append(max(1.0 -len(missing_IDs) / proposed_num_IDs, min_success_val_cutoff))\n",
    "\n",
    "            exclusion_success_rate_list = np.array(exclusion_success_rate_list)\n",
    "            inclusion_success_rate_list = np.array(inclusion_success_rate_list)\n",
    "            inclusion_log_success_min = np.log(np.min(inclusion_success_rate_list))\n",
    "            exclusion_log_success_min = np.log(np.min(exclusion_success_rate_list))\n",
    "            meta_log_scaling = 1.0/meta_cluster_size_scaling\n",
    "\n",
    "            scaled_cluster_score = np.log(proposed_num_IDs)  + meta_log_scaling * (inclusion_log_success_min + exclusion_log_success_min)/2.0\n",
    "\n",
    "            cluster_score_vs_cluster_size.append(scaled_cluster_score)\n",
    "            cluster_ID_list_vs_cluster_size.append(proposed_ID_list_this_cluster_size)\n",
    "\n",
    "            if scaled_cluster_score < prev_scaled_cluster_score : # probably not going to get better\n",
    "                break\n",
    "            prev_scaled_cluster_score = scaled_cluster_score\n",
    "\n",
    "        # Gather result and take best quality cluster within tolerance\n",
    "        cluster_score_vs_cluster_size = np.array(cluster_score_vs_cluster_size)\n",
    "        max_cluster_score = np.max(cluster_score_vs_cluster_size)\n",
    "        cluster_ind_with_max_score = np.max(np.where(np.abs(cluster_score_vs_cluster_size - max_cluster_score) < cluster_score_tolerance)[0])\n",
    "        best_cluster_ID_list = cluster_ID_list_vs_cluster_size[cluster_ind_with_max_score]\n",
    "\n",
    "        if max_cluster_score < 0:  # Ignore really bad clusters. Replace with the original ID only. \n",
    "            max_cluster_score = 0\n",
    "            best_cluster_ID_list = [target_ID]\n",
    "\n",
    "        cluster_unique_tuple_id = tuple(sorted(best_cluster_ID_list)) # for checking if this exact cluster is already found at this level\n",
    "        \n",
    "\n",
    "        if not (cluster_unique_tuple_id in significant_clusters_unique_tuple_id_list):\n",
    "            # Then we found a new cluster, so keep it \n",
    "            significant_clusters_unique_tuple_id_list.append(cluster_unique_tuple_id)\n",
    "            significant_clusters_IDgroups_list.append(best_cluster_ID_list)\n",
    "            significant_cluster_score_list.append(max_cluster_score)\n",
    "            \n",
    "    # Lastly, loop back through and check if each cluster is a strict subset of all the other ones. \n",
    "    significant_clusters_IDgroups_list = remove_complete_subsets(significant_clusters_IDgroups_list)\n",
    "    \n",
    "    print('  Clusters identified: ', len(significant_clusters_IDgroups_list))\n",
    "\n",
    "    # significant_clusters_IDgroups_list is a list of the IDs relative to THIS similarity metric! \n",
    "    return significant_clusters_IDgroups_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acca8c06-8693-4e94-8077-a28f15978ee7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Functions to extract amd combine embeddings, and similarity metric\n",
    "\n",
    "# def get_emb_vec_list_from_cluster_cardIDs(prev_significant_clusters_cardIDs_list,\n",
    "#                                            intersection_threshold_amount, knowledgeGraph):\n",
    "#     # Goes through and gets the ID group for each new cluster\n",
    "#     # Then combines data from all stuff contained in that group. \n",
    "\n",
    "#     global_emb_vec_list = []\n",
    "#     for cluster_cardIDs in prev_significant_clusters_cardIDs_list:\n",
    "#         # Gather stuff from within this card \n",
    "#         emb_vec_list = [knowledgeGraph.cards[_cardID].embedding_vector_trimmed for _cardID in cluster_cardIDs]\n",
    "#         cluster_emb_vec_untrimmed = emb_vec_intersection_with_threshold(emb_vec_list, knowledgeGraph, \n",
    "#                                                                         intersection_threshold_amount)\n",
    "#         cluster_emb_vec = trim_embedding_vector(cluster_emb_vec_untrimmed)\n",
    "#         global_emb_vec_list.append(cluster_emb_vec)\n",
    "        \n",
    "#     return global_emb_vec_list\n",
    "\n",
    "\n",
    "def get_emb_vec_list_from_cluster_cardIDs(prev_significant_clusters_cardIDs_list, knowledgeGraph):\n",
    "    # Goes through and gets the ID group for each new cluster\n",
    "    # Then combines data from all stuff contained in that group. \n",
    "\n",
    "    global_emb_vec_list = []\n",
    "    for cluster_cardIDs in prev_significant_clusters_cardIDs_list:\n",
    "        \n",
    "        # Gather unique concepts mentioned anywhere in this cluster. \n",
    "            # If mentioned multiple times, only counts once\n",
    "        concept_lists = [knowledgeGraph.cards[_cardID].concepts.get_concepts_list() for _cardID in cluster_cardIDs]\n",
    "        concept_set = set()\n",
    "        for concept_list in concept_lists:\n",
    "            concept_set.update(set(concept_list))\n",
    "        concept_list = list(concept_set)\n",
    "        \n",
    "        # Get embedding vector \n",
    "        cluster_emb_vec_untrimmed = emb_vec_weighted_union_of_nodes(concept_list, knowledgeGraph)\n",
    "        cluster_emb_vec = trim_embedding_vector(cluster_emb_vec_untrimmed)\n",
    "        global_emb_vec_list.append(cluster_emb_vec)\n",
    "        \n",
    "    return global_emb_vec_list\n",
    "\n",
    "def get_similarity_metric_from_emb_vec_list(emb_vec_list):\n",
    "    similarity_metric = np.zeros((len(emb_vec_list), len(emb_vec_list)))\n",
    "\n",
    "    for ind1, emb_vec1 in enumerate(emb_vec_list):\n",
    "        similarity_metric[ind1,ind1] = 1.0  # diag is 1 by definition\n",
    "        for ind2, emb_vec2 in enumerate(emb_vec_list):\n",
    "            if ind2 > ind1:\n",
    "                inner_prod = emb_vec_inner_product(emb_vec1, emb_vec2)\n",
    "                similarity_metric[ind1, ind2] = (inner_prod + 1e-13)/(1.0 + 1e-13)\n",
    "                similarity_metric[ind2, ind1] = (inner_prod + 1e-13)/(1.0 + 1e-13)\n",
    "                \n",
    "    return similarity_metric\n",
    "\n",
    "def get_new_cluster_cardIDs(this_level_IDgroups, prev_level_cluster_cardIDs):\n",
    "    # Takes IDs of clusters in the similarity metric, and the cardIDs corresponding to those IDs from the previous layer,\n",
    "    # and returns the new list of cardIDs in the new clusters\n",
    "    \n",
    "    this_level_cluster_cardIDs = []\n",
    "    for cluster_IDgroup in this_level_IDgroups:\n",
    "        cardID_set = set()\n",
    "        for similarityID in cluster_IDgroup:\n",
    "            cardID_set.update(set(prev_level_cluster_cardIDs[similarityID]))\n",
    "        this_level_cluster_cardIDs.append(list(cardID_set))\n",
    "    return this_level_cluster_cardIDs\n",
    "\n",
    "\n",
    "def display_clustering_metrics(similarity_metric, clusters_IDgroups_list, clusters_cardIDs_list,\n",
    "                              hierarchy_level):\n",
    "    \n",
    "    cluster_IDgroups_sizes = np.array([len(_clu) for _clu in clusters_IDgroups_list])\n",
    "    cluster_cardIDs_sizes = np.array([len(_clu) for _clu in clusters_cardIDs_list])\n",
    "    \n",
    "    fig, ax = plt.subplots(1,4, figsize=(8, 1.5))\n",
    "    titlesize=9\n",
    "    ax[0].imshow(similarity_metric, vmin=0, vmax=1, cmap='gnuplot2')\n",
    "    ax[0].set_xlabel('ID')\n",
    "    ax[0].set_ylabel('ID')\n",
    "    ax[0].set_title('Level '+str(hierarchy_level) +'\\nSimilarity Metric', size = titlesize)\n",
    "    ax[1].hist(similarity_metric.flatten(), bins=100)\n",
    "    ax[1].set_xlabel('ID overlap')\n",
    "    ax[1].set_ylabel('ID Counts')\n",
    "    ax[1].set_yscale('log')\n",
    "    ax[2].hist(cluster_IDgroups_sizes.flatten(), bins=20)\n",
    "    ax[2].set_xlabel('Number of IDs')\n",
    "    ax[2].set_ylabel('New Cluster Counts')\n",
    "    ax[2].set_yscale('log')\n",
    "    ax[3].hist(cluster_cardIDs_sizes.flatten(), bins=20)\n",
    "    ax[3].set_xlabel('Number of cards')\n",
    "    ax[3].set_ylabel('New Cluster Counts')\n",
    "    ax[3].set_yscale('log')\n",
    "    plt.subplots_adjust(wspace=0.7)\n",
    "    plt.show()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf697ec3-b9d6-4a8d-8d48-14b57292f3df",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def get_cluster_hierarchy(knowledgeGraph, \n",
    "                          max_num_hierarchy_levels=20,\n",
    "                          meta_cluster_size_scaling=1.0,  # Doesn't seem to matter too much? 0.1 to 2.0 mostly works\n",
    "                              # But smaller numbers give more well defined categories, that get combined more slowly\n",
    "                          target_rms_combination_size=2.75,  # Basically how large should cluster of objects be at each level? \n",
    "                          target_number_of_combination_depth=5.5, # How deeply to cluster everthing\n",
    "                          verbose=False):\n",
    "    \"\"\"\n",
    "    Function for extracting clusters hierarchically from small to large, based on card and cluster similarity\n",
    "    \"\"\"\n",
    "    \n",
    "    # want meta_cluster_size_scaling to be large enough so that we converge quickly\n",
    "    # but also small enough that some cards don't get put in clusters if they are really outliers. \n",
    "\n",
    "    target_number_large_clusters = len(knowledgeGraph.cards.keys()) / (target_rms_combination_size ** target_number_of_combination_depth)\n",
    "    if verbose:\n",
    "        print('Targeting this many large clusters:', target_number_large_clusters)\n",
    "    large_cluster_number_threshold = target_number_large_clusters\n",
    "    \n",
    "\n",
    "    # Get level 0 card clusters from card ID lists, as if they were the result of some previous computation.  \n",
    "    # List of similarity metric IDs in resulting groups\n",
    "    prev_significant_clusters_IDgroups_list = [[_id] for _id in list(range(len(knowledgeGraph.cards.keys())))]\n",
    "    # List of cards in the resulting groups \n",
    "    prev_significant_clusters_cardIDs_list = prev_significant_clusters_IDgroups_list.copy()\n",
    "    # Embedding vector for each group \n",
    "    prev_emb_vec_list = [knowledgeGraph.cards[cardID].embedding_vector_trimmed for cardID in list(range(len(knowledgeGraph.cards.keys())))]\n",
    "    \n",
    "    # storage arrays \n",
    "    hierarchy_of_significant_clusters_IDgroups_list = []\n",
    "    hierarchy_of_significant_clusters_cardIDs_list = []\n",
    "    hierarchy_of_emb_vec_lists = []\n",
    "    hierarchy_of_similarity_metrics = []\n",
    "    \n",
    "    for hierarchy_level in range(max_num_hierarchy_levels):  \n",
    "        print('Calculating level ', hierarchy_level, ' with target size', meta_cluster_size_scaling) \n",
    "        \n",
    "        # Adjust cluster size target based on how much we combined things last time\n",
    "        prev_cluster_sizes = np.array([len(clu) for clu in prev_significant_clusters_IDgroups_list])\n",
    "        combination_sizes = prev_cluster_sizes[np.where(prev_cluster_sizes > 1)]\n",
    "        if len(combination_sizes) > 0:\n",
    "            rms_combination_size = np.sqrt(np.average(combination_sizes **2))\n",
    "            meta_cluster_size_scaling *= target_rms_combination_size/rms_combination_size  # adjust size to target some combination rate \n",
    "        else:\n",
    "            rms_combination_size = 1.0\n",
    "        \n",
    "        if verbose:\n",
    "            print('  RMS combnation size:', rms_combination_size)\n",
    "        prev_emb_vec_list = get_emb_vec_list_from_cluster_cardIDs(prev_significant_clusters_cardIDs_list, knowledgeGraph)\n",
    "        \n",
    "        if verbose:\n",
    "            print('  Calculating similarity metric...')\n",
    "        prev_similarity_metric = get_similarity_metric_from_emb_vec_list(prev_emb_vec_list)    \n",
    "            \n",
    "        if verbose:\n",
    "            print('  Calculating clustering...')\n",
    "        new_significant_clusters_IDgroups_list = get_best_clusters_with_desired_size(prev_similarity_metric,\n",
    "                                                 meta_cluster_size_scaling=meta_cluster_size_scaling)\n",
    "        new_significant_clusters_cardIDs_list = get_new_cluster_cardIDs(new_significant_clusters_IDgroups_list, prev_significant_clusters_cardIDs_list)\n",
    "        prev_num_clusters = len(prev_significant_clusters_IDgroups_list)\n",
    "        new_num_clusters = len(new_significant_clusters_IDgroups_list)\n",
    "        \n",
    "        # Check overall hierarchy structure relative to cardIDs, to see if we should stop clustering \n",
    "        new_cluster_sizes = np.array([len(significant_cluster) for significant_cluster \n",
    "                                  in new_significant_clusters_cardIDs_list])\n",
    "        rms_new_cluster_size = np.sqrt(np.mean(new_cluster_sizes ** 2))\n",
    "        num_new_relatively_large_clusters = len(np.where(new_cluster_sizes >= rms_new_cluster_size)[0])  \n",
    "            # This is at minimum 1 if we have a huge cluster and many small ones, \n",
    "            # but if things are not well separated, there will be many.\n",
    "            # so when this reaches 1 is a good place to stop \n",
    "            \n",
    "        # save previous level\n",
    "        hierarchy_of_significant_clusters_IDgroups_list.append(prev_significant_clusters_IDgroups_list)\n",
    "        hierarchy_of_significant_clusters_cardIDs_list.append(prev_significant_clusters_cardIDs_list)\n",
    "        hierarchy_of_emb_vec_lists.append(prev_emb_vec_list)\n",
    "        hierarchy_of_similarity_metrics.append(prev_similarity_metric)\n",
    "        \n",
    "        # now update parameters to use for next round \n",
    "        prev_significant_clusters_IDgroups_list = new_significant_clusters_IDgroups_list\n",
    "        prev_significant_clusters_cardIDs_list = new_significant_clusters_cardIDs_list \n",
    "            \n",
    "        if verbose:\n",
    "            print('  Number of relatively large clusters:', num_new_relatively_large_clusters)\n",
    "            display_clustering_metrics(prev_similarity_metric,\n",
    "                                       prev_significant_clusters_IDgroups_list,\n",
    "                                       prev_significant_clusters_cardIDs_list,\n",
    "                                       hierarchy_level)\n",
    "        \n",
    "        if ((new_num_clusters == 1) or  # We have 1 cluster left total \n",
    "            (prev_num_clusters == new_num_clusters) or  # we didn't reduce the number of clusters last round\n",
    "            (num_new_relatively_large_clusters <= large_cluster_number_threshold)):  # we have a pretty skewed distribution of clusters, one big one \n",
    "            if verbose:\n",
    "                print('Completed with', new_num_clusters, ' final clusters.')\n",
    "            break # we are done \n",
    "        \n",
    "    cluster_hierarchy_meta_data = [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "            hierarchy_of_significant_clusters_cardIDs_list,\n",
    "            hierarchy_of_emb_vec_lists,\n",
    "            hierarchy_of_similarity_metrics]\n",
    "    \n",
    "    return cluster_hierarchy_meta_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a735378-698a-4fa7-a685-e12114b4c401",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some functions to report useful information output by clustering\n",
    "\n",
    "def extra_3space(num):\n",
    "    return \"\".join([' ']*(3*num))\n",
    "    \n",
    "def get_text_emb_vec_overlap_with_knowledgeGraph_short(emb_vec, knowledgeGraph, max_print=3):\n",
    "\n",
    "    # Overlap over nodes\n",
    "    overlap_dict_nodes = knowledgeGraph.get_dict_of_emb_vec_inner_product_over_nodes(emb_vec)\n",
    "    sorted_node_titles, sorted_node_overlaps = get_dict_items_sorted_by_decreasing_value(overlap_dict_nodes)\n",
    "    \n",
    "    # Overlap over cards\n",
    "    # overlap_dict_cards = knowledgeGraph.get_dict_of_emb_vec_inner_product_over_cards(emb_vec)\n",
    "    # sorted_cardIDs, sorted_card_overlaps = get_dict_items_sorted_by_decreasing_value(overlap_dict_cards)\n",
    "    \n",
    "    return \", \".join([node_title for node_title in sorted_node_titles[0:max_print]])\n",
    "\n",
    "def get_text_emb_vec_short(emb_vec, knowledgeGraph, max_print=3):\n",
    "\n",
    "    # Direct embedding vector \n",
    "    sorted_node_titles, sorted_node_overlaps = get_dict_items_sorted_by_decreasing_value(emb_vec)\n",
    "    \n",
    "    return \", \".join([node_title for node_title in sorted_node_titles[0:max_print]])\n",
    "\n",
    "def display_cluster_hierarchy_process_of_calculation(cluster_hierarchy_meta_data):\n",
    "    \"\"\"\n",
    "    Displays calculations during creating clusters and topics.\n",
    "        Things like the similarity metric, overlap histogram, and cluster size histograms \n",
    "    \"\"\"\n",
    "    \n",
    "    [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "     hierarchy_of_significant_clusters_cardIDs_list,\n",
    "     hierarchy_of_emb_vec_lists,\n",
    "     hierarchy_of_similarity_metrics]  = cluster_hierarchy_meta_data\n",
    "\n",
    "    # Display all the cluster hierarchies    \n",
    "    for hierarchy_level in range(len(hierarchy_of_similarity_metrics)):\n",
    "        display_clustering_metrics(hierarchy_of_similarity_metrics[hierarchy_level], \n",
    "                                   hierarchy_of_significant_clusters_IDgroups_list[hierarchy_level], \n",
    "                                   hierarchy_of_significant_clusters_cardIDs_list[hierarchy_level],\n",
    "                                  hierarchy_level)\n",
    "        \n",
    "def display_cluster_hierarchy_major_topics(cluster_hierarchy_meta_data, max_levels_to_display=5, max_print=5):\n",
    "    \"\"\"\n",
    "    Displays text summarizing the clusters and topics in the clusters, from large to small\n",
    "        Only displays clusters with large size compared to others (determined by RMS size)\n",
    "    \"\"\"\n",
    "    fraction_of_rms_considered_relatively_large = 0.5\n",
    "    \n",
    "    def get_relatively_large_cluster_inds(hierarchy_of_significant_clusters_cardIDs_list, hierarchy_level):\n",
    "        clusters_cardIDs_list = hierarchy_of_significant_clusters_cardIDs_list[hierarchy_level]\n",
    "        cluster_cardIDs_sizes = np.array([len(_clu) for _clu in clusters_cardIDs_list])\n",
    "        rms_cluster_size = np.sqrt(np.mean(cluster_cardIDs_sizes ** 2))\n",
    "        relatively_large_cluster_inds = np.where(cluster_cardIDs_sizes >= fraction_of_rms_considered_relatively_large*rms_cluster_size)[0]\n",
    "        return relatively_large_cluster_inds\n",
    "    \n",
    "    [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "     hierarchy_of_significant_clusters_cardIDs_list,\n",
    "     hierarchy_of_emb_vec_lists,\n",
    "     hierarchy_of_similarity_metrics]  = cluster_hierarchy_meta_data\n",
    "    \n",
    "    num_hierarchy_levels = len(hierarchy_of_significant_clusters_IDgroups_list)\n",
    "\n",
    "    for reverse_level in range(max_levels_to_display): # start from largest and go backwards \n",
    "        hierarchy_level = num_hierarchy_levels - reverse_level - 1\n",
    "        relatively_large_cluster_inds = get_relatively_large_cluster_inds(hierarchy_of_significant_clusters_cardIDs_list, \n",
    "                                                                          hierarchy_level)\n",
    "        print('Level:', hierarchy_level)\n",
    "        for _clusterInd in relatively_large_cluster_inds:\n",
    "            cluster_emb_vec = hierarchy_of_emb_vec_lists[hierarchy_level][_clusterInd]\n",
    "            cluster_cardIDs_list = hierarchy_of_significant_clusters_cardIDs_list[hierarchy_level][_clusterInd]\n",
    "            print_text_overlap = get_text_emb_vec_overlap_with_knowledgeGraph_short(cluster_emb_vec, kGraph, max_print=max_print)\n",
    "            print_text_vec = get_text_emb_vec_short(cluster_emb_vec, kGraph, max_print=max_print)\n",
    "            print('   Cluster ID:', _clusterInd, \" Size:\", len(cluster_cardIDs_list), ' Overlap Topics:', print_text_overlap,' -----', print_text_vec)\n",
    "        print('')\n",
    "        \n",
    "def get_family_of_cluster(cluster_hierarchy_meta_data, target_hierarchy_level, target_cluster_ID):\n",
    "    \"\"\"\n",
    "    Displays text summarizing the clusters and topics in the clusters, from large to small\n",
    "        Only displays clusters with large size compared to others (determined by RMS size)\n",
    "    \"\"\"\n",
    "    \n",
    "    [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "     hierarchy_of_significant_clusters_cardIDs_list,\n",
    "     hierarchy_of_emb_vec_lists,\n",
    "     hierarchy_of_similarity_metrics]  = cluster_hierarchy_meta_data\n",
    "    \n",
    "    num_hierarchy_levels = len(hierarchy_of_significant_clusters_IDgroups_list)\n",
    "    \n",
    "    # Get children clusters\n",
    "    child_clusters_IDs = list(hierarchy_of_significant_clusters_IDgroups_list[target_hierarchy_level][target_cluster_ID])\n",
    "    \n",
    "    # Get parent clusters\n",
    "    if target_hierarchy_level + 1 <= num_hierarchy_levels - 1:\n",
    "        # Parents\n",
    "        possible_parent_cluster_IDgroups_list = hierarchy_of_significant_clusters_IDgroups_list[target_hierarchy_level + 1]\n",
    "        parent_clusters_IDs = [_IDgroup_ind for _IDgroup_ind, _IDgroup in enumerate(possible_parent_cluster_IDgroups_list) \n",
    "                              if target_cluster_ID in _IDgroup]\n",
    "        # Siblinbs\n",
    "        list_of_sibling_sets = [set(_IDgroup) for _IDgroup_ind, _IDgroup in enumerate(possible_parent_cluster_IDgroups_list) \n",
    "                              if target_cluster_ID in _IDgroup]\n",
    "        sibling_clusters_IDs = set()\n",
    "        for sibling_set in list_of_sibling_sets:\n",
    "            sibling_clusters_IDs.update(sibling_set)\n",
    "        sibling_clusters_IDs.remove(target_cluster_ID)\n",
    "        sibling_clusters_IDs = list(sibling_clusters_IDs)\n",
    "            \n",
    "    else:\n",
    "        parent_clusters_IDs = []\n",
    "        sibling_clusters_IDs = []\n",
    "    \n",
    "    return child_clusters_IDs, parent_clusters_IDs, sibling_clusters_IDs\n",
    "    \n",
    "\n",
    "def display_cluster_hierarchy_of_cluster(cluster_hierarchy_meta_data, target_hierarchy_level, target_cluster_ID, max_print=4):\n",
    "    \n",
    "    def display_clusters(cluster_inds, hierarchy_level, extra_spaces):\n",
    "        for _clusterInd in cluster_inds:\n",
    "            cluster_emb_vec = hierarchy_of_emb_vec_lists[hierarchy_level][_clusterInd]\n",
    "            cluster_cardIDs_list = hierarchy_of_significant_clusters_cardIDs_list[hierarchy_level][_clusterInd]\n",
    "            print_text_overlap = get_text_emb_vec_overlap_with_knowledgeGraph_short(cluster_emb_vec, kGraph, max_print=max_print)\n",
    "            print_text_vec = get_text_emb_vec_short(cluster_emb_vec, kGraph, max_print=max_print)\n",
    "            print(extra_3space(extra_spaces), 'Level:', hierarchy_level, ' ID:', _clusterInd, \n",
    "                  \" Size:\", len(cluster_cardIDs_list), ' Topics:', print_text_overlap,' -----', print_text_vec)\n",
    "        \n",
    "    [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "     hierarchy_of_significant_clusters_cardIDs_list,\n",
    "     hierarchy_of_emb_vec_lists,\n",
    "     hierarchy_of_similarity_metrics]  = cluster_hierarchy_meta_data\n",
    "    \n",
    "    num_hierarchy_levels = len(hierarchy_of_significant_clusters_IDgroups_list)\n",
    "\n",
    "    children, parents, siblings = get_family_of_cluster(cluster_hierarchy_meta_data, target_hierarchy_level, target_cluster_ID)\n",
    "    children_hierarchy_level = target_hierarchy_level - 1\n",
    "    parent_hierarchy_level = target_hierarchy_level + 1\n",
    "    \n",
    "    # Display things, if they exist \n",
    "    if parent_hierarchy_level < num_hierarchy_levels - 1:\n",
    "        \n",
    "        # Get grandparents and display \n",
    "        grandparents_set = set()\n",
    "        for parent in parents:\n",
    "            _, grandparents, _ = get_family_of_cluster(cluster_hierarchy_meta_data, parent_hierarchy_level, parent)\n",
    "            grandparents_set.update(set(grandparents))\n",
    "        grandparents = list(grandparents_set)\n",
    "        print('Grandparents')\n",
    "        for grandparent in grandparents:\n",
    "            display_clusters([grandparent], parent_hierarchy_level + 1, 0)\n",
    "        \n",
    "        # Display parents \n",
    "        print('Parents')\n",
    "        display_clusters(parents, parent_hierarchy_level, 0)\n",
    "    \n",
    "    # Display siblings and self \n",
    "    print(extra_3space(1), 'Siblings')\n",
    "    display_clusters(siblings, target_hierarchy_level,1)\n",
    "    print(extra_3space(1), 'Self')\n",
    "    display_clusters([target_cluster_ID], target_hierarchy_level,1)\n",
    "    \n",
    "    # Get children and grandchildren \n",
    "    if children_hierarchy_level >= 0:\n",
    "        print(extra_3space(2), 'Children')\n",
    "        for child in children:\n",
    "            display_clusters([child], children_hierarchy_level, 2)\n",
    "            grandchildren_hierarchy_level = children_hierarchy_level - 1\n",
    "            if grandchildren_hierarchy_level >= 0:\n",
    "                print(extra_3space(3), 'Grandchildren')\n",
    "                grandchildren, *_ = get_family_of_cluster(cluster_hierarchy_meta_data, children_hierarchy_level, child)\n",
    "                display_clusters(grandchildren, grandchildren_hierarchy_level, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a135056-489c-4827-8108-d3dd3e4ef254",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_hierarchy_meta_data = get_cluster_hierarchy(kGraph, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50c5eee5-d47e-468b-8548-f23b514c25be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display context of whole graph clustering \n",
    "\n",
    "display_cluster_hierarchy_major_topics(cluster_hierarchy_meta_data, max_levels_to_display=5)\n",
    "display_cluster_hierarchy_process_of_calculation(cluster_hierarchy_meta_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a4b9518-aae5-4355-a76e-f491c7c89c84",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display context of a single cluster or card \n",
    "\n",
    "target_hierarchy_level = 1\n",
    "target_cluster_ID = 3\n",
    "display_cluster_hierarchy_of_cluster(cluster_hierarchy_meta_data, target_hierarchy_level, target_cluster_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0ee60fb-59dd-4571-a86a-e58a7a64406e",
   "metadata": {},
   "source": [
    "### Explore cluster overlaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "254301a7-2c63-4eca-b455-c1ba48d740ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Get a list of embedding vectors from all clusters, with IDs\n",
    "\n",
    "def get_all_unique_cluster_emb_vecs(cluster_hierarchy_meta_data):\n",
    "    \n",
    "    # This also removes non-unique clusters (ones that appear unchanged at multiple hierarchy levels, \n",
    "    # and it replaces them with only the lowest hierarchy where they appear. \n",
    "    \n",
    "    [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "     hierarchy_of_significant_clusters_cardIDs_list,\n",
    "     hierarchy_of_emb_vec_lists,\n",
    "     hierarchy_of_similarity_metrics]  = cluster_hierarchy_meta_data\n",
    "    \n",
    "    significant_clusters_unique_tuple_id_list = [] # for checking if this exact cluster is already found        \n",
    "    all_unique_cluster_emb_vecs = {}\n",
    "    for hierarchy_level in range(len(hierarchy_of_emb_vec_lists)):  # goes from low to high \n",
    "        \n",
    "        unique_tuple_cardID_list_this_level = {(hierarchy_level, cluster_ID): tuple(sorted(cardIDs_list)) for cluster_ID, cardIDs_list \n",
    "                                       in enumerate(hierarchy_of_significant_clusters_cardIDs_list[hierarchy_level])}\n",
    "        \n",
    "        cluster_emb_vecs_this_level = {(hierarchy_level, cluster_ID): emb_vec for cluster_ID, emb_vec \n",
    "                                       in enumerate(hierarchy_of_emb_vec_lists[hierarchy_level]) \n",
    "                                      if not (unique_tuple_cardID_list_this_level[(hierarchy_level, cluster_ID)] \n",
    "                                              in significant_clusters_unique_tuple_id_list)}\n",
    "        all_unique_cluster_emb_vecs.update(cluster_emb_vecs_this_level)\n",
    "        \n",
    "        # Update list of unique ideas not to use in future. \n",
    "        significant_clusters_unique_tuple_id_list.extend(list(unique_tuple_cardID_list_this_level.values()))\n",
    "    \n",
    "    return all_unique_cluster_emb_vecs\n",
    "\n",
    "def get_all_card_counts_over_unique_clusters(cluster_hierarchy_meta_data, all_unique_cluster_emb_vecs, knowledgeGraph):\n",
    "    \n",
    "    # This also removes non-unique clusters (ones that appear unchanged at multiple hierarchy levels, \n",
    "    # and it replaces them with only the lowest hierarchy where they appear. \n",
    "    \n",
    "    [hierarchy_of_significant_clusters_IDgroups_list, \n",
    "     hierarchy_of_significant_clusters_cardIDs_list,\n",
    "     hierarchy_of_emb_vec_lists,\n",
    "     hierarchy_of_similarity_metrics]  = cluster_hierarchy_meta_data\n",
    "    \n",
    "    all_card_counts = [{k: 0.0 for k in list(knowledgeGraph.cards.keys())} for _ in range(len(hierarchy_of_similarity_metrics))]\n",
    "    \n",
    "    for hierarchy_level, cluster_ID in all_unique_cluster_emb_vecs.keys():\n",
    "        cluster_cardIDs_list = hierarchy_of_significant_clusters_cardIDs_list[hierarchy_level][cluster_ID]\n",
    "        for _cardID in cluster_cardIDs_list:\n",
    "            all_card_counts[hierarchy_level][_cardID] += 1.0\n",
    "        \n",
    "    return all_card_counts\n",
    "\n",
    "all_unique_cluster_emb_vecs = get_all_unique_cluster_emb_vecs(cluster_hierarchy_meta_data)\n",
    "all_card_counts = get_all_card_counts_over_unique_clusters(cluster_hierarchy_meta_data, all_unique_cluster_emb_vecs, kGraph)\n",
    "\n",
    "display_dict_sorted_by_decreasing_value(all_unique_cluster_emb_vecs[(0,76)])\n",
    "display_dict_sorted_by_decreasing_value(all_card_counts[6])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "910589fb-85b4-4869-9b25-f18b87b9fe0c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "dfb10ebb9e28404e"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "9c27d54fd6303821"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "4d61fddcdcdf9165"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "f7a09870e8e4f672"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "6fd8a5db8e513237"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "80da3c44f488bf1f"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "f6405ccc1b0b1f1"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "53fd91f5037823c0"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "fe278178d308e341"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "d99bc160968959f2"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "c3b8de8688363bfc"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "2d594b543b9ab179"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "696824afef9b2425"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "6daaa260007406e2"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "32fba1468492a1c0"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "8a7318fef06ce759"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}