DP02_06b_Interactive_Catalog_Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactive Catalog Visualization\n",
    "\n",
    "<img align=\"left\" src = https://project.lsst.org/sites/default/files/Rubin-O-Logo_0.png width=250 style=\"padding: 10px\"> \n",
    "<br>\n",
    "Contact author(s): Leanne Guy <br>\n",
    "Last verified to run: 2024-09-16 <br>\n",
    "LSST Science Pipelines version: Weekly 2024_37 <br>\n",
    "Container Size: large <br>\n",
    "Targeted learning level: intermediate <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Description:** Interactive catalog data visualizations with three open-source python libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Skills:** Create linked interactive plots for large datasets, and output interactive plots to interactive HTML files. Use Bokeh, HoloViews, and Datashader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSST Data Products:** Object catalog."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Packages:** bokeh, holoviews, datashader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Credit:**\n",
    "This tutorial was inspired by a notebook originally developed by Keith Bechtol in the context of the LSST Stack Club. \n",
    "It has been updated and extended for DP0.2 by Leanne Guy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Get Support:**\n",
    "Find DP0-related documentation and resources at <a href=\"https://dp0.lsst.io\">dp0.lsst.io</a>. \n",
    "Questions are welcome as new topics in the <a href=\"https://community.lsst.org/c/support/dp0\">Support - Data Preview 0 Category</a> of the \n",
    "Rubin Community Forum. Rubin staff will respond to all questions posted there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "The Rubin Science Platform was designed to enable scientific analysis of the LSST data sets, which will be unprecedentedly large and complex. \n",
    "The software and techniques that are best suited for visualizing large data sets might be new to many astronomers.\n",
    "This notebook introduces learners with some knowledge of python to three open-source Python libraries that enable powerful interactive visualization of catalogs.\n",
    " 1. [**HoloViews**](http://holoviews.org): Produce high-quality interactive visualizations easily by annotating plots and images rather than using direct calls to a plotting library.\n",
    " 2. [**Bokeh**](https://bokeh.org): A powerful data visualization library that provides interactive tools including brushing and linking between multiple plots.\n",
    " 3. [**Datashader**](https://datashader.org): Accurately render very large datasets quickly and flexibly.\n",
    "\n",
    "These packages are part of the [Holoviz](http://holoviz.org/) ecosystem of tools intended for visualization in a web browser and can be used to create quite sophisticated dashboard-like interactive displays and widgets. The goal of this tutorial is to provide an introduction and starting point from which to create more advanced, custom interactive visualizations. Holoviz has a [vibrant and active community](https://discourse.holoviz.org/) where you can ask questions and discuss visualizations with a global community. \n",
    "\n",
    "> **Notice:** If the notebook or any interactive features seem to stall, first try going a few cells back and rerunning them in order (the order in which cells are run is imporant for this notebook's functionality). If that does not work, try restarting the kernel. If issues persist, try logging out and restarting the Notebook aspect using a \"large\" instance of the JupyterLab environment.\n",
    "\n",
    "> **Warning: It is not recommended to \"Restart Kernel and Run All Cells\" in this notebook, or to execute multiple cells very quickly.** Some of the examples require interaction (e.g., for the user to select points on a graph) in order to run correctly, and going too fast can cause some plots to not display properly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Import packages\n",
    "\n",
    "Import general scientific python packages (`os`, `numpy`, `pandas`),\n",
    "functions from the astronomy python package `astropy`,\n",
    "the Rubin function for accessing the TAP service (`lsst.rsp.get_tap_service`),\n",
    "and various functions from the `bokeh`, `holoviews`, and `datashader` packages\n",
    "that are used in this tutorial.\n",
    "\n",
    "> **Notice:** It is OK to ignore the warning that appears in pink below the following cell about upgrading `pyarrow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from astropy import units as u\n",
    "from astropy.coordinates import SkyCoord\n",
    "from astropy.units.quantity import Quantity\n",
    "\n",
    "from lsst.rsp import get_tap_service\n",
    "\n",
    "import bokeh\n",
    "from bokeh.io import output_notebook, show, output_file, reset_output\n",
    "from bokeh.models import ColumnDataSource, Range1d, HoverTool\n",
    "from bokeh.models import CDSView, GroupFilter\n",
    "from bokeh.plotting import figure, gridplot\n",
    "from bokeh.transform import factor_cmap\n",
    "\n",
    "import holoviews as hv\n",
    "from holoviews import streams, opts\n",
    "from holoviews.operation.datashader import datashade, dynspread\n",
    "from holoviews.plotting.util import process_cmap\n",
    "\n",
    "import datashader as dsh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show which versions of Bokeh, HoloViews, and Datashader we are working with.\n",
    "This is important when referring to online documentation as APIs can change between versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Bokeh version: \" + bokeh.__version__)\n",
    "print(\"HoloViews version: \" + hv.__version__)\n",
    "print(\"Datashader version: \" + dsh.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Define functions and parameters\n",
    "\n",
    "Update the maximum number of display rows for Pandas tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.max_rows', 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the display of the output Bokeh plots to be inline, in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the HoloViews plotting library to be bokeh. \n",
    "The HoloViews and Bokeh icons are displayed when the library is loaded successfully.\n",
    "**Always set the extension after executing `output_notebook()` to avoid issues with plot display.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hv.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Notice:** Sometimes the `bokeh.io.show` function can be finicky when output modes are switched (e.g., from inline to an HTML file and back again).\n",
    "\n",
    "To avert a \"Models must be owned by only a single document\" error (see, e.g., <a href=\"https://github.com/bokeh/bokeh/issues/8579\">https://github.com/bokeh/bokeh/issues/8579</a>), define the following two functions and use them in Section 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def show_bokeh_inline(p):\n",
    "    try:\n",
    "        reset_output()\n",
    "        output_notebook()\n",
    "        show(p)\n",
    "    except:\n",
    "        output_notebook()\n",
    "        show(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def show_bokeh_to_file(p, outputFile):\n",
    "    try:\n",
    "        reset_output()\n",
    "        output_file(outputFile)\n",
    "        show(p)\n",
    "    except:\n",
    "        output_file(outputFile)\n",
    "        show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Use the TAP service to obtain table data\n",
    "\n",
    "The basis for any data visualization is the underlying data.\n",
    "This tutorial works with tabular data that is retrieved from \n",
    "a cone search around a defined coordinate with a specified radius\n",
    "using the Rubin TAP service.\n",
    "\n",
    "Get a Rubin TAP service instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "service = get_tap_service(\"tap\")\n",
    "assert service is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a reference position on the sky and a radius in degrees for a cone search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "coord = SkyCoord(ra=55.65*u.degree, dec=-40.*u.degree, frame='icrs')\n",
    "radius = 1 * u.deg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the query pass to the TAP service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query = \"SELECT coord_ra, coord_dec, objectId, r_extendedness, \"\\\n",
    "        \"scisql_nanojanskyToAbMag(g_cModelFlux) AS mag_g_cModel, \"\\\n",
    "        \"scisql_nanojanskyToAbMag(r_cModelFlux) AS mag_r_cModel, \"\\\n",
    "        \"scisql_nanojanskyToAbMag(i_cModelFlux) AS mag_i_cModel \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec),\"\\\n",
    "        \"CIRCLE('ICRS', \" + str(coord.ra.value) + \", \" + str(coord.dec.value) + \", \" \\\n",
    "        + str(radius.to(u.deg).value) + \" )) = 1 \" \\\n",
    "        \"AND detect_isPrimary = 1 \"\\\n",
    "        \"AND scisql_nanojanskyToAbMag(r_cModelFlux) < 27.0 \"\\\n",
    "        \"AND r_extendedness IS NOT NULL\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As query will return a very large dataset, use an asynchronous query.\n",
    "This will take a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job = service.submit_job(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the job phase has completed, fetch the results into a Pandas table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = job.fetch_result().to_table().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Confirm that the expected number of rows has been returned (1345701).\n",
    "If true, the following cell will not return an assertion error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert len(data) == 1345701"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute three colors from the apparent magnitudes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data['gmi'] = data['mag_g_cModel'] - data['mag_i_cModel']\n",
    "data['rmi'] = data['mag_r_cModel'] - data['mag_i_cModel']\n",
    "data['gmr'] = data['mag_g_cModel'] - data['mag_r_cModel']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the r-band extendedness parameter to classify objects as having a \"shape_type\" of \"point\" or \"extended\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data['shape_type'] = data['r_extendedness'].map({0: 'point', 1: 'extended'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the `objectId` to a string because as an integer, it's too large for bokeh to handle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data['objectId'] = np.array(data['objectId']).astype('str')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Option to print the number of objects with `shape_type` as \"point\" or \"extended\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tx1 = np.where(data['shape_type'] == 'point')[0]\n",
    "# tx2 = np.where(data['shape_type'] == 'extended')[0]\n",
    "# print(len(tx1), len(tx2))\n",
    "# del tx1, tx2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-15T22:22:52.077102Z",
     "iopub.status.busy": "2024-01-15T22:22:52.076115Z",
     "iopub.status.idle": "2024-01-15T22:22:52.086329Z",
     "shell.execute_reply": "2024-01-15T22:22:52.084054Z",
     "shell.execute_reply.started": "2024-01-15T22:22:52.077047Z"
    }
   },
   "source": [
    "Confirm that the expected number of each `shape_type` have been returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert data[data[\"shape_type\"] == \"point\"].shape[0] == 232835\n",
    "assert data[data[\"shape_type\"] == \"extended\"].shape[0] == 1112866"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. HoloViews\n",
    "\n",
    "[HoloViews](https://holoviews.org) supports easy analysis and visualization by \n",
    "annotating data rather than utilizing direct calls to plotting packages. \n",
    "This tutorial uses [Bokeh](hrrps://bokeh.org) as the plotting \n",
    "library backend for HoloViews. \n",
    "HoloViews supports several plotting libraries, and exercise 1 in Section 6\n",
    "is to explore using HoloViews with other plotting packages. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a 2% random subsample of this dataset with which to demonstrate some basic HoloViews functionality.\n",
    "Print the length of this subset and confirm that it is 2% of the original size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "frac = 0.02\n",
    "data20K = data.sample(frac=frac, axis='index')\n",
    "print(len(data20K))\n",
    "assert len(data20K) == round(frac * len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Single plots\n",
    "\n",
    "The basic core primitives of HoloViews are [Elements](http://holoviews.org/Reference_Manual/holoviews.element.html) (`hv.Element`). \n",
    "Elements are simple wrappers for data which provide a semantically \n",
    "meaningful visual representation. \n",
    "An Element may be a set of Points, an Image, a Curve, a Histogram, etc. \n",
    "See the HoloViews [Reference Gallery](http://holoviews.org/reference/index.html) \n",
    "for all the various types of Elements that can be created with HoloViews. \n",
    "\n",
    "The example in this section uses the HoloViews [Scatter Element](http://holoviews.org/reference/elements/bokeh/Scatter.html) \n",
    "to quickly visualize the catalog data as a scatter plot. \n",
    "HoloViews maintains a strict separation between content and presentation. \n",
    "This separation is achieved by maintaining sets of keyword values as \n",
    "`options` that specify how `Elements` are to appear.  \n",
    "This example applies the default options and removes the toolbar.\n",
    "\n",
    "Make a simple scatter plot of the data using the Scatter element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hv.Scatter(data20K).options(toolbar=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 1: A non-interactive plot of `coord_ra` vs. `coord_dec` appears as a blue circle composed of individual, but mostly blended, blue dots.\n",
    "\n",
    "The `data20K` set contains many columns. If no columns are specified explicitly, the first 2 columns are taken for x and y respectively by the [Scatter Element](https://holoviews.org/reference/elements/bokeh/Scatter.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now bin the data in RA using the robust \n",
    "[Freedman Diaconis Estimator](https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html#numpy.histogram_bin_edges),\n",
    "plot the resulting distribution using the HoloViews \n",
    "[Histogram Element](http://holoviews.org/reference/elements/bokeh/Histogram.html),\n",
    "and add in some basic plot options. \n",
    "Read more about about [customizing plots](https://holoviews.org/user_guide/Customizing_Plots.html) via `options`. Note that `options` can be shortened to `opts`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(ra_bin, count) = np.histogram(data20K['coord_ra'], bins='fd')\n",
    "ra_distribution = hv.Histogram((ra_bin, count)).opts(\n",
    "    title=\"RA distribution\", color='darkmagenta',\n",
    "    xlabel='RA', fontscale=1.2,\n",
    "    height=400, width=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ra_distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 2: A histogram (bar chart in purple) of the number of objects in a given RA bin in the subset of twenty thousand objects. This plot is interactive and displays a tool bar at right for the user to, e.g., zoom in on the plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Layouts of unlinked plots\n",
    "\n",
    "Create a layout of several plots. \n",
    "A `Layout` is a type of `Container` that can contain any HoloViews object. \n",
    "Other types of Containers that exist include `Overlay`, `Gridspace`, `Dynamicmap`, etc. \n",
    "See the HoloViews [Reference Gallery](http://holoviews.org/reference/index.html) \n",
    "for the full list of `Layouts` that can be created with HoloViews. \n",
    "See [Building Composite Objects](http://holoviews.org/user_guide/Building_Composite_Objects.html) \n",
    "for the full details about the ways `Containers` can be composed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slice the data and set some more options, and then construct a layout using the `+` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "skyplot = hv.Scatter(data20K[[\"coord_ra\", \"coord_dec\"]]).opts(\n",
    "    title=\"Skyplot\", toolbar='above', tools=['hover'], \n",
    "    height=350, width=350, alpha=0.2, size=2)\n",
    "\n",
    "skyplots = skyplot + ra_distribution.options(height=350, width=350)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "skyplots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 3: Two side-by-side plots, Fig 1 from above on the left, and Fig 2 from above on the right, with the interactive toolbar at upper right.\n",
    "\n",
    "Note that these two plots above are not linked, they are two independent plots laid out next to each other.\n",
    "\n",
    "Zoom in on the skyplot and notice that the data are not rebinned in the RA distribution plot.\n",
    "Linking plots is demonstrated below.\n",
    "\n",
    "The tools, however, do apply to both plots. Try modifying both plots and then use the \"reset\" tool (the circular arrow symbol). Notice that both plots are reset to their original states. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Layouts of linked plots\n",
    "\n",
    "Set up some default plot options to avoid duplicating long lists for every new plot.\n",
    "\n",
    "Different plotting packages typically provide different customization capabilities.\n",
    "Below, one set of options is defined for a Bokeh backend, and one for a matplotlib backend.\n",
    "\n",
    "Set Bokeh customizations as a python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plot_style_bkh = dict(alpha=0.5, color='darkmagenta',\n",
    "                      marker='triangle', size=3,\n",
    "                      xticks=5, yticks=5,\n",
    "                      height=400, width=400,\n",
    "                      toolbar='above')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set matplotlib customizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plot_style_mpl = dict(alpha=0.2, color='c', marker='s',\n",
    "                      fig_size=200, s=2,\n",
    "                      fontsize=14, xticks=8, yticks=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose to use the Bokeh plot style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_style = plot_style_bkh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of subsetting a dataset to choose which columns to plot, \n",
    "HoloViews allows the user to specify the dimensionality directly. \n",
    "\n",
    "`kdims` are the key dimensions or the independent variable(s) and \n",
    "`vdims` are the value dimensions or the dependenent variable(s). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hv.Scatter(data20K,\n",
    "           kdims=['gmi'], vdims=['mag_g_cModel']\n",
    "           ).opts(invert_yaxis=True,\n",
    "                  xlabel=\" (g-i)\", ylabel=\"g\",\n",
    "                  **plot_style)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 4: The g-band magnitude as a function of g-i color, with points as small purple triangles. The interactive toolbar is at upper right.\n",
    "\n",
    "The dimensions have be specified as strings above, but they are in fact rich objects. Dimension objects support a long descriptive label, which complements the short programmer-friendly name.\n",
    "\n",
    "Below, create a color-color diagram of the point-like sources in the dataset, and also display the distribution of samples along both value dimensions using the `hist()` method of the [Scatter Element](http://holoviews.org/reference/elements/bokeh/Scatter.html).\n",
    "\n",
    "Set the axes as rich objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "rmi = hv.Dimension('rmi', label='(r-i)', range=(-0.8, 3.0))\n",
    "gmr = hv.Dimension('gmr', label='(g-r)', range=(-0.8, 3.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identify the point-like objects as \"points\" and create the scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points = data20K[data20K[\"shape_type\"] == 'point']\n",
    "col_col = hv.Scatter(points, kdims=gmr,\n",
    "                     vdims=rmi).opts(**plot_style)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the hist method to show the distribution of samples along both value dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "col_col = col_col.hist(dimension=[gmr, rmi],\n",
    "                       num_bins=100, adjoin=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "col_col"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 5: The r-i color versus the g-r color (purple triangles), with histograms (blue bar charts) of the color distributions above and to the right. The interactive tool bar is at upper right.\n",
    "\n",
    "Try zooming in on regions of the plot. The histograms are automatically recomputed.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4. Magics\n",
    "\n",
    "The techniques to apply customizations in the cells above use standard Python syntax and are the recommended way to customize visualizations in HoloViews. HoloViews also supports IPython magic commands. Magics are a much older approach that is not standard Python and is specific to notebooks. [HoloViews notebook magic](https://holoviews.org/user_guide/Notebook_Magics.html) supports both line and cell magics. Here is an example of using magics to plot the same spatial distribution of Objects as above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%opts Scatter [tools=['hover'], toolbar='above',height=400, width=400](color='grey')\n",
    "hv.Scatter(data20K).opts(alpha=0.3, size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 6: Same as Fig 1, but plotted with grey points and with an interactive toolbar at upper right.\n",
    "\n",
    "Hover the mouse over the plot above and see that many (too many) columns are\n",
    "included for the hover tool to display. \n",
    "It is more user-friendly to be selective about which information is shown \n",
    "with the hover tool. \n",
    "\n",
    "Create a custom hover tool with select columns: the RA and Dec, the r-band \n",
    "magnitude, and the `shape_type`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "raDecHover = HoverTool(\n",
    "    tooltips=[\n",
    "        ('ra,dec', '@coord_ra / @coord_dec'),\n",
    "        ('rmag', '@mag_r_cModel'),\n",
    "        ('type', '@shape_type'),\n",
    "    ],\n",
    "    formatters={\n",
    "        'ra/dec': 'printf',\n",
    "        'rmag': 'numeral',\n",
    "        'type': 'printf',\n",
    "    },\n",
    "    point_policy=\"follow_mouse\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hv.Scatter(data20K).opts(tools=[raDecHover], **plot_style_bkh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 7: Same as Fig 6, but plotted with purple points.\n",
    "\n",
    "Hover the mouse over the plot above and see only the selected columns' data are displayed.\n",
    "\n",
    "It might be necessary to zoom in until points are distinguishable in order to notice how\n",
    "the hover tool box contents list only three properties per point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Bokeh\n",
    "\n",
    "A very useful feature of Bokeh is the ability to add connected interactivity between plots \n",
    "that show different attributes of the same data.\n",
    "**This is called linking**. \n",
    "\n",
    "With linked plots it is possible to carry out data **brushing**, whereby \n",
    "data can be selected and manipulated synchronously across multiple linked plots. \n",
    "\n",
    "For example, if a skyplot is linked with a colour-magnitude diagram of the same \n",
    "dataset, it becomes possible to interactively explore the relationship between \n",
    "the positions of objects in each plot.\n",
    "\n",
    "This section uses the Bokeh plotting library to demonstrate how to set up brushing \n",
    "and linking between two panels showing different representations of the same dataset. \n",
    "A selection applied to either panel will highlight the selected points in the other panel.\n",
    "\n",
    "This section is based on [Bokeh linked brushing](http://bokeh.pydata.org/en/latest/docs/user_guide/interaction/linking.html#linked-brushing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Data preparation\n",
    "Getting the data preparation phase right is key to creating powerful visualizations. \n",
    "Bokeh works with a ColumnDataSource (CDS).\n",
    "A CDS is essentially a collection of sequences of data that have their own unique column name. \n",
    "\n",
    "The CDS is the core of bokeh plots.\n",
    "Bokeh automatically creates a CDS from data passed as python lists or numpy arrays.\n",
    "CDS are useful as they allow data to be shared between multiple plots and renderers, \n",
    "enabling brushing and linking.\n",
    "\n",
    "Below, a CDS is created from the data returned by the query above and passed directly to Bokeh. \n",
    "\n",
    "Create a CDS for the plots to share.\n",
    "The data defined as `x0`, `y0`, `x1`, `y1` will be used to plot the left and right plots, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "col_data = dict(x0=data20K['coord_ra'] - coord.ra.value,\n",
    "                y0=data20K['coord_dec'] - coord.dec.value,\n",
    "                x1=data20K['gmi'],\n",
    "                y1=data20K['mag_g_cModel'],\n",
    "                ra=data20K['coord_ra'], dec=data20K['coord_dec'])\n",
    "source = ColumnDataSource(data=col_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional data can be added to the CDS after creation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "source.data['objectId'] = data20K['objectId']\n",
    "source.data['rmi'] = data20K['rmi']\n",
    "source.data['gmr'] = data20K['gmr']\n",
    "source.data['mag_r_cModel'] = data20K['mag_r_cModel']\n",
    "source.data['shape_type'] = data20K['shape_type']\n",
    "source.data['r_extendedness'] = data20K['r_extendedness']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a \"points\" view with a filter that requires \"shape_type\" to be \"point\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points = CDSView()\n",
    "points.filter = GroupFilter(column_name='shape_type', group=\"point\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Linked plots with data brushing\n",
    "\n",
    "Use Bokeh to plot a color-magnitude (g vs. g-i) diagram and a plot of the \n",
    "sky coordinates, and then link them.\n",
    "\n",
    "Create a custom hover tool for each panel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hover_left = HoverTool(tooltips=[(\"ObjectId\", \"@objectId\"),\n",
    "                                 (\"(ra,dec)\", \"(@ra, @dec)\"),\n",
    "                                 (\"type\", \"@shape_type\")\n",
    "                                 ])\n",
    "hover_right = HoverTool(tooltips=[(\"ObjectId\", \"@objectId\"),\n",
    "                                  (\"(g-i,g)\", \"(@x1, @y1)\"),\n",
    "                                  (\"extendedness\", \"@r_extendedness\")\n",
    "                                  ])\n",
    "tools = \"box_zoom,box_select,lasso_select,reset,help\"\n",
    "tools_left = [hover_left, tools]\n",
    "tools_right = [hover_right, tools]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new two-panel plot and add a renderer. \n",
    "Use the \"points\" view defined above.\n",
    "\n",
    "Create the left-side plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left = figure(tools=tools_left,\n",
    "              width=400, height=400,\n",
    "              title='Spatial: Centered on (RA, Dec) = (%.2f, %.2f)' %\n",
    "              (coord.ra.value, coord.dec.value))\n",
    "left.scatter('x0', 'y0', hover_color='firebrick',\n",
    "            size=3, alpha=0.7,\n",
    "            source=source,\n",
    "            view=points)\n",
    "\n",
    "left.x_range = Range1d(1.5, -1.5)\n",
    "left.y_range = Range1d(-1.5, 1.5)\n",
    "left.xaxis.axis_label = 'Delta ra'\n",
    "left.yaxis.axis_label = 'Delta dec'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the right-side plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "right = figure(tools=tools_right, width=400, height=400,\n",
    "               title='CMD')\n",
    "right.scatter('x1', 'y1', hover_color='firebrick',\n",
    "             size=4, alpha=0.8,\n",
    "             source=source,\n",
    "             view=points)\n",
    "\n",
    "right.x_range = Range1d(-1.5, 2.8)\n",
    "right.y_range = Range1d(32., 16.)\n",
    "right.xaxis.axis_label = '(g-i)'\n",
    "right.yaxis.axis_label = 'g'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the grid of plots. This can take a moment to render."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p = gridplot([[left, right]])\n",
    "show_bokeh_inline(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 8: At left, the offset from the specified coordinate ($\\Delta$ Dec vs. $\\Delta$ RA), and at right, the g-band magnitude as a function of g-i color. All points are blue circles and the interactive toolbar is at upper right.\n",
    "\n",
    "Use the hover tool to see information about individual data points (e.g., the \"ObjectId\").\n",
    "This information should appear automatically when the mouse is hovered over the data points.\n",
    "Notice the data points highlighted in red on one panel with the hover tool are also \n",
    "highlighted on the other panel.\n",
    "\n",
    "Next, click on the selection box icon or the selection lasso icon found \n",
    "in the upper right corner of the figure. \n",
    "Use the selection box and selection lasso to make various selections in either panel by \n",
    "clicking and dragging on either panel. \n",
    "The selected data points will be displayed in the other panel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3. Output to interactive HTML file\n",
    "\n",
    "Output this interactive plot to an interactive HTML file.\n",
    "\n",
    "Define the output file name to store it in the home directory `~/.`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "outputDir = os.path.expanduser('~')\n",
    "outputFileName = 'nb06b_plot1.html'\n",
    "outputFile = os.path.join(outputDir, outputFileName)\n",
    "\n",
    "print('The full pathname of the interactive HTML file will be '+outputFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `bokeh.io.show` method, embedded in the `show_bokeh_to_file` function\n",
    "defined in Section 1.2 above, to output the interactive HTML file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "show_bokeh_to_file(p, outputFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the interactive HTML file navigate to it via the directory listing in the \n",
    "left-hand side panel of this Jupyter notebook graphical interface, and click on its name.\n",
    "This will open another tab containing the HTML file.\n",
    "\n",
    "Since the file is relatively small (about 3.6 MB) it should load quickly (within a few seconds).\n",
    "Once loading is complete, click on the \"Trust HTML\" button at the top-left of the tab's window.\n",
    "Then, a near-duplicate of the two linked plots above should be displayed.\n",
    "\n",
    "It is also possible to download the HTML file, interact with it in a browser as a \n",
    "a local file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4. Linked streams\n",
    "\n",
    "To do subsequent calculations with the set of selected points, \n",
    "it is possible to use HoloViews linked streams for custom interactivity.\n",
    "The following visualization is a modification of this example. \n",
    "\n",
    "As for the example above, the plots generated below use the selection box \n",
    "and selection lasso to choose data points on the left panel, and then\n",
    "the selected points appear in the right panel. \n",
    "\n",
    "Notice that as the selection is changed in the left panel, the mean x- and\n",
    "y-values for selected data points are shown in the title of the right panel.\n",
    "\n",
    "This section is based on [HoloViews Selection1D points](http://holoviews.org/reference/streams/bokeh/Selection1D_points.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare some points, and define a selection for the stream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points = hv.Points((data20K['coord_ra'] - coord.ra.value,\n",
    "                    data20K['coord_dec'] - coord.dec.value)\n",
    "                   ).options(tools=['box_select', 'lasso_select'])\n",
    "\n",
    "selection = streams.Selection1D(source=points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function that uses the selection indices to slice points and compute statistics.\n",
    "This function is defined here (and not in Section 1.2) because it is only used in the\n",
    "subsequent (i.e., it is not a globally defined and used function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def selected_info(index) -> str:\n",
    "    selected = points.iloc[index]\n",
    "    if index:\n",
    "        label = 'Mean x, y: %.3f, %.3f' % tuple(selected.array().mean(axis=0))\n",
    "    else:\n",
    "        label = 'No selection'\n",
    "    return selected.relabel(label).options(color='red')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine \"points\" and DynamicMap. Notice the syntax used here, how the \"+\" sign makes side-by-side panels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points + hv.DynamicMap(selected_info, streams=[selection])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 9: At left, the offset from a specified coordinate (as in Fig 8). The plot at right has the same axes, but will be empty until the action below is taken to select points in the left plot. The interactive toolbar is at upper right.\n",
    "\n",
    "Use the lasso- or box-select to select a region in the left-hand plot.\n",
    "\n",
    "Print the number of objects selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(selection.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Option to list the indices of the selected objects, if the number selected is less than 200.\n",
    "\n",
    "If desired, these indices could be used to define a population for subsequence analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if len(selection.index) < 200:\n",
    "#     print(selection.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.0  Datashader\n",
    "\n",
    "The interactive features of Bokeh work well with datasets up to a few tens of thousands of data points.\n",
    "To efficiently explore larger datasets it is recommended to use another visualization model that offers better scalability: Datashader.\n",
    "\n",
    "The examples below will show how, when zooming in on the datashaded two-dimensional histograms,\n",
    "the bin sizes are dynamically adjusted to show finer or coarser granularity in the distribution.\n",
    "This allows for the interactive exploration of large datasets without having to manually adjust\n",
    "the bin sizes while panning and zooming. \n",
    "\n",
    "Zoom in all the way to see individual points (i.e., bins contain either zero or one count).\n",
    "Soom in far enough and find that the individual points are represented by extremely small\n",
    "pixels in datashader that are difficult to see.\n",
    "A solution is to [dynspread](https://datashader.org/api.html#datashader.transfer_functions.dynspread) \n",
    "instead of datashade, which will preserve a finite size of the plotted points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1. Plotting thousands of data points\n",
    "\n",
    "Plot a color-colour diagram using Bokeh with a customized hover tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plot_options = {'height': 400, 'width': 800,\n",
    "                'tools': ['pan', 'box_zoom', 'box_select',\n",
    "                          'wheel_zoom', 'reset', 'help']}\n",
    "\n",
    "p = figure(title=\"Color-Color Diagram\",\n",
    "           x_axis_label=\"(g-r)\", y_axis_label=\"(r-i)\",\n",
    "           x_range=(-2.0, 3.0), y_range=(-2.0, 3.0),\n",
    "           **plot_options)\n",
    "p.scatter(x='gmr', y='rmi', source=source,\n",
    "         size=2, alpha=0.2,\n",
    "         hover_color='firebrick',\n",
    "         legend_field=\"shape_type\",\n",
    "         color=factor_cmap('shape_type', 'Category10_3',\n",
    "                           ['point', 'extended']))\n",
    "\n",
    "hover = HoverTool(tooltips=[(\"objectId\", \"@objectId\"),\n",
    "                            (\"(ra,dec)\", \"(@coord_ra, @coord_dec)\"),\n",
    "                            (\"(g-r,r-i)\", \"(@gmr, @rmi)\"),\n",
    "                            (\"type\", \"@shape_type\")])\n",
    "p.add_tools(hover)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "show_bokeh_inline(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 10: The r-i color vs. the g-r color, with extended objects in orange and point-like objects in blue. The interactive tool bar is at right.\n",
    "\n",
    "The plot above suffers from overplotting (confusion), even though the dataset only contains ~20K points.\n",
    "A classic strategy to visualize the dense areas is to specify the transparency (`alpha`) of the glyphs;\n",
    "in the plot above, `alpha=0.2` was used.\n",
    "This has helped, but washes out the detail in the sparser regions. \n",
    "\n",
    "HoloViews + Datashader allow millions to billions of points to be plotted, and this \n",
    "produces much more informative plots.\n",
    "Datashader rasterizes or aggregates datasets into regular grids that can then be \n",
    "further analysed or viewed as plots or images.\n",
    "\n",
    "Create a Holoviews object \"points\" to hold and plot data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points = hv.Points((source.to_df()['gmr'],source.to_df()['rmi']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the linked streams instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "boundsxy = (0, 0, 0, 0)\n",
    "box = streams.BoundsXY(source=points, bounds=boundsxy)\n",
    "bounds = hv.DynamicMap(lambda bounds: hv.Bounds(bounds), streams=[box])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the datashader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p = dynspread(datashade(points, cmap=\"Viridis\"))\n",
    "p = p.opts(width=800, height=300, padding=0.05, show_grid=True,\n",
    "           xlim=(-2.0, 7.0), ylim=(-5.0, 3.0), xlabel=\"(g-r)\", ylabel=\"(r-i)\",\n",
    "           tools=['box_select'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Render the datashaded plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p * bounds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 11: The r-i vs. the g-r color as in Fig 10, but displayed as a 2-dimensional density map (with a purple-green-yellow colormap) in regions of high density, and as individual points in regions of low density. Interactive toolbar at right.\n",
    "\n",
    "This datashaded plot of the same color-color diagram as above does not require any magic-number parameters such as size and alpha and automatically ensures that there is no saturation or overplotting.\n",
    "\n",
    "Above, select the wheel zoom and adjust the image to interact with the plot. Note how the shades of color of the data points change according to the local density."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. Plotting millions of data points\n",
    "\n",
    "The datasest of ~20K points used above is actually too small to demonstrate the power of datashader. \n",
    "\n",
    "Below, visualize the full >1 million object dataset returned by the query.\n",
    "\n",
    "Create a Points Element for the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points = hv.Points((data['gmr'], data['rmi']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the linked streams instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "boundsxy = (0, 0, 0, 0)\n",
    "box = streams.BoundsXY(source=points, bounds=boundsxy)\n",
    "bounds = hv.DynamicMap(lambda bounds: hv.Bounds(bounds), streams=[box])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the datashader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p = dynspread(datashade(points, cmap=\"Viridis\"))\n",
    "p = p.opts(width=800, height=300, padding=0.05, show_grid=True,\n",
    "           xlim=(-2.0, 7.0), ylim=(-5.0, 3.0), xlabel=\"(g-r)\", ylabel=\"(r-i)\",\n",
    "           tools=['box_select'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Render the datashaded plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p * bounds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 12: Same as Fig 11, but with many more objects included.\n",
    "\n",
    "Above, use the box zoom or wheel zoom functionality to zoom in on the plot,\n",
    "and watch how datashader adapts the view to optimally display the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3. Adding a callback function\n",
    "\n",
    "Add callback functionality to the color-color diagram above to retrieve the indices of selected points.\n",
    "\n",
    "Above, use the box selection tool to select data. \n",
    "\n",
    "> **STOP** - Select some data points from the plot above using the box select tool before proceeding\n",
    "\n",
    "Print the number of objects selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "selection = (points.data.x > box.bounds[0]) \\\n",
    "    & (points.data.y > box.bounds[1]) \\\n",
    "    & (points.data.x < box.bounds[2]) \\\n",
    "    & (points.data.y < box.bounds[3])\n",
    "print('The selection box contains %i data points'%(np.sum(selection)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4. Interactive selection\n",
    "\n",
    "Below, create two side-by-side plots.\n",
    "\n",
    "The left-hand plot will show the datashaded spatial distribution on the sky,\n",
    "and the right-hand plot will be a linked and brushed plot showing the r-band\n",
    "magnitude distribution for objects selected in the left-hand plot.\n",
    "It will be possible to use the box selection in the spatial distribution plot\n",
    "to change which data are included in the histogram.\n",
    "\n",
    "First, create a holoviews dataset instance, and label some of the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "kdims = [('coord_ra', 'RA(deg)'), ('coord_dec', 'dec(deg)')]\n",
    "vdims = [('mag_r_cModel', 'r(mag)')]\n",
    "ds = hv.Dataset(data, kdims, vdims)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "points = hv.Points(ds)\n",
    "boundsxy = (np.min(ds.data['coord_ra']), np.min(ds.data['coord_dec']),\n",
    "            np.max(ds.data['coord_ra']), np.max(ds.data['coord_dec']))\n",
    "box = streams.BoundsXY(source=points, bounds=boundsxy)\n",
    "box_plot = hv.DynamicMap(lambda bounds: hv.Bounds(bounds), streams=[box])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create custom callback functionality to update the linked histogram.\n",
    "\n",
    "These functions are defined here (and not in Section 1.2) because they are only used in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def log_inf(x) -> float:\n",
    "    return np.log(x) if x > 0 else 0\n",
    "\n",
    "def update_histogram(bounds=bounds) -> hv.Histogram:\n",
    "    selection = (ds.data['coord_ra'] > bounds[0]) & \\\n",
    "                (ds.data['coord_dec'] > bounds[1]) & \\\n",
    "                (ds.data['coord_ra'] < bounds[2]) & \\\n",
    "                (ds.data['coord_dec'] < bounds[3])\n",
    "    selected_mag = ds.data.loc[selection]['mag_r_cModel']\n",
    "    frequencies, edges = np.histogram(selected_mag)\n",
    "    hist = hv.Histogram(\n",
    "        (list(map(log_inf, frequencies)), edges)).opts(xlabel='r (mag)')\n",
    "    return hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dmap = hv.DynamicMap(\n",
    "    update_histogram, streams=[box]).options(height=400, width=400)\n",
    "datashade(points, cmap=process_cmap(\"Viridis\", provider=\"bokeh\")) * \\\n",
    "    box_plot.options(height=400, width=400, tools=['box_select']) + \\\n",
    "dmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Figure 13: At right, Dec vs. RA as a 2-dimensional density map with a purple-green-yellow colormap. At left, a histogram (blue bar chart) of the fraction of objects in bins of r-band magnitude.\n",
    "\n",
    "Try changing the box selection and watch as the histogram is recomputed and displayed. \n",
    "\n",
    "Finally, delete the job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Exercises for the learner\n",
    "\n",
    " 1. HoloViews works with a wide range of plotting libraries; Bokeh, matplotlib, plotly, mpld3, pygal to name a few. As an exercise, try changing the HoloViews plotting library to be `matplotlib` instead of `bokeh` at the beginning of the notebook with `hv.extension('matplotlib')`. Notice the holoviews + matplotlib icons displayed when the library is loaded successfully. Recreate a few plots and compare the outputs. Try again with some other plotting library. Don't forget to set the plotting library back to `Bokeh`, which is used for this tutorial. Note that some warnings might be raised.\n",
    "  \n",
    " 2. Try writing a different callback function to use in Section 5.3. \n",
    " \n",
    " 3. As in Section 4.3, try writing other interactive plots from this notebook to an interactive HTML file. Which ones can easily be output to an interactive HTML file? Which HTML files show full interactive functionality? Which ones don't?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}