diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 891e0e5b4..5ab02398f 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -92,3 +92,8 @@ jobs: - run: gh workflow run ci.yml --repo nextstrain/docker-base env: GITHUB_TOKEN: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_WORKFLOW_DISPATCH }} + build-docs: + uses: nextstrain/.github/.github/workflows/docs-ci.yaml@master + with: + docs-directory: docs/ + environment-file: docs/environment.yml diff --git a/CHANGELOG.md b/CHANGELOG.md index caad1b302..5793105af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,4 @@ ---- -title: Changelog ---- +# Changelog ## version 2.45.0 - 2023/03/08 diff --git a/docs/.gitignore b/docs/.gitignore index 3c2820b16..948330d9f 100644 --- a/docs/.gitignore +++ b/docs/.gitignore @@ -1,4 +1,3 @@ # read the docs generated files _templates -_static _build \ No newline at end of file diff --git a/docs/advanced-functionality/drag-drop-csv-tsv.md b/docs/advanced-functionality/drag-drop-csv-tsv.md deleted file mode 100644 index 7198d5be1..000000000 --- a/docs/advanced-functionality/drag-drop-csv-tsv.md +++ /dev/null @@ -1,117 +0,0 @@ -# Adding extra metadata via CSV/TSV/XLSX - -A common use case is to have additional metadata which you would like to add to the current dataset. -If you created the dataset itself, then you may wish to keep certain data out of the dataset, as it may change frequently or be sensitive information which you don't want to share publicly. - -Additional metadata (CSV / TSV / XLSX file(s)) can be dragged onto an existing dataset in Auspice. -These extra data are processed within the browser, so no information leaves the client, which can be useful for viewing private metadata. - -The general format is compatible with other popular tools such as [MicroReact](https://microreact.org/). -The first column defines the names of the strains / samples in the tree, while the first row (header row) defines the metadata names. -You can add as many columns you want, each will result in a different colouring of the data being made available. -The separator can be either a tab character or a comma & the file extension should be `.tsv` or `.csv`, respectively. -Excel files with file extension `.xlsx` are also supported, but the metadata must be in the first sheet of the workbook. -Older Excel files with the `.xls` extension are not supported. - -## Example: - -A TSV file as follows can be dragged onto [nextstrain.org/zika](https://nextstrain.org/zika) to add a "secret" color-by: - -```text -strain secret -USVI/19/2016 A -USVI/28/2016 B -USVI/41/2016 C -USVI/42/2016 C -``` - -![auspice with extra data shown via csv](../assets/csv-extra-data.png) - -A more complex metadata file may look like the following, which makes use of additional features available. -This defines colours for the metadata (e.g. `A` is yellow, `B` is orange) as well as associating strains with (made up) geographic coordinates. - -```text -strain secret secret__colour latitude longitude -USVI/19/2016 A #f4e409 0 -120 -USVI/28/2016 B #f49015 0 -115 -USVI/41/2016 C #710000 0 -100 -USVI/42/2016 C #710000 0 -120 -``` - -![auspice with extra data shown via csv](../assets/csv-extra-data-2.png) - - -## Adding extra colorings and filters - -Most metadata columns will be added as colourings; once the data has been added they should appear as new entries in the "Color By" dropdown (Left-hand sidebar of Auspice). -This means you can also filter by these traits using the "Filter Data" box. - -An extra colouring is automatically created to represent the set of samples which were in the CSV/TSV/XLSX file -- this allows you to easily filter the dataset to just those samples which you had in your metadata file. - -You can choose the colours you want to associate with values by adding in a separate column with the same name + `__colour` (see above example), or the suffix `__color` may also be used. -Currently the values in this column must be hex values such as `#3498db` (blue). -If the same value of metadata is associated with multiple, distinct, colours then the colours are blended together. - -## Adding geographic locations - -If the columns `latitude` and `longitude` exist (or `__latitude` and `__longitude`) then you can see these samples on the map. -This means that there will be a new geographic resolution available in the sidebar dropdown, labelled the same as the metadata filename you dropped on, which will plot the location on the map for those samples in the metadata file for which you provided positions for. - - -Additional metadata of this format defines lat-longs _per-sample_, which is different to Nextstrain's approach (where we associate a location to a metadata trait). -To resolve this, we create a new (dummy) trait whose values represent the unique lat/longs provided. -In the above example screenshot, note that auspice groups `USVI/19/2016` and `USVI/42/2016` together on the map as their lat/longs are identical; the other metadata columns (e.g. `secret`) are irrelevant in this case. - - -> P.S. If the dataset itself doesn't contain any geographic data, then adding metadata will trigger the map to be displayed. - - -## Privacy - -All data added via these additional metadata files remains in-browser, and doesn't leave your computer. -This makes it safe for sensitive data. - -## Schema - -The following fields are ignored completely. -(Some of these may be allowed in the future when we have increased the features available here.) - -```yaml -name -div -vaccine -labels -hidden -mutations -url -authors -accession -traits -children -date -num_date -year -month -day -``` - -Fields which end with certain strings are treated as follows: -- `__autocolour`: this suffix is dropped, but the column is otherwise parsed as normal -- `__colour`: see above section on adding colours -- `__shape`: this column is currently ignored - - -The following columns are interpreted as geographic locations (see section above) and therefore are not added as a colouring: -```yaml -__latitude -__longitude -latitude -longitude -``` - -The name of the first column is not used, but the first column is always taken to be the sample (strain) name. - -## Scale types -The type of the data is currently always categorical. -This means that while numeric data will work, it won't be very usable if there are many values. - diff --git a/docs/advanced-functionality/drag-drop-csv-tsv.rst b/docs/advanced-functionality/drag-drop-csv-tsv.rst new file mode 100644 index 000000000..cf1a11cb4 --- /dev/null +++ b/docs/advanced-functionality/drag-drop-csv-tsv.rst @@ -0,0 +1,110 @@ +Adding extra metadata via CSV/TSV/XLSX +====================================== + +A common use case is to have additional metadata which you would like to add to the current dataset. If you created the dataset itself, then you may wish to keep certain data out of the dataset, as it may change frequently or be sensitive information which you don't want to share publicly. + +Additional metadata (CSV / TSV / XLSX file(s)) can be dragged onto an existing dataset in Auspice. These extra data are processed within the browser, so no information leaves the client, which can be useful for viewing private metadata. + +The general format is compatible with other popular tools such as `MicroReact `__. The first column defines the names of the strains / samples in the tree, while the first row (header row) defines the metadata names. You can add as many columns you want, each will result in a different colouring of the data being made available. The separator can be either a tab character or a comma & the file extension should be ``.tsv`` or ``.csv``, respectively. Excel files with file extension ``.xlsx`` are also supported, but the metadata must be in the first sheet of the workbook. Older Excel files with the ``.xls`` extension are not supported. + +Example: +-------- + +A TSV file as follows can be dragged onto `nextstrain.org/zika `__ to add a "secret" color-by: + +.. code:: text + + strain secret + USVI/19/2016 A + USVI/28/2016 B + USVI/41/2016 C + USVI/42/2016 C + +.. figure:: ../assets/csv-extra-data.png + :alt: auspice with extra data shown via csv + + auspice with extra data shown via csv + +A more complex metadata file may look like the following, which makes use of additional features available. This defines colours for the metadata (e.g. ``A`` is yellow, ``B`` is orange) as well as associating strains with (made up) geographic coordinates. + +.. code:: text + + strain secret secret__colour latitude longitude + USVI/19/2016 A #f4e409 0 -120 + USVI/28/2016 B #f49015 0 -115 + USVI/41/2016 C #710000 0 -100 + USVI/42/2016 C #710000 0 -120 + +.. figure:: ../assets/csv-extra-data-2.png + :alt: auspice with extra data shown via csv + + auspice with extra data shown via csv + +Adding extra colorings and filters +---------------------------------- + +Most metadata columns will be added as colourings; once the data has been added they should appear as new entries in the "Color By" dropdown (Left-hand sidebar of Auspice). This means you can also filter by these traits using the "Filter Data" box. + +An extra colouring is automatically created to represent the set of samples which were in the CSV/TSV/XLSX file -- this allows you to easily filter the dataset to just those samples which you had in your metadata file. + +You can choose the colours you want to associate with values by adding in a separate column with the same name + ``__colour`` (see above example), or the suffix ``__color`` may also be used. Currently the values in this column must be hex values such as ``#3498db`` (blue). If the same value of metadata is associated with multiple, distinct, colours then the colours are blended together. + +Adding geographic locations +--------------------------- + +If the columns ``latitude`` and ``longitude`` exist (or ``__latitude`` and ``__longitude``) then you can see these samples on the map. This means that there will be a new geographic resolution available in the sidebar dropdown, labelled the same as the metadata filename you dropped on, which will plot the location on the map for those samples in the metadata file for which you provided positions for. + +Additional metadata of this format defines lat-longs *per-sample*, which is different to Nextstrain's approach (where we associate a location to a metadata trait). To resolve this, we create a new (dummy) trait whose values represent the unique lat/longs provided. In the above example screenshot, note that auspice groups ``USVI/19/2016`` and ``USVI/42/2016`` together on the map as their lat/longs are identical; the other metadata columns (e.g. ``secret``) are irrelevant in this case. + + P.S. If the dataset itself doesn't contain any geographic data, then adding metadata will trigger the map to be displayed. + +Privacy +------- + +All data added via these additional metadata files remains in-browser, and doesn't leave your computer. This makes it safe for sensitive data. + +Schema +------ + +The following fields are ignored completely. (Some of these may be allowed in the future when we have increased the features available here.) + +.. code:: yaml + + name + div + vaccine + labels + hidden + mutations + url + authors + accession + traits + children + date + num_date + year + month + day + +Fields which end with certain strings are treated as follows: + +- ``__autocolour``: this suffix is dropped, but the column is otherwise parsed as normal +- ``__colour``: see above section on adding colours +- ``__shape``: this column is currently ignored + +The following columns are interpreted as geographic locations (see section above) and therefore are not added as a colouring: + +.. code:: yaml + + __latitude + __longitude + latitude + longitude + +The name of the first column is not used, but the first column is always taken to be the sample (strain) name. + +Scale types +----------- + +The type of the data is currently always categorical. This means that while numeric data will work, it won't be very usable if there are many values. diff --git a/docs/advanced-functionality/misc.md b/docs/advanced-functionality/misc.md deleted file mode 100644 index 07487d5ab..000000000 --- a/docs/advanced-functionality/misc.md +++ /dev/null @@ -1,33 +0,0 @@ -# Miscellaneous - - -## How auspice handles unknown or missing data - -Attributes assigned to nodes in the tree -- such as `country` -- may have values which are _missing_ or _unknown_. -Auspice will ignore values such as these, and they will not be displayed in the legend or the tree info-boxes (e.g. hovering over the tree). -Tips & branches across the tree with values such as these will be gray. (Branches with low confidence for an inferred trait may also show as gray, and hovering over the branches will help identify this.) -Note that, if a discrete trait is selected, then a proportion of the pie-chart on the map may also be gray to represent the proportion of tips with missing data. - - -If a trait is not set on a node it is considered missing, as well as if (after coersion to lower-case) it has one of the following values: -```js -["unknown", "?", "nan", "na", "n/a", "", "unassigned"] -``` - -## GISAID specific changes to behavior - - -#### **GISAID data provenance annotation** -If the dataset JSON defines a data provenance named `"GISAID"` (`JSON → metadata → data_provenance`, see [schema](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json)), then there are two changes to Auspice behaviour: -1. The GISAID data provenance text (displayed at the top of the page in auspice) will be replaced with the GISAID logo, which is also a link to [gisaid.org](https://gisaid.org). -2. The available metadata for download is different. We now use a “Per-sample acknowledgments table” where each row is a strain in the tree, with the following columns: - - `strain` - - `gisaid_epi_isl` - - `genbank_accession` - - `originating_lab` - - `submitting_lab` - - `author` - -#### **Node annotated with `gisaid_epi_isl`** - -When hovering or clicking on tips (in the tree), nodes annotated with `gisaid_epi_isl` will behave slightly differently. These info-boxes will display “GISAID EPI ISL” and, in the tip-clicked info-panel, the value will also be a link to [gisaid.org](https://gisaid.org). diff --git a/docs/advanced-functionality/misc.rst b/docs/advanced-functionality/misc.rst new file mode 100644 index 000000000..a4b526781 --- /dev/null +++ b/docs/advanced-functionality/misc.rst @@ -0,0 +1,36 @@ +Miscellaneous +============= + +How auspice handles unknown or missing data +------------------------------------------- + +Attributes assigned to nodes in the tree -- such as ``country`` -- may have values which are *missing* or *unknown*. Auspice will ignore values such as these, and they will not be displayed in the legend or the tree info-boxes (e.g. hovering over the tree). Tips & branches across the tree with values such as these will be gray. (Branches with low confidence for an inferred trait may also show as gray, and hovering over the branches will help identify this.) Note that, if a discrete trait is selected, then a proportion of the pie-chart on the map may also be gray to represent the proportion of tips with missing data. + +If a trait is not set on a node it is considered missing, as well as if (after coersion to lower-case) it has one of the following values: + +.. code:: js + + ["unknown", "?", "nan", "na", "n/a", "", "unassigned"] + +GISAID specific changes to behavior +----------------------------------- + +GISAID data provenance annotation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the dataset JSON defines a data provenance named ``"GISAID"`` (``JSON → metadata → data_provenance``, see `schema `__), then there are two changes to Auspice behaviour: + +1. The GISAID data provenance text (displayed at the top of the page in auspice) will be replaced with the GISAID logo, which is also a link to `gisaid.org `__. +2. The available metadata for download is different. We now use a “Per-sample acknowledgments table” where each row is a strain in the tree, with the following columns: + + - ``strain`` + - ``gisaid_epi_isl`` + - ``genbank_accession`` + - ``originating_lab`` + - ``submitting_lab`` + - ``author`` + +Node annotated with ``gisaid_epi_isl`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When hovering or clicking on tips (in the tree), nodes annotated with ``gisaid_epi_isl`` will behave slightly differently. These info-boxes will display “GISAID EPI ISL” and, in the tip-clicked info-panel, the value will also be a link to `gisaid.org `__. diff --git a/docs/advanced-functionality/second-trees.md b/docs/advanced-functionality/second-trees.md deleted file mode 100644 index e1ff74f56..000000000 --- a/docs/advanced-functionality/second-trees.md +++ /dev/null @@ -1,26 +0,0 @@ -# Displaying multiple trees - -Auspice has the ability to display two trees side-by-side, and to draw lines between tips with the same name (aka tanglegrams). -This is useful to compare the shape of different trees, especially when they are from the same organism -- for instance comparing phylogenies constructed from different -segments of the same influenza virus can tell you a lot about the different histories of the segments which have the capacity to reassort (see image below). - -## How to load multiple trees - -You can compare any two datasets which you have available -- for instance if you had "flu/seasonal/h3n2/ha/2y" and "flu/seasonal/h3n2/na/2y" then loading the URL "flu/seasonal/h3n2/ha/2y:flu/seasonal/h3n2/na/2y" would load them both. -A toggle is made available in the sidebar to turn off the lines drawn between tips. - - -![two-trees](../assets/tangle.png) -*Comparing epitope mutations between HA and NA (worldwide influena H3N2).* -*Notice how the segments can differ drastically in how many epitope mutations they acquire!* -*While the crossing of the lines between the tips doesn't always prove reassortment, it's usually a good indication that reassortment is present.* - - -## Showing potential datasets in the sidebar -Depending on the way you've labelled your datasets, potential second trees are available in a sidebar dropdown. -These are defined by the [getAvailable API request](server/api.md#charon-getavailable). -Currently, the logic in `auspice view` is to match all datasets which: -- contain the same first "part" of the URL -- interpreted here to represent the same pathogen. -- have the same number of "parts" in the URL (parts are delimited by a `_` in the filename or a `/` in the URL). -- differ from the currently selected dataset by only 1 part. - diff --git a/docs/advanced-functionality/second-trees.rst b/docs/advanced-functionality/second-trees.rst new file mode 100644 index 000000000..ee5554868 --- /dev/null +++ b/docs/advanced-functionality/second-trees.rst @@ -0,0 +1,22 @@ +Displaying multiple trees +========================= + +Auspice has the ability to display two trees side-by-side, and to draw lines between tips with the same name (aka tanglegrams). This is useful to compare the shape of different trees, especially when they are from the same organism -- for instance comparing phylogenies constructed from different segments of the same influenza virus can tell you a lot about the different histories of the segments which have the capacity to reassort (see image below). + +How to load multiple trees +-------------------------- + +You can compare any two datasets which you have available -- for instance if you had "flu/seasonal/h3n2/ha/2y" and "flu/seasonal/h3n2/na/2y" then loading the URL "flu/seasonal/h3n2/ha/2y:flu/seasonal/h3n2/na/2y" would load them both. A toggle is made available in the sidebar to turn off the lines drawn between tips. + +|two-trees| *Comparing epitope mutations between HA and NA (worldwide influena H3N2).* *Notice how the segments can differ drastically in how many epitope mutations they acquire!* *While the crossing of the lines between the tips doesn't always prove reassortment, it's usually a good indication that reassortment is present.* + +Showing potential datasets in the sidebar +----------------------------------------- + +Depending on the way you've labelled your datasets, potential second trees are available in a sidebar dropdown. These are defined by the :ref:`getAvailable API request `. Currently, the logic in ``auspice view`` is to match all datasets which: + +- contain the same first "part" of the URL -- interpreted here to represent the same pathogen. +- have the same number of "parts" in the URL (parts are delimited by a ``_`` in the filename or a ``/`` in the URL). +- differ from the currently selected dataset by only 1 part. + +.. |two-trees| image:: ../assets/tangle.png diff --git a/docs/advanced-functionality/view-settings.md b/docs/advanced-functionality/view-settings.md deleted file mode 100644 index b9bb473c8..000000000 --- a/docs/advanced-functionality/view-settings.md +++ /dev/null @@ -1,90 +0,0 @@ -# View Settings - - -View settings refer to things such as how we display the tree (radial? root-to-tip?), what panels we display (map? tree? both?), what colouring we are using etcetera. -There are three ways these can be controlled: -1. The defaults are configured by the dataset creators (and stored as "display defaults" in the dataset JSON). -This allows -2. Interacting with the visualisation (e.g. changing the color-by) modifies the view, and the URL is changed accordingly. -For instance, change [nextstrain.org/zika](https://nextstrain.org/zika) to have a color-by of author, and you'll see the URL silently update to [?c=author](https://nextstrain.org/zika?c=author). -If you reload the page or share this URL, then the color-by is set via this URL. -3. Narratives, in which the narrative author chooses different "views" for each page, are created by associating each page with a URL (see (2)) which defines a specific view into the data. - -## Auspice (hardcoded) defaults - -Auspice has some hardcoded defaults, largely for historical reasons. -Each of these can be overridden by the JSON `display_defaults`, and then the view can be further modified by the URL query (see below). - -* Default phylogeny layout is rectangular. -* Default phylogeny distance measure is time, if available. -* Default geographic resolution is "country", if available. -* Default colouring is "country", if available. -* Default branch labelling is "clade", if available. - -## Dataset (JSON) configurable defaults - -These are exported as the (optional) property of the dataset JSON `meta.display_defaults` (see JSON schema [here](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json)). -The defaults (as set here) will be what are displayed when the page is loaded with no URL queries, but be aware that URL queries (see below) can modify how the view looks. -For instance, if you set `display_defaults.color_by` to `country`, but load the page with `?c=region` then the view will be coloured by region. - -| Property | Description | Example | -| ------------- | ------------- | ------- | -| `color_by` | Colouring | "country" | -| `geo_resolution` | Geographic resolution | "country" | -| `distance_measure` | Phylogeny x-axis measure | "div" or "num_date" | -| `map_triplicate` | Should the map repeat, so that you can pan further in each direction? | Boolean | -| `layout` | Tree layout | "rect", "radial", "unrooted", "clock" or "scatter" | -| `branch_label` | Which set of branch labels are to be displayed | "aa", "lineage" | -| `sidebar` | Should the sidebar start open or closed? | "open" or "closed" | -| `panels` | List of panels which (if available) are to be displayed | ["tree", "map"] | -| `transmission_lines`| Should transmission lines (if available) be rendered on the map? | Boolean | -| `language` | Language to display Auspice in | "ja" | - -Note that `meta.display_defaults.panels` (optional) differs from `meta.panels` (required), where the latter lists the possible panels that auspice may display for the dataset. -See the [JSON schema](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json) for more details. - -**See this in action:** - -For instance, go to [nextstrain.org/flu/seasonal/h3n2/ha/2y](https://nextstrain.org/flu/seasonal/h3n2/ha/2y) and you'll see how the colouring is "Clade" -- this has been set via the `display_defaults` in the JSON. - -## URL query options - -URL queries are the part of the URL coming after the `?` character, and typically consist of `key=value` -- for instance [nextstrain.org/zika?c=author](https://nextstrain.org/zika?c=author) has a query with a key `c` and value `author`. -Multiple queries are separated by the `&` character. -All URL queries modify the view away from the default settings -- if you change back to a default then that URL query will disappear. - -| Key | Description | Example(s) | -| --- | ----------- | -------------- | -| `c` | Colouring to use | `c=author`, `c=region` | -| `r` | Geographic resolution | `r=region` | -| `m` | Phylogeny x-axis measure | `m=div` | -| `l` | Phylogeny layout | `l=clock` | -| `scatterX` | Scatterplot X variable | `scatterX=num_date` | -| `scatterY` | Scatterplot Y variable | `scatterY=num_date` | -| `branches` | Hide branches | `branches=hide` | -| `regression` | Show/Hide regression line | `regression=hide`, `regression=show` | -| `transmissions` | Hide transmission lines | `transmissions=hide`| -| `lang` | Language | `lang=ja` (Japanese) | -| `dmin` | Temporal range (minimum) | `dmin=2008-05-13` | -| `dmax` | Temporal range (maximum) | `dmax=2010-05-13` | -| `f_` | Data filter. Multiple values per key are `,` separated. | `f_region=Oceania` | -| `gt` | Genotype filtering | -| `d` | List of panels to display, `,` separated | `d=tree,map` | -| `p` | Panel layout (buggy!) | `p=full`, `p=grid` | -| `gmin` | Entropy panel zoom (minimum) bound | `gmin=1000` | -| `gmax` | Entropy panel zoom (maximum) bound | `gmax=2000` | -| `animate` | Animation settings | | -| `n` | Narrative page number | `n=1` goes to the first page | -| `s` | Selected strain | `s=1_0199_PF` | -| `branchLabel` | Branch labels to display | `branchLabel=aa` | -| `showBranchLabels` | Force all branch labels to be displayed | `showBranchLabels=all` | -| `label` | Labeled branch that tree is zoomed to | `label=clade:B3`, `label=lineage:relapse` | -| `clade` | _DEPRECATED_ Labeled clade that tree is zoomed to | `clade=B3` should now become `label=clade:B3` | -| `sidebar` | Force the sidebar into a certain state | `sidebar=closed` or `sidebar=open` | -| `legend` | Force the legend into a certain state | `legend=closed` or `legend=open` | -| `onlyPanels` | Do not display the footer / header. Useful for iframes. | `onlyPanels` | -| `ci` | Display confidence intervals on the tree. | `ci` | - -**See this in action:** - -For instance, go to [nextstrain.org/flu/seasonal/h3n2/ha/2y?c=num_date&d=tree,map&m=div&r=region](https://nextstrain.org/flu/seasonal/h3n2/ha/2y?c=num_date&d=tree,map&m=div&p=grid&r=region) and you'll see how we've changed the coloring to a temporal scale (`c=num_date`), we're only showing the tree & map panels (`d=tree,map`), the tree x-axis is divergence (`m=div`) and the map resolution is region (`r=region`). diff --git a/docs/advanced-functionality/view-settings.rst b/docs/advanced-functionality/view-settings.rst new file mode 100644 index 000000000..3750ce740 --- /dev/null +++ b/docs/advanced-functionality/view-settings.rst @@ -0,0 +1,125 @@ +View Settings +============= + +View settings refer to things such as how we display the tree (radial? root-to-tip?), what panels we display (map? tree? both?), what colouring we are using etcetera. There are three ways these can be controlled: + +1. The defaults are configured by the dataset creators (and stored as "display defaults" in the dataset JSON). This allows +2. Interacting with the visualisation (e.g. changing the color-by) modifies the view, and the URL is changed accordingly. For instance, change `nextstrain.org/zika `__ to have a color-by of author, and you'll see the URL silently update to `?c=author `__. If you reload the page or share this URL, then the color-by is set via this URL. +3. Narratives, in which the narrative author chooses different "views" for each page, are created by associating each page with a URL (see (2)) which defines a specific view into the data. + +Auspice (hardcoded) defaults +---------------------------- + +Auspice has some hardcoded defaults, largely for historical reasons. Each of these can be overridden by the JSON ``display_defaults``, and then the view can be further modified by the URL query (see below). + +- Default phylogeny layout is rectangular. +- Default phylogeny distance measure is time, if available. +- Default geographic resolution is "country", if available. +- Default colouring is "country", if available. +- Default branch labelling is "clade", if available. + +Dataset (JSON) configurable defaults +------------------------------------ + +These are exported as the (optional) property of the dataset JSON ``meta.display_defaults`` (see JSON schema `here `__). The defaults (as set here) will be what are displayed when the page is loaded with no URL queries, but be aware that URL queries (see below) can modify how the view looks. For instance, if you set ``display_defaults.color_by`` to ``country``, but load the page with ``?c=region`` then the view will be coloured by region. + ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| Property | Description | Example | ++===========================+=======================================================================+====================================================+ +| ``color_by`` | Colouring | "country" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``geo_resolution`` | Geographic resolution | "country" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``distance_measure`` | Phylogeny x-axis measure | "div" or "num_date" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``map_triplicate`` | Should the map repeat, so that you can pan further in each direction? | Boolean | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``layout`` | Tree layout | "rect", "radial", "unrooted", "clock" or "scatter" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``branch_label`` | Which set of branch labels are to be displayed | "aa", "lineage" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``sidebar`` | Should the sidebar start open or closed? | "open" or "closed" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``panels`` | List of panels which (if available) are to be displayed | ["tree", "map"] | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``transmission_lines`` | Should transmission lines (if available) be rendered on the map? | Boolean | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ +| ``language`` | Language to display Auspice in | "ja" | ++---------------------------+-----------------------------------------------------------------------+----------------------------------------------------+ + +Note that ``meta.display_defaults.panels`` (optional) differs from ``meta.panels`` (required), where the latter lists the possible panels that auspice may display for the dataset. See the `JSON schema `__ for more details. + +**See this in action:** + +For instance, go to `nextstrain.org/flu/seasonal/h3n2/ha/2y `__ and you'll see how the colouring is "Clade" -- this has been set via the ``display_defaults`` in the JSON. + +URL query options +----------------- + +URL queries are the part of the URL coming after the ``?`` character, and typically consist of ``key=value`` -- for instance `nextstrain.org/zika?c=author `__ has a query with a key ``c`` and value ``author``. Multiple queries are separated by the ``&`` character. All URL queries modify the view away from the default settings -- if you change back to a default then that URL query will disappear. + ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| Key | Description | Example(s) | ++======================+===========================================================+===================================================+ +| ``c`` | Colouring to use | ``c=author``, ``c=region`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``r`` | Geographic resolution | ``r=region`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``m`` | Phylogeny x-axis measure | ``m=div`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``l`` | Phylogeny layout | ``l=clock`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``scatterX`` | Scatterplot X variable | ``scatterX=num_date`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``scatterY`` | Scatterplot Y variable | ``scatterY=num_date`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``branches`` | Hide branches | ``branches=hide`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``regression`` | Show/Hide regression line | ``regression=hide``, ``regression=show`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``transmissions`` | Hide transmission lines | ``transmissions=hide`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``lang`` | Language | ``lang=ja`` (Japanese) | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``dmin`` | Temporal range (minimum) | ``dmin=2008-05-13`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``dmax`` | Temporal range (maximum) | ``dmax=2010-05-13`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``f_`` | Data filter. Multiple values per key are ``,`` separated. | ``f_region=Oceania`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``gt`` | Genotype filtering | | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``d`` | List of panels to display, ``,`` separated | ``d=tree,map`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``p`` | Panel layout (buggy!) | ``p=full``, ``p=grid`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``gmin`` | Entropy panel zoom (minimum) bound | ``gmin=1000`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``gmax`` | Entropy panel zoom (maximum) bound | ``gmax=2000`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``animate`` | Animation settings | | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``n`` | Narrative page number | ``n=1`` goes to the first page | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``s`` | Selected strain | ``s=1_0199_PF`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``branchLabel`` | Branch labels to display | ``branchLabel=aa`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``showBranchLabels`` | Force all branch labels to be displayed | ``showBranchLabels=all`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``label`` | Labeled branch that tree is zoomed to | ``label=clade:B3``, ``label=lineage:relapse`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``clade`` | *DEPRECATED* Labeled clade that tree is zoomed to | ``clade=B3`` should now become ``label=clade:B3`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``sidebar`` | Force the sidebar into a certain state | ``sidebar=closed`` or ``sidebar=open`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``legend`` | Force the legend into a certain state | ``legend=closed`` or ``legend=open`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``onlyPanels`` | Do not display the footer / header. Useful for iframes. | ``onlyPanels`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ +| ``ci`` | Display confidence intervals on the tree. | ``ci`` | ++----------------------+-----------------------------------------------------------+---------------------------------------------------+ + +**See this in action:** + +For instance, go to `nextstrain.org/flu/seasonal/h3n2/ha/2y?c=num_date&d=tree,map&m=div&r=region `__ and you'll see how we've changed the coloring to a temporal scale (``c=num_date``), we're only showing the tree & map panels (``d=tree,map``), the tree x-axis is divergence (``m=div``) and the map resolution is region (``r=region``). diff --git a/docs/conf.py b/docs/conf.py index 538c9bf70..4aa61de59 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -44,7 +44,6 @@ # This pattern also affects html_static_path and html_extra_path. exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'README.md', 'narratives', - 'contributing', ] @@ -55,11 +54,6 @@ # html_theme = 'nextstrain-sphinx-theme' -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - html_theme_options = { 'logo_only': False, # if True, don't display project name at top of the sidebar 'collapse_navigation': False, # if True, no [+] icons in sidebar diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst deleted file mode 100644 index 081d99120..000000000 --- a/docs/contributing/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -====================================== -Contributing -====================================== - -This page shouldn't be rendered & should jump to the first TOC entry instead. TODO. - -.. toctree:: - :hidden: - :maxdepth: 0 - :titlesonly: - - overview diff --git a/docs/contributing/overview.md b/docs/contributing/overview.md deleted file mode 120000 index e54d03a0a..000000000 --- a/docs/contributing/overview.md +++ /dev/null @@ -1 +0,0 @@ -../../DEV_DOCS.md \ No newline at end of file diff --git a/docs/customise-client/api.md b/docs/customise-client/api.md deleted file mode 100644 index dc298aaa5..000000000 --- a/docs/customise-client/api.md +++ /dev/null @@ -1,166 +0,0 @@ -# Client Customisation API - -> The functionality detailed in this page needs more attention, both in terms of testing and code development. -We expect there to be some bugs and possible API changes. -If you rely on this functionality, we recommend you pin your installation of Auspice to a specific version. -Please [get in touch with us](mailto:hello@nextstrain.org) if you are using these customisations so that we can work with you! - - -This page details the available options and format of the customisations available at (client) build time. -They are contained in a JSON file supplied to Auspice via -```bash -auspice build --extend -``` - - -*Note that the hot-reloading development functionality does not work for code which is included via this client customisation mechanism.* -*Thus, while you can run `auspice develop --extend ` it will not update as you may expect!* - - -## Available Customisations -The following are definable as top-level keys of the JSON file. -A useful reference may be the [customisation JSON file](https://github.com/nextstrain/nextstrain.org/blob/master/auspice-client/customisations/config.json) used by nextstrain.org. - -* `sidebarTheme` allows modifications to the aesthetics of the sidebar. See below. -* `navbarComponent` a (relative) path to a JS file exporting a React component to be rendered as the nav bar. See below. -* `splashComponent` a (relative) path to a JS file exporting a React component to be rendered as the splash page. See below. -* `browserTitle` The browser title for the page. Defaults to "auspice" if not defined. -* `finePrint` String of Markdown to add to the "fine print" at the bottom of pages. -* `plausibleDataDomain` plausible.io analytics (see below) -* `googleAnalyticsKey` You can specify a Google Analytics key to enable (some) analytics functionality. This is deprecated and will be removed from an upcoming release. -* `serverAddress` Specify the address / prefix which the auspice client uses for API requests. -* `mapTiles` Specify the address (and other information) for the tiles used to render the map. - - -> Please remember to make any modifications, including customisations described here, publicly available. See [the previous page](./index.rst) for more details. - ---- - -### Sidebar Theme - -The appearence of the sidebar can be customised by specifing a theme in the config JSON used to build Auspice. -This theme is then available (via [styled-components](https://www.styled-components.com/)) to the components rendered in the sidebar. -It is also passed to the nav bar component (see below) as the `theme` prop. - -For instance, here is the customisation used by nextstrain.org: - -```json -{ - "sidebarTheme": { - "background": "#F2F2F2", - "color": "#000", - "sidebarBoxShadow": "rgba(0, 0, 0, 0.2)", - "font-family": "Lato, Helvetica Neue, Helvetica, sans-serif", - "selectedColor": "#5097BA", - "unselectedColor": "#333" - } -} -``` - - -| Properties | CSS string of | Description | -| ------------- |--------------- | ------ | -| color | color | Text color | -| selectedColor | color | Text color of selected text / button text | -| unselectedColor | color | Text color of unselected text / button text | -| font-family | font | Inner shadow of the sidebar on the right hand side | -| background | color | Background color | - - - -## Components - -One way to extend Auspice is by replacing React components with your own custom components. -These custom components will receive props defined here, which can be used to update the rendering of the component using the normal react lifecycle methods. -Right now this is only available for the splash page and nav-bar components, whose interfaces are defined here. - -Each component must be the default export of a javascript file which is specified in the (client) config JSON passed to Auspice at build time (`auspice build` or `auspice develop`). - - -### Nav-bar Component - -**Build config:** -```json -{ - "navbarComponent": "" -} -``` - -Where the javascript file contains a default export of a React component. - -**React Props Available:** - -| Prop | Type | Description | -| ----------- |--------- | ------ | -| `narrativeTitle` | String | | -| `sidebar ` | Bool | Is it to be displayed in the sidebar? | -| `width ` | Number | Width of the sidebar, in pixels | -| `theme ` | Object | See above. Use this to style components. | - - - -### Splash component - -Define a custom splash page for Auspice. Please note that this is extremely expirimental and the interface is expected to change. - -**Build config:** -```json -{ - "splashComponent": "" -} -``` -Where the javascript file contains a default export of a React component. - -**React Props available:** - -| Prop | Type | Description | -| ----------- |--------- | ------ | -| `isMobile` | Bool | | -| `available` | Object | available datasets and narratives | -| `browserDimensions` | Object | Browser width & height | -| `dispatch` | function | access to redux's dispatch mechanism | -| `errorMessage` | function | to do | -| `changePage` | function | to do | - ---- - -### Specifying the API server address - -By default, the client makes API requests ([as detailed here](requests.md)) to "/charon/getAvailable", "/charon/getDataset" etc. -This is using the default server address of "/charon". -This can be changed by specifying `serverAddress` in the customisation JSON. - -> Note that if you specify a `serverAddress` on a different origin (protocol + domain + port) than Auspice, the server will need to send CORS headers to permit the requests from Auspice. - ---- - -### Custom Map tiles - -Auspice uses [Leaflet](https://leafletjs.com/) to render the map, which requires access to a tile set in order to render the geography. -By default, auspice uses [Mapbox](https://www.mapbox.com/) for these tiles, and we make these available for local use of auspice. -If you are distributing your own version of auspice (i.e. not running it locally) you must set an appropriate API address here so that the map can fetch suitable tiles. - -```json -{ - "mapTiles": { - "api": "API address for Leaflet to fetch map tiles", - "attribution": "HTML-formatted attribution string to be displayed in bottom-right-hand corner of map", - "mapboxWordmark": "(optional) should the Mapbox logo be displayed in the bottom-left of the map? (boolean)" - } -} -``` - -Please see [this discussion post](https://discussion.nextstrain.org/t/build-with-newest-nextstrain-ncov-has-api-requests-to-mapbox-403-forbidden/396/11?u=james) for a hands-on guide to setting custom map tile info. -For some examples of other tile sets you may use, see the [OpenStreetMap wiki](https://wiki.openstreetmap.org/wiki/Tile_servers), and please remember to adhere to the licenses and terms of use for each tile server. -The API address contains parameters as specified by the [Leaflet API](https://docs.mapbox.com/api/overview/). - ---- - -### Tracking Analytics - -Auspice has in-built support for [Plausible Analytics](https://plausible.io/docs). -To enable this you will need to provide the `plausibleDataDomain` in your extensions. -The analytics are not included when running Auspice in development mode. - -Auspice has support for Google Analytics but this is deprecated and will be removed in a future release. -Google Analytics run when the `googleAnalyticsKey` extension is set and only run in production mode. diff --git a/docs/customise-client/api.rst b/docs/customise-client/api.rst new file mode 100644 index 000000000..e750e95c1 --- /dev/null +++ b/docs/customise-client/api.rst @@ -0,0 +1,179 @@ +Client Customisation API +======================== + +.. warning:: + + The functionality detailed in this page needs more attention, both in terms of testing and code development. We expect there to be some bugs and possible API changes. If you rely on this functionality, we recommend you pin your installation of Auspice to a specific version. Please `get in touch with us `__ if you are using these customisations so that we can work with you! + +This page details the available options and format of the customisations available at (client) build time. They are contained in a JSON file supplied to Auspice via + +.. code:: bash + + auspice build --extend + +.. note:: + + The hot-reloading development functionality does not work for code which is included via this client customisation mechanism. Thus, while you can run ``auspice develop --extend `` it will not update as you may expect! + +.. _client-api-available-customisations: + +Available Customisations +------------------------ + +The following are definable as top-level keys of the JSON file. A useful reference may be the `customisation JSON file `__ used by nextstrain.org. + +- ``sidebarTheme`` allows modifications to the aesthetics of the sidebar. See below. +- ``navbarComponent`` a (relative) path to a JS file exporting a React component to be rendered as the nav bar. See below. +- ``splashComponent`` a (relative) path to a JS file exporting a React component to be rendered as the splash page. See below. +- ``browserTitle`` The browser title for the page. Defaults to "auspice" if not defined. +- ``finePrint`` String of Markdown to add to the "fine print" at the bottom of pages. +- ``plausibleDataDomain`` plausible.io analytics (see below) +- ``googleAnalyticsKey`` You can specify a Google Analytics key to enable (some) analytics functionality. This is deprecated and will be removed from an upcoming release. +- ``serverAddress`` Specify the address / prefix which the auspice client uses for API requests. +- ``mapTiles`` Specify the address (and other information) for the tiles used to render the map. + +.. note:: + + Please remember to make any modifications, including customisations described here, publicly available. See :doc:`the previous page ` for more details. + +-------------- + +Sidebar Theme +~~~~~~~~~~~~~ + +The appearence of the sidebar can be customised by specifing a theme in the config JSON used to build Auspice. This theme is then available (via `styled-components `__) to the components rendered in the sidebar. It is also passed to the nav bar component (see below) as the ``theme`` prop. + +For instance, here is the customisation used by nextstrain.org: + +.. code:: json + + { + "sidebarTheme": { + "background": "#F2F2F2", + "color": "#000", + "sidebarBoxShadow": "rgba(0, 0, 0, 0.2)", + "font-family": "Lato, Helvetica Neue, Helvetica, sans-serif", + "selectedColor": "#5097BA", + "unselectedColor": "#333" + } + } + ++--------------------------+------------------------------+----------------------------------------------------+ +| Properties | CSS string of | Description | ++==========================+==============================+====================================================+ +| color | color | Text color | ++--------------------------+------------------------------+----------------------------------------------------+ +| selectedColor | color | Text color of selected text / button text | ++--------------------------+------------------------------+----------------------------------------------------+ +| unselectedColor | color | Text color of unselected text / button text | ++--------------------------+------------------------------+----------------------------------------------------+ +| font-family | font | Inner shadow of the sidebar on the right hand side | ++--------------------------+------------------------------+----------------------------------------------------+ +| background | color | Background color | ++--------------------------+------------------------------+----------------------------------------------------+ + +Components +---------- + +One way to extend Auspice is by replacing React components with your own custom components. These custom components will receive props defined here, which can be used to update the rendering of the component using the normal react lifecycle methods. Right now this is only available for the splash page and nav-bar components, whose interfaces are defined here. + +Each component must be the default export of a javascript file which is specified in the (client) config JSON passed to Auspice at build time (``auspice build`` or ``auspice develop``). + +Nav-bar Component +~~~~~~~~~~~~~~~~~ + +**Build config:** + +.. code:: json + + { + "navbarComponent": "" + } + +Where the javascript file contains a default export of a React component. + +**React Props Available:** + ++-----------------------------+-----------------------+------------------------------------------+ +| Prop | Type | Description | ++=============================+=======================+==========================================+ +| ``narrativeTitle`` | String | | ++-----------------------------+-----------------------+------------------------------------------+ +| ``sidebar`` | Bool | Is it to be displayed in the sidebar? | ++-----------------------------+-----------------------+------------------------------------------+ +| ``width`` | Number | Width of the sidebar, in pixels | ++-----------------------------+-----------------------+------------------------------------------+ +| ``theme`` | Object | See above. Use this to style components. | ++-----------------------------+-----------------------+------------------------------------------+ + +Splash component +~~~~~~~~~~~~~~~~ + +Define a custom splash page for Auspice. Please note that this is extremely expirimental and the interface is expected to change. + +**Build config:** + +.. code:: json + + { + "splashComponent": "" + } + +Where the javascript file contains a default export of a React component. + +**React Props available:** + ++-----------------------------+-----------------------+--------------------------------------+ +| Prop | Type | Description | ++=============================+=======================+======================================+ +| ``isMobile`` | Bool | | ++-----------------------------+-----------------------+--------------------------------------+ +| ``available`` | Object | available datasets and narratives | ++-----------------------------+-----------------------+--------------------------------------+ +| ``browserDimensions`` | Object | Browser width & height | ++-----------------------------+-----------------------+--------------------------------------+ +| ``dispatch`` | function | access to redux's dispatch mechanism | ++-----------------------------+-----------------------+--------------------------------------+ +| ``errorMessage`` | function | to do | ++-----------------------------+-----------------------+--------------------------------------+ +| ``changePage`` | function | to do | ++-----------------------------+-----------------------+--------------------------------------+ + +-------------- + +Specifying the API server address +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By default, the client makes API requests (:doc:`as detailed here `) to "/charon/getAvailable", "/charon/getDataset" etc. This is using the default server address of "/charon". This can be changed by specifying ``serverAddress`` in the customisation JSON. + +.. note:: + + If you specify a ``serverAddress`` on a different origin (protocol + domain + port) than Auspice, the server will need to send CORS headers to permit the requests from Auspice. + +-------------- + +Custom Map tiles +~~~~~~~~~~~~~~~~ + +Auspice uses `Leaflet `__ to render the map, which requires access to a tile set in order to render the geography. By default, auspice uses `Mapbox `__ for these tiles, and we make these available for local use of auspice. If you are distributing your own version of auspice (i.e. not running it locally) you must set an appropriate API address here so that the map can fetch suitable tiles. + +.. code:: json + + { + "mapTiles": { + "api": "API address for Leaflet to fetch map tiles", + "attribution": "HTML-formatted attribution string to be displayed in bottom-right-hand corner of map", + "mapboxWordmark": "(optional) should the Mapbox logo be displayed in the bottom-left of the map? (boolean)" + } + } + +Please see `this discussion post `__ for a hands-on guide to setting custom map tile info. For some examples of other tile sets you may use, see the `OpenStreetMap wiki `__, and please remember to adhere to the licenses and terms of use for each tile server. The API address contains parameters as specified by the `Leaflet API `__. + +-------------- + +Tracking Analytics +~~~~~~~~~~~~~~~~~~ + +Auspice has in-built support for `Plausible Analytics `__. To enable this you will need to provide the ``plausibleDataDomain`` in your extensions. The analytics are not included when running Auspice in development mode. + +Auspice has support for Google Analytics but this is deprecated and will be removed in a future release. Google Analytics run when the ``googleAnalyticsKey`` extension is set and only run in production mode. diff --git a/docs/customise-client/requests.md b/docs/customise-client/requests.md deleted file mode 100644 index fc4218f22..000000000 --- a/docs/customise-client/requests.md +++ /dev/null @@ -1,50 +0,0 @@ -# Requests Made from the Client - -The Auspice client will make a number of requests to various endpoints, which this document aims to provide an overview of. -Some of these are dependent on the [build-time customisations](./api) in use, and we plan to allow more flexibility here using this mechanism. - - -## Requests to the Auspice Server -For the purposes of this section we are assuming you are running locally on "localhost:4000", however the idea is the same if Auspice is deployed to a server. - - -### Auspice Client JavaScript Code -The (bundled) client code (which is generated by `auspice build`) is broken up into a number of "chunks" which are delivered to the client as needed. -These originate from the following block in the `index.html` file: -```html - -``` - -### "Charon" GET requests -When a dataset, narrative, or listing of available datasets is to be displayed in Auspice, a selection of the following requests are made to localhost:4000. - -* `/charon/getAvailable` -- return a list of available datasets and narratives -* `/charon/getDataset` -- return the requested dataset -* `/charon/getNarrative` -- return the requested narrative - -See [the server API documentation](../server/api.md) for more details about the requests, or the [client customisation API](api.md) for the ability to change "/charon" to a different value. - -### Image Requests - -The initial Auspice page (i.e. the one which displays a listing of available datasets) requests some images for the footer. - -## External Requests - -### Fonts -Two initial requests for fonts are made, for [Google-hosted Lato fonts (CSS)](https://fonts.googleapis.com/css?family=Lato:100,200,300,400,500,700) and [font-awesome (CSS)](https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">). -These in turn make a number of subsequent requests for the individual font files etc. - - -### Leaflet -We use [Leaflet](https://leafletjs.com/) to display the map tiles. -There is one [initial CSS request](https://unpkg.com/leaflet@1.0.1/dist/leaflet.css) which is always made, regardless of whether a map is displayed. - -If a map is displayed by Auspice the individual tiles are requested from api.mapbox.com. -Panning and zooming of the map result in futher requests for tiles. -These are requested using our hardcoded access key, however this may change to a build-time-customisation in the future. - - -### Google Analytics (Optional) -Auspice has the potential to include Google Analytics in a limited fashion. -See [the client customisation API documentation](customise-client/api.md#available-customisations) for how to set this. -Note that these requests are only made if you specify a Google Analytics key during build-time customisation of the client. diff --git a/docs/customise-client/requests.rst b/docs/customise-client/requests.rst new file mode 100644 index 000000000..eb4351774 --- /dev/null +++ b/docs/customise-client/requests.rst @@ -0,0 +1,54 @@ +Requests Made from the Client +============================= + +The Auspice client will make a number of requests to various endpoints, which this document aims to provide an overview of. Some of these are dependent on the `build-time customisations <./api>`__ in use, and we plan to allow more flexibility here using this mechanism. + +Requests to the Auspice Server +------------------------------ + +For the purposes of this section we are assuming you are running locally on "localhost:4000", however the idea is the same if Auspice is deployed to a server. + +Auspice Client JavaScript Code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The (bundled) client code (which is generated by ``auspice build``) is broken up into a number of "chunks" which are delivered to the client as needed. These originate from the following block in the ``index.html`` file: + +.. code:: html + + + +"Charon" GET requests +~~~~~~~~~~~~~~~~~~~~~ + +When a dataset, narrative, or listing of available datasets is to be displayed in Auspice, a selection of the following requests are made to localhost:4000. + +- ``/charon/getAvailable`` -- return a list of available datasets and narratives +- ``/charon/getDataset`` -- return the requested dataset +- ``/charon/getNarrative`` -- return the requested narrative + +See :doc:`the server API documentation <../server/api>` for more details about the requests, or the :doc:`client customisation API ` for the ability to change "/charon" to a different value. + +Image Requests +~~~~~~~~~~~~~~ + +The initial Auspice page (i.e. the one which displays a listing of available datasets) requests some images for the footer. + +External Requests +----------------- + +Fonts +~~~~~ + +Two initial requests for fonts are made, for `Google-hosted Lato fonts (CSS) `__ and `font-awesome (CSS) `__. These in turn make a number of subsequent requests for the individual font files etc. + +Leaflet +~~~~~~~ + +We use `Leaflet `__ to display the map tiles. There is one `initial CSS request `__ which is always made, regardless of whether a map is displayed. + +If a map is displayed by Auspice the individual tiles are requested from api.mapbox.com. Panning and zooming of the map result in futher requests for tiles. These are requested using our hardcoded access key, however this may change to a build-time-customisation in the future. + +Web Analytics (Optional) +~~~~~~~~~~~~~~~~~~~~~~~~ + +Auspice has the potential to include either Plausible Analytics or Google Analytics in a limited fashion. See :ref:`the client customisation API documentation ` for how to set this. Note that these requests are only made if you set up build-time customisation of the client. (For nextstrain.org we use `Plausible analytics `__.) diff --git a/docs/environment.yml b/docs/environment.yml index 213d71ebe..76cf405ef 100644 --- a/docs/environment.yml +++ b/docs/environment.yml @@ -11,3 +11,4 @@ dependencies: - sphinx-autobuild - sphinx-markdown-tables - sphinx-argparse + - pygments>=2.12.0 # For JSON with comments diff --git a/docs/introduction/how-to-run.md b/docs/introduction/how-to-run.md deleted file mode 100644 index 6433cdbd3..000000000 --- a/docs/introduction/how-to-run.md +++ /dev/null @@ -1,125 +0,0 @@ -# How to Run Auspice - -Auspice is run as a command line program -- `auspice` -- with various subcommands. -You can run each command with `--help` attached to see help from the command line. - -* `auspice view --help` (this is the main command for interacting with Auspice) -* `auspice build --help` -* `auspice develop --help` -* `auspice convert --help` - - -## How to Get an Example Dataset Up and Running - -In order to get up and running you'll need to have some datasets to visualise. -For the purposes of getting Auspice up and running you can download the current Zika dataset via: - -```bash -mkdir datasets -curl http://data.nextstrain.org/zika.json --compressed -o datasets/zika.json -``` - -And then run `auspice` via: -```bash -auspice view --datasetDir datasets -``` - -This will allow you to run Auspice locally (i.e. from your computer) and view the dataset which is behind [nextstrain.org/zika](https://nextstrain.org/zika). -[See below](#obtaining-a-set-of-input-files) for how to download all of the data available on [nextstrain.org](https://nextstrain.org). - - -To analyse your own data, please see the tutorials on the [nextstrain docs](https://nextstrain.org/docs/). - -## `auspice view` - -This is the main command we'll run Auspice with, as it makes Auspice available in a web browser for you. -There are two common arguments used: - -| argument name | data supplied | description | -| -------- | ---------- | -------- | -|datasetDir | PATH | Directory where datasets (JSONs) are sourced. This is ignored if you define custom handlers. | -|narrativeDir | PATH | Directory where narratives (Markdown files) are sourced. This is ignored if you define custom handlers. | - -For more complicated setups, where you define your own server handlers, see [suppling custom handlers to the Auspice server](../server/api.md#suppling-custom-handlers-to-the-auspice-server). - -## `auspice build` - -Build the client source code bundle. -This is needed in three cases: -1. You have installed Auspice from source, or updated the source code. -1. You are editing the source code and need to rebuild the client -1. You wish to build a customised version of the Auspice client. -See [Customising Auspice](../customise-client/index) for more info. - - -## `auspice develop` - -Launch Auspice in development mode. This runs a local server and uses -hot-reloading to allow automatic updating as you edit the code. - -This is only useful if you are editing the client source code! - -## `auspice convert` - -This is a utility command to convert between dataset formats. -Currently, it only converts "Auspice v1" JSONs into "Auspice v2" JSONs, using the same code that is [programatically importable](../server/api.md#convertfromv1). - -Right now, `auspice view` will automatically convert "v1" JSONs into "v2" JSONs, so there's no need to do this yourself. - - -## Input File Formats - -Auspice is agnostic about the data it visualises -- they don't have to be viral genomes, or real-time, or generated in Augur. -(They do, however have to be in a specific file format.) - -Auspice takes two different file types: -* Datasets (the tree, map, etc.), which are defined as one or more JSON files. -* Narratives, which are specified as a Markdown file. - -See the [Server API](../server/api.md) for more details about the file formats an Auspice server (e.g. `auspice view`) sends to the client. - -### Dataset JSONs - -Currently we mainly use [Augur](https://github.com/nextstrain/augur) to create these datasets. -See [the Nextstrain documentation](https://nextstrain.org/docs/bioinformatics/introduction-to-augur) for more details on this process. - -Datasets JSONs include: -* Auspice v2 JSON (required) - [schema here](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json). It includes the following information: - * `tree`: The tree structure encoded as a deeply nested JSON object. - * Traits (such as country, divergence, collection date, attributions, etc.) are stored on each node. - * The presence of a `children` property indicates that it's an internal node and contains the child objects. - * `metadata`: Additional data to control and inform the visualization. -* Frequency JSON (optional) - [example here](http://data.nextstrain.org/flu_seasonal_h3n2_ha_2y_tip-frequencies.json) - * Generates the frequencies panel, e.g. on [nextstrain.org/flu](https://nextstrain.org/flu). - -> We are working on ways to make datasets in Newick / Nexus formats available. You can see an early prototype of this at [auspice-us.herokuapp.com](https://auspice-us.herokuapp.com/) where you can drop on Newick (and CSV) files. -Using BEAST trees is possible, but you have to use Augur to convert them first. - -> If you happen to maintain builds that rely on Auspice v1, don’t worry - Auspice v2 is backward compatible and accepts the v1 format as well. The v1 schema is also available below. -See [the v2 release notes](../releases/v2.md) for details and motivation for the v2 format. - -#### Auspice v1 JSONs (for backwards compatibility use only) - -Auspice works with the v1 format for backwards compatibility. _The zika dataset we download above is in this format._ - -This format includes: -* Tree JSON (required) - [schema here](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v1-tree.json) - * The same tree information as in the v2 JSON above in a separate file whose name _must_ end with `_tree.json`. -* Metadata JSON (required) - [schema here](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v1-meta.json) - * The same metadata information as in the v2 JSON above in a separate file whose name _must_ end with `_meta.json` and have the same prefix as the tree JSON above. -* Frequency JSON (optional) - [example here](http://data.nextstrain.org/flu_seasonal_h3n2_ha_2y_tip-frequencies.json) - * Generates the frequencies panel, e.g. on [nextstrain.org/flu](https://nextstrain.org/flu). - - -### Narratives -For narratives, please see [Writing a Narrative](https://docs.nextstrain.org/en/latest/tutorials/narratives-how-to-write.html) for a description of the file format. - - -## Obtaining a Set of Input Files - -If you'd like to download the dataset JSONs which are behind the core-datasets shown on [nextstrain.org](https://nextstrain.org), then you can run [this script](https://github.com/nextstrain/auspice/blob/master/scripts/get-data.sh) which will create a `./data` directory for you. - -The nextstrain-maintained narratives are stored in the [nextstrain/narratives github repo](https://github.com/nextstrain/narratives). -You can obtain these by cloning that repo. - -You can then run `auspice view --datasetDir data --narrativeDir ` to visualise all of the [nextstrain.org](https://nextstrain.org) datasets locally. diff --git a/docs/introduction/how-to-run.rst b/docs/introduction/how-to-run.rst new file mode 100644 index 000000000..f0276493b --- /dev/null +++ b/docs/introduction/how-to-run.rst @@ -0,0 +1,132 @@ +How to Run Auspice +================== + +Auspice is run as a command line program -- ``auspice`` -- with various subcommands. You can run each command with ``--help`` attached to see help from the command line. + +- ``auspice view --help`` (this is the main command for interacting with Auspice) +- ``auspice build --help`` +- ``auspice develop --help`` +- ``auspice convert --help`` + +How to Get an Example Dataset Up and Running +-------------------------------------------- + +In order to get up and running you'll need to have some datasets to visualise. For the purposes of getting Auspice up and running you can download the current Zika dataset via: + +.. code:: bash + + mkdir datasets + curl http://data.nextstrain.org/zika.json --compressed -o datasets/zika.json + +And then run ``auspice`` via: + +.. code:: bash + + auspice view --datasetDir datasets + +This will allow you to run Auspice locally (i.e. from your computer) and view the dataset which is behind `nextstrain.org/zika `__. :ref:`See below ` for how to download all of the data available on `nextstrain.org `__. + +To analyse your own data, please see the tutorials on the `nextstrain docs `__. + +``auspice view`` +---------------- + +This is the main command we'll run Auspice with, as it makes Auspice available in a web browser for you. There are two common arguments used: + ++---------------------+--------------------------+---------------------------------------------------------------------------------------------------------+ +| argument name | data supplied | description | ++=====================+==========================+=========================================================================================================+ +| datasetDir | PATH | Directory where datasets (JSONs) are sourced. This is ignored if you define custom handlers. | ++---------------------+--------------------------+---------------------------------------------------------------------------------------------------------+ +| narrativeDir | PATH | Directory where narratives (Markdown files) are sourced. This is ignored if you define custom handlers. | ++---------------------+--------------------------+---------------------------------------------------------------------------------------------------------+ + +For more complicated setups, where you define your own server handlers, see :ref:`supplying custom handlers to the Auspice server `. + +``auspice build`` +----------------- + +Build the client source code bundle. This is needed in three cases: + +1. You have installed Auspice from source, or updated the source code. +2. You are editing the source code and need to rebuild the client +3. You wish to build a customised version of the Auspice client. See :doc:`Customising Auspice <../customise-client/overview>` for more info. + +``auspice develop`` +------------------- + +Launch Auspice in development mode. This runs a local server and uses hot-reloading to allow automatic updating as you edit the code. + +This is only useful if you are editing the client source code! + +``auspice convert`` +------------------- + +This is a utility command to convert between dataset formats. Currently, it only converts "Auspice v1" JSONs into "Auspice v2" JSONs, using the same code that is :ref:`programatically importable `. + +Right now, ``auspice view`` will automatically convert "v1" JSONs into "v2" JSONs, so there's no need to do this yourself. + +Input File Formats +------------------ + +Auspice is agnostic about the data it visualises -- they don't have to be viral genomes, or real-time, or generated in Augur. (They do, however have to be in a specific file format.) + +Auspice takes two different file types: + +* Datasets (the tree, map, etc.), which are defined as one or more JSON files. +* Narratives, which are specified as a Markdown file. + +See the :doc:`Server API <../server/api>` for more details about the file formats an Auspice server (e.g. ``auspice view``) sends to the client. + +Dataset JSONs +~~~~~~~~~~~~~ + +Currently we mainly use `Augur `__ to create these datasets. See `the Nextstrain documentation `__ for more details on this process. + +Datasets JSONs include: + +* Auspice v2 JSON (required) - `schema here `__. It includes the following information: + * ``tree``: The tree structure encoded as a deeply nested JSON object. + * Traits (such as country, divergence, collection date, attributions, etc.) are stored on each node. + * The presence of a ``children`` property indicates that it's an internal node and contains the child objects. + * ``metadata``: Additional data to control and inform the visualization. +* Frequency JSON (optional) - `example here `__ + * Generates the frequencies panel, e.g. on `nextstrain.org/flu `__. + +.. note:: + + We are working on ways to make datasets in Newick / Nexus formats available. You can see an early prototype of this at `auspice-us.herokuapp.com `__ where you can drop on Newick (and CSV) files. Using BEAST trees is possible, but you have to use Augur to convert them first. + +.. note:: + + If you happen to maintain builds that rely on Auspice v1, don't worry - Auspice v2 is backward compatible and accepts the v1 format as well. The v1 schema is also available below. See :doc:`the v2 release notes <../releases/v2>` for details and motivation for the v2 format. + +Auspice v1 JSONs (for backwards compatibility use only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Auspice works with the v1 format for backwards compatibility. *The zika dataset we download above is in this format.* + +This format includes: + +* Tree JSON (required) - `schema here `__ + * The same tree information as in the v2 JSON above in a separate file whose name *must* end with ``_tree.json``. +* Metadata JSON (required) - `schema here `__ + * The same metadata information as in the v2 JSON above in a separate file whose name *must* end with ``_meta.json`` and have the same prefix as the tree JSON above. +* Frequency JSON (optional) - `example here `__ + * Generates the frequencies panel, e.g. on `nextstrain.org/flu `__. + +Narratives +~~~~~~~~~~ + +For narratives, please see `Writing a Narrative `__ for a description of the file format. + +.. _introduction-obtaining-a-set-of-input-files: + +Obtaining a Set of Input Files +------------------------------ + +If you'd like to download the dataset JSONs which are behind the core-datasets shown on `nextstrain.org `__, then you can run `this script `__ which will create a ``./data`` directory for you. + +The nextstrain-maintained narratives are stored in the `nextstrain/narratives github repo `__. You can obtain these by cloning that repo. + +You can then run ``auspice view --datasetDir data --narrativeDir `` to visualise all of the `nextstrain.org `__ datasets locally. diff --git a/docs/introduction/install.rst b/docs/introduction/install.rst index 4c2d70d68..9d545d056 100644 --- a/docs/introduction/install.rst +++ b/docs/introduction/install.rst @@ -27,7 +27,7 @@ Install Auspice as a user npm install --global auspice -If you look at the `release notes `__ you can see the changes that have been made to Auspice (see your version of Auspice via ``auspice --version``). To upgrade, you can run +If you look at the :doc:`release notes <../releases/changelog>` you can see the changes that have been made to Auspice (see your version of Auspice via ``auspice --version``). To upgrade, you can run .. code:: bash diff --git a/docs/releases/v2.md b/docs/releases/v2.md deleted file mode 100644 index c8a1abb14..000000000 --- a/docs/releases/v2.md +++ /dev/null @@ -1,209 +0,0 @@ -# Auspice Version 2.0 - -## What's New - -* A new JSON format for datasets, which condenses metadata and tree data into a single JSON and allows for expanded functionality in v2. -* A whole host of UI updates to improve Auspice's visualisations, including pie charts, better rendering of dates, and more easily accessible metadata. - -Read on for rationale behind this update and a detailed explanation of the changes. - -![auspice-v2-gif](../assets/v2-pie-charts.gif) -*Switching between colorings in Auspice v2* - -## Backstory - -Auspice started as the [nextstrain.org](https://nextstrain.org) visualisation tool, in fact for a long time nextstrain.org _was_ Auspice. -Slowly, we turned Auspice into a stand-alone tool and started expanding its capacity. -This resulted in many v1.x releases (we made it all the way to v1.39.0), and things had a habit of changing a little too frequently, with not enough focus on documentation and communication. - -So in a way, this is the first proper Auspice release! -There are a few big and visible changes, including using pie charts to represent discrete variables on a map, but a lot of the changes were done to make things easier behind the scenes. -There are also a bunch of really cool features such as the ability to define custom servers, change the aesthetics of the Auspice client, display multiple trees and write narratives which were technically part of Auspice v1 but undocumented. -This documentation website is now up to date and should allow you to understand how to use Auspice to its full potential ([contributions are always welcome!](../contributing/overview.md)) - -In general, we hope that this documentation and the help messages of the various `auspice` subcommands are vastly improved. -The major changes which Auspice v2 brings are detailed below 👇 -If you have any comments please [email us](mailto:hello@nextstrain.org) or say hi on [twitter](https://twitter.com/nextstrain). - -## I don't want to upgrade - how can I continue to use Auspice v1? - -While we don't think there is any reason not to upgrade, version 1 will always be available via npm: -```bash -npm install --global auspice@version1 -``` - - -## Pie charts to represent discrete variables on a map - -We used to use colour blending to represent all the different traits present at a given location. -This worked pretty well for continuous traits (e.g. the dates of isolates in each country), but performed poorly for discrete traits (e.g. the different flu clades present in each country). -We now use pie charts for discrete traits which give a much nicer summary of the data: - -![pie-charts](../assets/v2-pie-charts.png) -*Notice the difference? Auspice v1 (left) & v2 (right)* - - -## New dataset JSON format - -For about a year we've been seeing the limitations of the "v1" meta + tree JSONs (the dataset files behind Auspice). -Additionally the format of the input JSONs to Auspice changed a bit throughout the lifetime of Auspice v1 - fine when it's an internal tool, but not so great when other people start to use it! - -We've settled on a new single "v2" JSON, containing pretty similar information to the v1 JSONs but in a format which allows us to expand the functionality of Auspice. - -> Don't panic - `auspice view` and [nextstrain.org](https://nextstrain.org) will automatically convert the "v1" JSONs into the new format for you so all of the old datasets will continue to work with Auspice v2. - -| file | schema | auspice versions | description | -| ---- | ---- | ---- | ---- | -|"tree" JSON | [Link](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v1-tree.json) | v1, v2 | Decorated phylogenetic tree | -|"meta" JSON | [Link](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v1-meta.json) | v1, v2 | The "metadata" associated with a phylogenetic tree | -|"v2" JSON | [Link](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json) | v2 | The single input format required for Auspice v2 | - -If you use Augur to construct your JSONs, then the next version of Augur (v6) will give you the option to produce "v1" or "v2" JSONs. -If you have specific queries, the JSON schemas (above) should help you out, but here is a list of the main changes (with links to PRs where relevent, if you're _really_ keen!): - - -#### Strings parsed unchanged -For historical reasons, we used to "prettify" certain strings in Auspice - e.g. "usa" became "USA", "new_zealand" became "New Zealand". -We dug ourselves into a bit of a hole here, as you can imagine the list of exceptions kept growing and growing. -Auspice v2 now generally parses strings as they appear in the JSON, allowing you full control over how things appear. -(For backwards compatability, we've kept the "prettifying" function in the v1-v2 conversion logic so the look of v1 JSONs shouldn't change.) - -#### Both metadata and tree data in a single JSON -The most visible change is probably the reduction of two JSONs (`_meta.json` + `_tree.json`) into a single JSON. -This single JSON has three required (and no optional) keys: -```json -{ - "version": "schema version", - "meta": {}, - "tree": {} -} -``` - - -#### Gene / Genome definitions are now in GFF format. -v1 Auspice JSONs used 0-based starts for the gene positions and `1`/`-1` for which strand the gene was on. -We now use one-based start & end coordinates, following GFF format, and a "+" or "-" to denote the strand. -These definitions are now under the `genome_annotations` property (formerly `annotations`). -See [PR #770](https://github.com/nextstrain/auspice/pull/770). - - -#### Changes to how node data is stored - -Each tree node (internal & terminal) can now contain the following properties -* `name` (required) -- formerly this was `strain` -* `node_attrs` -- attributes associated with the node (sequence, date, location) as opposed to changes from one node to another. - * Node attributes can now be objects and contain confidence information if available. - * A `hidden` node attribute can control Auspice's display of the node - * Author information is now contained under the `author` key, and the `author_info` dictionary is no longer present in the JSON. -* `branch_attrs` -- attributes associated with the branch from the parent node to this node, such as branch lengths, mutations, support values - * `branch_attrs.mutations` -- both AA & nucleotide mutations are now defined in the same object. -* `children` (unchanged) - -#### Colorings, geographic resolutions, and defaults -The `colorings` property (formerly `color_options`) is now an array of objects, the properties of which are easier to understand. -This guarantees the ordering appears in Auspice as you define it in the JSON. -See [PR #748](https://github.com/nextstrain/auspice/pull/748). - - -The `geo_resolutions` property (formerly `geo`) is similarly an array of objects. - -The `display_defaults` property (formerly `defaults`) now contains keys which are snake_case instead of camelCase. - -#### Multiple maintainers -The maintainer, displayed in the Auspice footer, was previously limited to a single string value and corresponding URL. -We now allow multiple maintainers, each with their own (optional) URL. - - -#### Continous, Categorical, Ordinal, and Boolean Color Scales -Traits with the "boolean" colour type which will use a pre-defined yellow & blue colour scale. -Currently "continous" and "categorical" scales both use the same colour scale. -Note that "discrete" types from v1 JSONs will be interpreted as "categorical". - - -## More information in tree info boxes -We've made more things available when you hover over the tree or click on a tree tip. -For instance, v1 would use the aa-nt toggle in the entropy panel to decide which mutations to display, and it was frustrating to have to scroll down to switch the toggle just to see what nucleotide mutations were on a branch! -We now show both. - - -![more-tree-info](../assets/v2-tree-info.png) -*Auspice v1 (left) & v2 (right). v2 shows more information on both tree hover (upper panel) & when clicking on tips (lower panel).* - - -## Display of second trees -Auspice has had the ability to display two trees side-by-side for a while now (and [finally documented](../advanced-functionality/second-trees.md)). -If you wanted to, say, compare influenza HA & NA trees, the URL used to look like "flu/seasonal/h3n2/ha:na/2y". -This turned out to be problematic when coming up with suitable candidates for potential second-trees, and also made it impossible to compare, for instance, "ha/2y" with "na/3y" - -We now use a more verbose syntax to define the display of multiple trees, specifying the entire pathname for both datasets. -The above example is now "flu/seasonal/h3n2/ha/2y:flu/seasonal/h3n2/na/2y". -Any available datasets can be compared using this URL syntax, even if the result is rather nonsensical. -The old syntax will continue to work and will automatically correct to the new syntax (and show you a warning). - -P.S. The list of available second trees, which is displayed in the sidebar, is now handed to Auspice by the [getAvailable API request](server/api.md#charon-getavailable). - - -## Display better dates on the tree axis -Internally, we use decimal dates (e.g. 2012.3 is around the start of may) so that's what we displayed on the tree. -It turns out this is pretty hard to interpret when looking at small timespans! -We now (a) show dates on the tree's x-axis using months & days, depending on the timespan displayed, and (b) try to use more informative grid spacings. -These help with the interpretation of trees over smaller time scales. -See [PR #804](https://github.com/nextstrain/auspice/pull/804). - -![time-labels](../assets/v2-time-labels.png) -*Above: Auspice v1's decimal labels were somewhat hard to interprete. Below: v2 displays calendar dates as appropriate, and uses more intelligent grid spacing.* - -## Map "reset zoom" button zooms to include all demes -There's now a button at the top-right of the map which will trigger the map to reset the zoom. -See [PR #802](https://github.com/nextstrain/auspice/pull/802). - - - - -## Consistent colouring of missing data in the tree -If your analysis produces results in `-` (gaps), `X` (unknown residue) or `N` (unknown nucelotide) then we now colour these grey, making it much easier to see when data is missing. -See [PR #799](https://github.com/nextstrain/auspice/pull/799). - -![base-colours](../assets/v2-base-colours.png) -*Same analysis, different colour schemes, different interpretation.* - -## Removal of Twitter & Google Analytics -These were a holdover from the early days when [nextstrain.org](https://nextstrain.org) and Auspice were the same thing. -We've now removed all calls to Twitter, and made Google Analytics opt in. -See [requests made from the client](../customise-client/requests.md) for details on exactly what requests are made and how to opt-in to Google Analytics if you desire. - - -## Improvements in the entropy panel - -We improved the usability of the entropy (genomic diversity) panel, as well as fixing a few hidden bugs -- see [PR #771](https://github.com/nextstrain/auspice/pull/771). -For instance, you can now see which codon a nucleotide codes for (and vice-versa). - -![entropy](../assets/v2-entropy.gif) - - -## Auspice responds to server redirects for datasets -This allows custom servers ([nextstrain.org](https://nextstrain.org), for instance!) to smoothly inform Auspice that, e.g., a `getDataset` request to "/flu" (which doesn't actually exist) should be "/flu/seasonal/h3n2/ha/3y". -See [PR #778](https://github.com/nextstrain/auspice/pull/778). - - -## Importing (server) code from Auspice -Auspice now makes a few helper commands available for those who are writing custom Auspice servers. -See [these docs](server/api.md#importing-code-from-auspice) for more info. - - -## New Auspice subcommand: `auspice convert` -This is a utility command to convert between dataset formats. -Currently, it only converts "Auspice v1" JSONs into "Auspice v2" JSONs, using the same code that is [programatically importable](server/api.md#convertfromv1). - -Right now, `auspice view` will automatically convert "v1" JSONs into "v2" JSONs, so there's no need to do this yourself. - - -## Ability to show a "build" source URL in the sidebar -Auspice used to contain some hard-coded logic which was used by nextstrain to display a link to the GitHub repo behind community URLs. -We have now generalised this, and the [getAvailable API request](server/api.md#charon-getavailable) can define a `buildUrl` property for each dataset which auspice will display in the sidebar. - - -## `auspice view` uses a custom Auspice client if present -It's possible to use `auspice build` to [build a custom auspice client](../customise-client/introduction.md). -If this has been done, then running `auspice view` will serve it -- before you had to run `auspice view --customBuild`. -This streamlines generating custom auspice bundles and serving them locally. diff --git a/docs/releases/v2.rst b/docs/releases/v2.rst new file mode 100644 index 000000000..0449c67ad --- /dev/null +++ b/docs/releases/v2.rst @@ -0,0 +1,188 @@ +Auspice Version 2.0 +=================== + +What's New +---------- + +- A new JSON format for datasets, which condenses metadata and tree data into a single JSON and allows for expanded functionality in v2. +- A whole host of UI updates to improve Auspice's visualisations, including pie charts, better rendering of dates, and more easily accessible metadata. + +Read on for rationale behind this update and a detailed explanation of the changes. + +|auspice-v2-gif| *Switching between colorings in Auspice v2* + +Backstory +--------- + +Auspice started as the `nextstrain.org `__ visualisation tool, in fact for a long time nextstrain.org *was* Auspice. Slowly, we turned Auspice into a stand-alone tool and started expanding its capacity. This resulted in many v1.x releases (we made it all the way to v1.39.0), and things had a habit of changing a little too frequently, with not enough focus on documentation and communication. + +So in a way, this is the first proper Auspice release! There are a few big and visible changes, including using pie charts to represent discrete variables on a map, but a lot of the changes were done to make things easier behind the scenes. There are also a bunch of really cool features such as the ability to define custom servers, change the aesthetics of the Auspice client, display multiple trees and write narratives which were technically part of Auspice v1 but undocumented. This documentation website is now up to date and should allow you to understand how to use Auspice to its full potential (`contributions are always welcome! `__) + +In general, we hope that this documentation and the help messages of the various ``auspice`` subcommands are vastly improved. The major changes which Auspice v2 brings are detailed below 👇 If you have any comments please `email us `__ or say hi on `twitter `__. + +I don't want to upgrade - how can I continue to use Auspice v1? +--------------------------------------------------------------- + +While we don't think there is any reason not to upgrade, version 1 will always be available via npm: + +.. code:: bash + + npm install --global auspice@version1 + +Pie charts to represent discrete variables on a map +--------------------------------------------------- + +We used to use colour blending to represent all the different traits present at a given location. This worked pretty well for continuous traits (e.g. the dates of isolates in each country), but performed poorly for discrete traits (e.g. the different flu clades present in each country). We now use pie charts for discrete traits which give a much nicer summary of the data: + +|pie-charts| *Notice the difference? Auspice v1 (left) & v2 (right)* + +New dataset JSON format +----------------------- + +For about a year we've been seeing the limitations of the "v1" meta + tree JSONs (the dataset files behind Auspice). Additionally the format of the input JSONs to Auspice changed a bit throughout the lifetime of Auspice v1 - fine when it's an internal tool, but not so great when other people start to use it! + +We've settled on a new single "v2" JSON, containing pretty similar information to the v1 JSONs but in a format which allows us to expand the functionality of Auspice. + + Don't panic - ``auspice view`` and `nextstrain.org `__ will automatically convert the "v1" JSONs into the new format for you so all of the old datasets will continue to work with Auspice v2. + ++-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+ +| file | schema | auspice versions | description | ++=================+==================================================================================================+==================+====================================================+ +| "tree" JSON | `Link `__ | v1, v2 | Decorated phylogenetic tree | ++-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+ +| "meta" JSON | `Link `__ | v1, v2 | The "metadata" associated with a phylogenetic tree | ++-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+ +| "v2" JSON | `Link `__ | v2 | The single input format required for Auspice v2 | ++-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+ + +If you use Augur to construct your JSONs, then the next version of Augur (v6) will give you the option to produce "v1" or "v2" JSONs. If you have specific queries, the JSON schemas (above) should help you out, but here is a list of the main changes (with links to PRs where relevent, if you're *really* keen!): + +Strings parsed unchanged +^^^^^^^^^^^^^^^^^^^^^^^^ + +For historical reasons, we used to "prettify" certain strings in Auspice - e.g. "usa" became "USA", "new_zealand" became "New Zealand". We dug ourselves into a bit of a hole here, as you can imagine the list of exceptions kept growing and growing. Auspice v2 now generally parses strings as they appear in the JSON, allowing you full control over how things appear. (For backwards compatability, we've kept the "prettifying" function in the v1-v2 conversion logic so the look of v1 JSONs shouldn't change.) + +Both metadata and tree data in a single JSON +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The most visible change is probably the reduction of two JSONs (``_meta.json`` + ``_tree.json``) into a single JSON. This single JSON has three required (and no optional) keys: + +.. code:: json + + { + "version": "schema version", + "meta": {}, + "tree": {} + } + +Gene / Genome definitions are now in GFF format. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +v1 Auspice JSONs used 0-based starts for the gene positions and ``1``/``-1`` for which strand the gene was on. We now use one-based start & end coordinates, following GFF format, and a "+" or "-" to denote the strand. These definitions are now under the ``genome_annotations`` property (formerly ``annotations``). See `PR #770 `__. + +Changes to how node data is stored +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Each tree node (internal & terminal) can now contain the following properties \* ``name`` (required) -- formerly this was ``strain`` \* ``node_attrs`` -- attributes associated with the node (sequence, date, location) as opposed to changes from one node to another. \* Node attributes can now be objects and contain confidence information if available. \* A ``hidden`` node attribute can control Auspice's display of the node \* Author information is now contained under the ``author`` key, and the ``author_info`` dictionary is no longer present in the JSON. \* ``branch_attrs`` -- attributes associated with the branch from the parent node to this node, such as branch lengths, mutations, support values \* ``branch_attrs.mutations`` -- both AA & nucleotide mutations are now defined in the same object. \* ``children`` (unchanged) + +Colorings, geographic resolutions, and defaults +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``colorings`` property (formerly ``color_options``) is now an array of objects, the properties of which are easier to understand. This guarantees the ordering appears in Auspice as you define it in the JSON. See `PR #748 `__. + +The ``geo_resolutions`` property (formerly ``geo``) is similarly an array of objects. + +The ``display_defaults`` property (formerly ``defaults``) now contains keys which are snake_case instead of camelCase. + +Multiple maintainers +^^^^^^^^^^^^^^^^^^^^ + +The maintainer, displayed in the Auspice footer, was previously limited to a single string value and corresponding URL. We now allow multiple maintainers, each with their own (optional) URL. + +Continous, Categorical, Ordinal, and Boolean Color Scales +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Traits with the "boolean" colour type which will use a pre-defined yellow & blue colour scale. Currently "continous" and "categorical" scales both use the same colour scale. Note that "discrete" types from v1 JSONs will be interpreted as "categorical". + +More information in tree info boxes +----------------------------------- + +We've made more things available when you hover over the tree or click on a tree tip. For instance, v1 would use the aa-nt toggle in the entropy panel to decide which mutations to display, and it was frustrating to have to scroll down to switch the toggle just to see what nucleotide mutations were on a branch! We now show both. + +|more-tree-info| *Auspice v1 (left) & v2 (right). v2 shows more information on both tree hover (upper panel) & when clicking on tips (lower panel).* + +Display of second trees +----------------------- + +Auspice has had the ability to display two trees side-by-side for a while now (and `finally documented <../advanced-functionality/second-trees.md>`__). If you wanted to, say, compare influenza HA & NA trees, the URL used to look like "flu/seasonal/h3n2/ha:na/2y". This turned out to be problematic when coming up with suitable candidates for potential second-trees, and also made it impossible to compare, for instance, "ha/2y" with "na/3y" + +We now use a more verbose syntax to define the display of multiple trees, specifying the entire pathname for both datasets. The above example is now "flu/seasonal/h3n2/ha/2y:flu/seasonal/h3n2/na/2y". Any available datasets can be compared using this URL syntax, even if the result is rather nonsensical. The old syntax will continue to work and will automatically correct to the new syntax (and show you a warning). + +P.S. The list of available second trees, which is displayed in the sidebar, is now handed to Auspice by the :ref:`getAvailable API request `. + +Display better dates on the tree axis +------------------------------------- + +Internally, we use decimal dates (e.g. 2012.3 is around the start of may) so that's what we displayed on the tree. It turns out this is pretty hard to interpret when looking at small timespans! We now (a) show dates on the tree's x-axis using months & days, depending on the timespan displayed, and (b) try to use more informative grid spacings. These help with the interpretation of trees over smaller time scales. See `PR #804 `__. + +|time-labels| *Above: Auspice v1's decimal labels were somewhat hard to interprete. Below: v2 displays calendar dates as appropriate, and uses more intelligent grid spacing.* + +Map "reset zoom" button zooms to include all demes +-------------------------------------------------- + +There's now a button at the top-right of the map which will trigger the map to reset the zoom. See `PR #802 `__. + +Consistent colouring of missing data in the tree +------------------------------------------------ + +If your analysis produces results in ``-`` (gaps), ``X`` (unknown residue) or ``N`` (unknown nucelotide) then we now colour these grey, making it much easier to see when data is missing. See `PR #799 `__. + +|base-colours| *Same analysis, different colour schemes, different interpretation.* + +Removal of Twitter & Google Analytics +------------------------------------- + +These were a holdover from the early days when `nextstrain.org `__ and Auspice were the same thing. We've now removed all calls to Twitter, and made Google Analytics opt in. See :doc:`requests made from the client <../customise-client/requests>` for details on exactly what requests are made and how to opt-in to Google Analytics if you desire. + +Improvements in the entropy panel +--------------------------------- + +We improved the usability of the entropy (genomic diversity) panel, as well as fixing a few hidden bugs -- see `PR #771 `__. For instance, you can now see which codon a nucleotide codes for (and vice-versa). + +.. figure:: ../assets/v2-entropy.gif + :alt: entropy + + entropy + +Auspice responds to server redirects for datasets +------------------------------------------------- + +This allows custom servers (`nextstrain.org `__, for instance!) to smoothly inform Auspice that, e.g., a ``getDataset`` request to "/flu" (which doesn't actually exist) should be "/flu/seasonal/h3n2/ha/3y". See `PR #778 `__. + +Importing (server) code from Auspice +------------------------------------ + +Auspice now makes a few helper commands available for those who are writing custom Auspice servers. See :ref:`these docs ` for more info. + +New Auspice subcommand: ``auspice convert`` +------------------------------------------- + +This is a utility command to convert between dataset formats. Currently, it only converts "Auspice v1" JSONs into "Auspice v2" JSONs, using the same code that is :ref:`programatically importable `. + +Right now, ``auspice view`` will automatically convert "v1" JSONs into "v2" JSONs, so there's no need to do this yourself. + +Ability to show a "build" source URL in the sidebar +--------------------------------------------------- + +Auspice used to contain some hard-coded logic which was used by nextstrain to display a link to the GitHub repo behind community URLs. We have now generalised this, and the :ref:`getAvailable API request ` can define a ``buildUrl`` property for each dataset which auspice will display in the sidebar. + +``auspice view`` uses a custom Auspice client if present +-------------------------------------------------------- + +It's possible to use ``auspice build`` to :doc:`build a custom auspice client <../customise-client/overview>`. If this has been done, then running ``auspice view`` will serve it -- before you had to run ``auspice view --customBuild``. This streamlines generating custom auspice bundles and serving them locally. + +.. |auspice-v2-gif| image:: ../assets/v2-pie-charts.gif +.. |pie-charts| image:: ../assets/v2-pie-charts.png +.. |more-tree-info| image:: ../assets/v2-tree-info.png +.. |time-labels| image:: ../assets/v2-time-labels.png +.. |base-colours| image:: ../assets/v2-base-colours.png diff --git a/docs/server/api.md b/docs/server/api.md deleted file mode 100644 index 01512a1e9..000000000 --- a/docs/server/api.md +++ /dev/null @@ -1,179 +0,0 @@ -# Server API - - -## Auspice client requests - -The Auspice server handles requests to 3 API endpoints made by the Auspice client: -* `/charon/getAvailable` (returns a list of available datasets and narratives) -* `/charon/getDataset` (returns the requested dataset) -* `/charon/getNarrative` (returns the requested narrative) - - - -### `/charon/getAvailable` - -**URL query arguments:** - -* `prefix` (optional) - the pathname of the requesting page in Auspice. -The `getAvailable` handler can use this to respond according appropriately. -Unused by the default Auspice handler. - -**JSON Response (on success):** - -```json -{ - "datasets": [ - { - "request": "[required] The pathname of a valid dataset. \ - Will become the prefix of the getDataset request.", - "buildUrl": "[optional] A URL to display in the sidebar representing \ - the build used to generate this analysis.", - "secondTreeOptions": "[optional] A list of requests which should \ - appear as potential second-trees in the sidebar dropdown" - }, - ... - ], - "narratives": [ - {"request": "URL of a narrative. Will become the prefix in a getNarrative request"}, - ... - ] -} -``` - -Failure to return a valid JSON will result in a warning notification shown in Auspice. - -### `/charon/getDataset` - -**URL query arguments:** - -* `prefix` (required) - the pathname of the requesting page in Auspice. Use this to determine which dataset to return. -* `type` (optional) -- if specified, then the request is for an additional file (e.g. "tip-frequencies"), not the main dataset. - -**JSON Response (on success):** - -The JSON response depends on the file-type being requested. - -If the type is not specified, i.e. we're requesting the "main" dataset JSON then [see this JSON schema](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json). -Note that the Auspice client _cannot_ process v1 (meta / tree) JSONs -- [see below](server/api.md#importing-code-from-auspice) for how to convert these. - -Alternative file type reponses are to be documented. - -**Alternative responses:** - -A `204` reponse will cause Auspice to show its splash page listing the available datasets & narratives. -Any other non-200 reponse behaves similarly but also displays a large "error" message indicating that the dataset was not valid. - - - -### `/charon/getNarrative` - -**URL query arguments:** - -* `prefix` (required) - the pathname of the requesting page in Auspice. Use this to determine which narrative to return. -* `type` (optional) - the format of the data returned (see below for more information). -Current valid values are "json" and "md". -If no type is specified the server will use "json" as a default (for backwards compatibility reasons). -Requests to this API from the Auspice client are made with `type=md`. - -**Response (on success):** - -The response depends on the `type` specified in the query. - -If a markdown format is requested, then the narrative file is sent to the client unmodified to be parsed on the client. - -If a JSON is requested then the narrative file is parsed into JSON format by the server. -For Auspice versions prior to v2.18 this was the only expected behavior. -The transformation from markdown (i.e. the narrative file itself) to JSON is via the `parseNarrativeFile()` function (see below for how this is exported from Auspice for use in other servers). -Here, roughly, is the code we use in the auspice server for this transformation: - -```js -const fileContents = fs.readFileSync(pathName, 'utf8'); -if (type === "json") { - const blocks = parseNarrative(fileContents); - res.send(JSON.stringify(blocks).replace(/ While the Auspice client (from v2.18 onwards) always requests the `type=md`, it will attempt to parse the response as JSON if markdown parsing fails, in an effort to remain backwards compatible with servers which may be using an earlier API. - ---- - -## Supplying custom handlers to the Auspice server - -The provided Auspice servers -- i.e. `auspice view` and `auspice develop` both have a `--handlers ` option which allows you to define your own handlers. -The provided JavaScript file must export three functions, each of which handles one of the GET requests described above and must respond accordingly (see above for details). - -| function name | arguments | API endpoint | -| ---------- | --------- | ---------- | -| getAvailable | req, res | /charon/getAvailable | -| getDataset | req, res | /charon/getDataset | -| getNarrative | req, res | /charon/getNarrative | - -For information about the `req` and `res` arguments see the express documentation for the [request object](https://expressjs.com/en/api.html#req) and [response object](https://expressjs.com/en/api.html#res), respectively. - -You can see [nextstrain.org](https://nextstrain.org)'s implementation of these handlers [here](https://github.com/nextstrain/nextstrain.org/blob/master/server.js). - -Here's a pseudocode example of an implementation for the `getAvailable` handler which may help understanding: - -```js -const getAvailable = (req, res) => { - try { - /* collect available data */ - res.json(data); - } catch (err) { - const errorMessage = `error message to display in client`; - console.log(errorMessage); /* printed by the server, not the client */ - return res.status(500).type("text/plain").send(errorMessage); - } -}; -``` - ---- - -## Importing code from Auspice - -The servers included in Auspice contain lots of useful code which you may want to use to either write your own handlers or entire servers. -For instance, the code to convert v1 dataset JSONs to v2 JSONs (which the client requires) can be imported into your code so you don't have to reinvent the wheel! - - -Currently -```js -const auspice = require("auspice"); -``` -returns an object with two properties: - -### `convertFromV1` - -**Signature:** -```js -const v2json = convertFromV1({tree, meta}) -``` -where `tree` is the v1 tree JSON, and `meta` the v1 meta JSON. - -**Returns:** - -An object representing the v2 JSON [defined by this schema](https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json). - - - -### `parseNarrativeFile` - -> This function is deprecated as of vXXX. -You can now send the untransformed contents of the narrative file (markdown) for client-side parsing. -See [above](#charon-getnarrative) for more details. - -**Signature:** - -```js -const blocks = parseNarrativeFile(fileContents); -``` -where `fileContents` is a string representation of the narrative Markdown file. - -**Returns:** - -An array of objects, each entry representing a different narrative "block" or "page". -Each object has properties -* `__html` -- the HTML to render in the sidebar to form the narrative -* `dataset` -- the dataset associated with this block -* `query` -- the query associated with this block - diff --git a/docs/server/api.rst b/docs/server/api.rst new file mode 100644 index 000000000..3df8e7237 --- /dev/null +++ b/docs/server/api.rst @@ -0,0 +1,187 @@ +Server API +========== + +Auspice client requests +----------------------- + +The Auspice server handles requests to 3 API endpoints made by the Auspice client: + +* ``/charon/getAvailable`` (returns a list of available datasets and narratives) +* ``/charon/getDataset`` (returns the requested dataset) +* ``/charon/getNarrative`` (returns the requested narrative) + +.. _server-api-charon-getavailable: + +``/charon/getAvailable`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +**URL query arguments:** + +- ``prefix`` (optional) - the pathname of the requesting page in Auspice. The ``getAvailable`` handler can use this to respond according appropriately. Unused by the default Auspice handler. + +**JSON Response (on success):** + +.. code:: json + + { + "datasets": [ + { + "request": "[required] The pathname of a valid dataset. \ + Will become the prefix of the getDataset request.", + "buildUrl": "[optional] A URL to display in the sidebar representing \ + the build used to generate this analysis.", + "secondTreeOptions": "[optional] A list of requests which should \ + appear as potential second-trees in the sidebar dropdown" + }, + // ... + ], + "narratives": [ + {"request": "URL of a narrative. Will become the prefix in a getNarrative request"}, + // ... + ] + } + +Failure to return a valid JSON will result in a warning notification shown in Auspice. + +``/charon/getDataset`` +~~~~~~~~~~~~~~~~~~~~~~ + +**URL query arguments:** + +- ``prefix`` (required) - the pathname of the requesting page in Auspice. Use this to determine which dataset to return. +- ``type`` (optional) -- if specified, then the request is for an additional file (e.g. "tip-frequencies"), not the main dataset. + +**JSON Response (on success):** + +The JSON response depends on the file-type being requested. + +If the type is not specified, i.e. we're requesting the "main" dataset JSON then `see this JSON schema `__. Note that the Auspice client *cannot* process v1 (meta / tree) JSONs -- :ref:`see below ` for how to convert these. + +Alternative file type reponses are to be documented. + +**Alternative responses:** + +A ``204`` reponse will cause Auspice to show its splash page listing the available datasets & narratives. Any other non-200 reponse behaves similarly but also displays a large "error" message indicating that the dataset was not valid. + +.. _server-api-charon-getnarrative: + +``/charon/getNarrative`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +**URL query arguments:** + +- ``prefix`` (required) - the pathname of the requesting page in Auspice. Use this to determine which narrative to return. +- ``type`` (optional) - the format of the data returned (see below for more information). Current valid values are "json" and "md". If no type is specified the server will use "json" as a default (for backwards compatibility reasons). Requests to this API from the Auspice client are made with ``type=md``. + +**Response (on success):** + +The response depends on the ``type`` specified in the query. + +If a markdown format is requested, then the narrative file is sent to the client unmodified to be parsed on the client. + +If a JSON is requested then the narrative file is parsed into JSON format by the server. For Auspice versions prior to v2.18 this was the only expected behavior. The transformation from markdown (i.e. the narrative file itself) to JSON is via the ``parseNarrativeFile()`` function (see below for how this is exported from Auspice for use in other servers). Here, roughly, is the code we use in the auspice server for this transformation: + +.. code:: js + + const fileContents = fs.readFileSync(pathName, 'utf8'); + if (type === "json") { + const blocks = parseNarrative(fileContents); + res.send(JSON.stringify(blocks).replace(/`` option which allows you to define your own handlers. The provided JavaScript file must export three functions, each of which handles one of the GET requests described above and must respond accordingly (see above for details). + +============= ========= ==================== +function name arguments API endpoint +============= ========= ==================== +getAvailable req, res /charon/getAvailable +getDataset req, res /charon/getDataset +getNarrative req, res /charon/getNarrative +============= ========= ==================== + +For information about the ``req`` and ``res`` arguments see the express documentation for the `request object `__ and `response object `__, respectively. + +You can see `nextstrain.org `__'s implementation of these handlers `here `__. + +Here's a pseudocode example of an implementation for the ``getAvailable`` handler which may help understanding: + +.. code:: js + + const getAvailable = (req, res) => { + try { + /* collect available data */ + res.json(data); + } catch (err) { + const errorMessage = `error message to display in client`; + console.log(errorMessage); /* printed by the server, not the client */ + return res.status(500).type("text/plain").send(errorMessage); + } + }; + +-------------- + +.. _server-api-importing-code-from-auspice: + +Importing code from Auspice +--------------------------- + +The servers included in Auspice contain lots of useful code which you may want to use to either write your own handlers or entire servers. For instance, the code to convert v1 dataset JSONs to v2 JSONs (which the client requires) can be imported into your code so you don't have to reinvent the wheel! + +Currently + +.. code:: js + + const auspice = require("auspice"); + +returns an object with two properties: + +.. _server-api-convertfromv1: + +``convertFromV1`` +~~~~~~~~~~~~~~~~~ + +**Signature:** + +.. code:: js + + const v2json = convertFromV1({tree, meta}) + +where ``tree`` is the v1 tree JSON, and ``meta`` the v1 meta JSON. + +**Returns:** + +An object representing the v2 JSON `defined by this schema `__. + +``parseNarrativeFile`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. warning:: + + This function is deprecated as of vXXX. You can now send the untransformed contents of the narrative file (markdown) for client-side parsing. See :ref:`above ` for more details. + +**Signature:** + +.. code:: js + + const blocks = parseNarrativeFile(fileContents); + +where ``fileContents`` is a string representation of the narrative Markdown file. + +**Returns:** + +An array of objects, each entry representing a different narrative "block" or "page". Each object has properties + +* ``__html`` -- the HTML to render in the sidebar to form the narrative +* ``dataset`` -- the dataset associated with this block +* ``query`` -- the query associated with this block diff --git a/docs/server/authentication.md b/docs/server/authentication.md deleted file mode 100644 index 8f2cb3f1b..000000000 --- a/docs/server/authentication.md +++ /dev/null @@ -1,27 +0,0 @@ -# Authentication - -While Auspice was designed to facilitate open data sharing and rapid dissemination of results, it may be necessary to authenticate certain datasets (or indeed the entire instance). -Auspice itself contains no authentication ability, but if you are running a server then it is possible -- and relatively simple -- to build in authentication. - -The exact implementation details are beyond the scope of this documentation, but the following provides an outline of the approach we have implemented for [nextstrain.org](https://nextstrain.org). - - -## Using the Server to Verify Cookies - -![auth-cartoon](../assets/authentication.svg) - - -The server can examine cookies sent with each API request (or, the request for `index.html`) to verify the status of a user. -This allows the server to examine the cookie and: -* Deliver different available datasets depending on the cookie -* Accept or reject specific dataset requests depending on the cookie -* Redirect requests to a custom authentication page (referred to as `login.html`) - * If this is from a request for `index.html` (i.e. you want to secure the entire site), then the redirect is simple - * For redirects from an API request, you may have to respond with a 302 or 303 redirect header. - - -It is this custom authentication page which can process a login and set a cookie appropriately. -As Auspice is served from the same domain, the cookie should remain with all requests. - - -Implementing authentication is beyond the scope of this documentation, but we can recommend [PassportJS](http://www.passportjs.org) and [Auth0](https://auth0.com/), the latter of which allows you to easily use single sign-on strategies. diff --git a/docs/server/authentication.rst b/docs/server/authentication.rst new file mode 100644 index 000000000..a667b4e8c --- /dev/null +++ b/docs/server/authentication.rst @@ -0,0 +1,26 @@ +Authentication +============== + +While Auspice was designed to facilitate open data sharing and rapid dissemination of results, it may be necessary to authenticate certain datasets (or indeed the entire instance). Auspice itself contains no authentication ability, but if you are running a server then it is possible -- and relatively simple -- to build in authentication. + +The exact implementation details are beyond the scope of this documentation, but the following provides an outline of the approach we have implemented for `nextstrain.org `__. + +Using the Server to Verify Cookies +---------------------------------- + +.. figure:: ../assets/authentication.svg + :alt: auth-cartoon + + auth-cartoon + +The server can examine cookies sent with each API request (or, the request for ``index.html``) to verify the status of a user. This allows the server to examine the cookie and: + +* Deliver different available datasets depending on the cookie +* Accept or reject specific dataset requests depending on the cookie +* Redirect requests to a custom authentication page (referred to as ``login.html``) + * If this is from a request for ``index.html`` (i.e. you want to secure the entire site), then the redirect is simple + * For redirects from an API request, you may have to respond with a 302 or 303 redirect header. + +It is this custom authentication page which can process a login and set a cookie appropriately. As Auspice is served from the same domain, the cookie should remain with all requests. + +Implementing authentication is beyond the scope of this documentation, but we can recommend `PassportJS `__ and `Auth0 `__, the latter of which allows you to easily use single sign-on strategies. diff --git a/docs/server/overview.rst b/docs/server/overview.rst index 6f6f4c313..9725977cf 100644 --- a/docs/server/overview.rst +++ b/docs/server/overview.rst @@ -45,7 +45,7 @@ Customising the Default Auspice Server ====================================== You can customise the default Auspice server by supplying your own handlers for each of the three GET requests. -See `the API documentation <../server/api.html#suppling-custom-handlers-to-the-auspice-server>`_ for how to define these and provide them to `auspice view`. +See `the API documentation <../server/api.html#supplying-custom-handlers-to-the-auspice-server>`_ for how to define these and provide them to `auspice view`. AGPL Source Code Requirement diff --git a/releaseNewVersion.sh b/releaseNewVersion.sh index 2ae493088..0fd76e9d8 100755 --- a/releaseNewVersion.sh +++ b/releaseNewVersion.sh @@ -75,9 +75,9 @@ perl -pi -e "s/\"version\": \"${packagesVersion}\"/\"version\": \"${newVersion}\ perl -pi -e "s/version = \"${srcVersion}\";/version = \"${newVersion}\";/" src/version.js unset packagesVersion srcVersion parts bumps yn -# step 3: add h2 title to CHANGELOG.md with newVersion & date, while preserving YAML frontmatter for docs +# step 3: add h2 title to CHANGELOG.md with newVersion & date, while preserving the h1 title today=$(date +'%Y/%m/%d') -echo -e "---\ntitle: Changelog\n---\n\n## version ${newVersion} - ${today}\n\n$(tail -n +4 CHANGELOG.md)" > CHANGELOG.md +echo -e "# Changelog\n\n## version ${newVersion} - ${today}\n\n$(tail -n +2 CHANGELOG.md)" > CHANGELOG.md unset today # step 4: commit to current branch (master) & push to github (origin)