Auspice consumes v2 JSONs #720

jameshadfield · 2019-04-22T19:07:14Z

Tl;DR - It’d be fantastic if you could test this version of auspice if you have any augur-produced v2 JSONs 🎉

npm install --global auspice@alpha
auspice view --datasetDir <dir with jsons>

Any v2 JSON (without a _tree.json or _meta.json) should be picked up (it should also work fine for v1 JSONs). If you need an example JSON:

curl https://raw.githubusercontent.com/nextstrain/auspice/json-v2/examples/minimal.json > minimal.json

Some background

The “v1” JSONs (current schemas here and here) have changed over time and there has never been any guarantees that a JSON from one time would work with a later/earlier version of auspice (in fact this has been the source of many headaches!).

Furthermore, about a year ago the auspice (and nextstrain.org) server began transforming these JSONs before sending to the client. Thus the data that auspice (the client) received was a different format altogether, somewhere half way between schema v1 & v2.

The v2 schema contains a number of improvements over the previous schema (and probably most noticeably, it’s a single JSON instead of two). Since there are no v1 features not in v2, it’s possible to transform v1 JSONs into v2 JSONs (see below).

Furthermore, as we want to expand the capabilities of auspice to build “static” sites (e.g. GitHub pages), we lose the ability to rely on the server to transform data. This PR is a huge step in that direction, as now v2 JSONs can be handed directly to the client with no modifications.

This PR shouldn’t result in any aesthetic changes.

Server changes

The server must now respond with v2 JSONs. There is a conversion function which the auspice server (“charon”) uses to convert metadata + tree JSONs into a v2 JSON. For release, this function (and others) will be made available so that other servers (e.g. nextstain.org) can use it along the lines of:

const convertJsons = require(“auspice”).convert;

In terms of the API requests there are very few changes, and they all revolve around requests for two trees. Previously a server was required to handle dataset requests with the ?type=tree query as well as ?prefix=".../tree1:tree2/... encoding. Both of these are no longer required, instead auspice makes two independent requests and expects back two (v2) JSONs. The requests for the (v2) JSONs, as well as tip-frequencies and all other requests remain unchanged. The one modification a server can make — and it’s completely optional — is to set a property which instructs auspice to display a different URL.

Producing v2 JSONs

Augur has been producing v2 JSONs for quite some time, but these were (as far as I know) a dead end with no use. It’s therefore to be expected that they may be in short supply for testing!
Don’t forget that the auspice server — and shortly the nextstrain.org server — will convert v1 JSONs to v2, so v1 JSONs will still work without issues.

A minimal example may be found here
Auspice can now convert v1 -> v2 using the same conversion function that the server uses to deliver v1 JSONs to auspice:

auspice convert --meta-json <json> --tree-json <json> --output <json>

Proposed v2 schema changes

During development it became clear that we’re missing a few fields. Thus, the v2 JSONs here don’t exactly match the current v2 schema. See this augur PR for more info

To-do

I anticipate there will be bugs (which is why it’s an alpha release!)
Expose (server) helper functions, such as convertFromV1. This should make building custom servers much easier.
Documentation
nextstrain.org PR using this version
augur PR with proposed schema changes
Test on IE / Edge @emmahodcroft 🙏
Reinstate the ability to drag & drop a CSV containing new metadata (colorings) for the tips.
Fully test static sites / different customizations etc
Remove the logging from the server / cli (but which is useful during alpha testing).

Main changes are a result of author names no longer being the keys of `author_info` or the value of `node.attr.authors` (this caused problems because they weren't unique)

`virus_count` is not part of the v2 schema

Part of the move from v1 to v2 JSONs. Note that now the metadata json is no longer sent to the client.

There are a lot of changes in this commit, simply as a result of moving `node.attr.authors` -> `node.authors` (v1 -> v2 JSON). The reason is twofold: (a) the author key is no longer the author string to display, and it needs to be looked up via `authorInfo` (b) Code previously relied on things being set on `node.attr[name]` and this commit begins the process of looking for these keys on both the node (`node[name]`) and in the (not yet present) traits dict (`node.traits[name]`).

Filters no longer expect values to be found on `node.attr[trait]`

The vaccine strains are now displayed in the hover/click info boxes. All properties of the v2 schema are displayed (`selection_date`, `start_date`, `end_date`, `serum`). This has the benefit that serum strains are now displayed (these had been set in v1 tree JSONs but never rendered in auspice)

v1 JSONs didn't define these -- "clade" and "aa" were set by auspice. This shifts the logic out of auspice and to the JSON creation, which is more appropriate.

The biggest change was the removal of `node.clade`. We used this extensively for DOM IDs but we now use `node.strain` (note that we have to remove certain characters from the strain name which are invalid as DOM ids)

NOTE: This commit removes the (somewhat unused) functionality of dropping a CSV onto the browser with color definitions. TO DO.

This is currently missing from the v2 schema but must be there

Previously it was finding v1 meta + tree JSONs and converting them. Note - tangle trees are untested.

Use async/await in auspice to make things much clearer (to do: test on IE).

A (completely optional) JSON field `auspice_url_should_be` will instruct the client to change the URL. This is not part of the JSON schema, but can be used by servers to mimic redirects. Note that I first implemented this using a custom HTTP header but there are issues with it being cached, so this option is both simpler and cleaner. This commit also fixes some issues where adding a 2nd tree would sometimes not show up in the URL.

Once published (with an alpha tag), this should allow installation via `npm install --global auspice@alpha`

Reinstate important but somewhat undocumented functionality allowing one to drag & drop a CSV (or TSV) defining additional / private traits for strains and having these appear as additional color-bys. Provided an example CSV file for the "minimal" dataset.

rneher · 2019-05-11T17:17:28Z

some more general thoughts: Our author field is very similar to bibtex. should we try to align completely? there are node bibtex parsers....

rneher · 2019-05-11T17:22:37Z

similarly, genome annotations have accepted formats such as GFF3:
https://www.ensembl.org/info/website/upload/gff3.html

emmahodcroft · 2019-05-13T16:12:26Z

npm install --global auspice@alpha

@jameshadfield Unfortunately this doesn't work for me... Running with sudo (otherwise I immediately get errors) about access, I get errors about 'gifsicle'. I can copy them to you somewhere?

I don't usually install with --global - unsure if that's part of the problem. But it won't let me install without that, as it complains I'm stalling a package a package under a package of the same name...

emmahodcroft · 2019-05-13T16:27:51Z

My apologies! After more Googling, this seems to work for me... sudo npm install --global auspice@alpha --unsafe-perm=true --allow-root

jameshadfield · 2019-06-17T22:08:42Z

This is now in the v2 branch. Smaller PRs will follow with fixes as we approach the v2 release :)

jameshadfield requested review from tsibley, trvrb and emmahodcroft April 22, 2019 19:07

jameshadfield mentioned this pull request Apr 23, 2019

how to develop v2 #721

Closed

jameshadfield added 26 commits April 30, 2019 15:05

set up convert util

75412e9

use v2.0 schema color definitions

6c5c418

convert title, version, updated & maintainers to v2 json format

a6a5387

update author_info to v2

db13b3c

Main changes are a result of author names no longer being the keys of `author_info` or the value of `node.attr.authors` (this caused problems because they weren't unique)

Update [genome-]annotations to v2

09cb203

move filters, panels and [display_]defaults to v2 schema

78e96c3

add geographic_info JSON v2 key

5660fc3

remove dependence on virus_count

25e0388

`virus_count` is not part of the v2 schema

remove vaccine_choices and set vaccine info on tree nodes

4afe9c5

Part of the move from v1 to v2 JSONs. Note that now the metadata json is no longer sent to the client.

move url & accession to v2 schema

5dc5853

filters now work with v2 schema

342db45

Filters no longer expect values to be found on `node.attr[trait]`

move branch labels to v2 spec

2770a96

v1 JSONs didn't define these -- "clade" and "aa" were set by auspice. This shifts the logic out of auspice and to the JSON creation, which is more appropriate.

move num_date to v2 spec

c61999b

move div to v2 spec

552ba1a

move muts and aa_muts to mutations (v2 schema)

9d6d6a4

Remove all non-v2 node properties (except attr)

2c81300

The biggest change was the removal of `node.clade`. We used this extensively for DOM IDs but we now use `node.strain` (note that we have to remove certain characters from the strain name which are invalid as DOM ids)

move attr dict to traits (v2 schema)

06ad734

NOTE: This commit removes the (somewhat unused) functionality of dropping a CSV onto the browser with color definitions. TO DO.

create tree_name property

cbdb3d7

This is currently missing from the v2 schema but must be there

add v2 minimal example

1807f1c

server can deliver v2 JSONs

5b91b68

Previously it was finding v1 meta + tree JSONs and converting them. Note - tangle trees are untested.

reinstate second tree functionality

4dfd285

Use async/await in auspice to make things much clearer (to do: test on IE).

Add auspice convert to allow cli v1 -> v2 JSON conversion

6838b78

bump version to 2.0.0-alpha.0

5aa0f79

Once published (with an alpha tag), this should allow installation via `npm install --global auspice@alpha`

jameshadfield force-pushed the json-v2 branch from c05225c to a8b0a71 Compare April 30, 2019 09:43

jameshadfield force-pushed the json-v2 branch from a8b0a71 to ba9f159 Compare April 30, 2019 15:28

jameshadfield mentioned this pull request May 16, 2019

Test npm install --global on different setups #724

Closed

jameshadfield closed this Jun 17, 2019

jameshadfield mentioned this pull request Jun 17, 2019

fix & test augur export for v2 (unified) JSONs nextstrain/augur#297

Closed

jameshadfield deleted the json-v2 branch July 29, 2019 04:13

jameshadfield mentioned this pull request Jul 29, 2019

plan to get rid of magics from data JSON #753

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auspice consumes v2 JSONs #720

Auspice consumes v2 JSONs #720

jameshadfield commented Apr 22, 2019 •

edited

Loading

rneher commented May 11, 2019

rneher commented May 11, 2019

emmahodcroft commented May 13, 2019

emmahodcroft commented May 13, 2019

jameshadfield commented Jun 17, 2019

Auspice consumes v2 JSONs #720

Auspice consumes v2 JSONs #720

Conversation

jameshadfield commented Apr 22, 2019 • edited Loading

Some background

Server changes

Producing v2 JSONs

Proposed v2 schema changes

To-do

rneher commented May 11, 2019

rneher commented May 11, 2019

emmahodcroft commented May 13, 2019

emmahodcroft commented May 13, 2019

jameshadfield commented Jun 17, 2019

jameshadfield commented Apr 22, 2019 •

edited

Loading