Support non-null DRS URIs in file descriptors

[theathorn]

Needed for LungMAP managed access data files stored in BDCat.
Pre-requisite is a metadata schema change to support DRS URIs - to be proposed by Andrew Herbst (Broad).

@hannes-ucsc:

Support for null DRS URIs in file descriptors ("phantom files") was added in #3956. For files with a non-null DRS URIs in the file descriptor, the /repository/files endpoint and its /fetch/repository/files cousin should return a 501 Not Implemented status code.

The /index/files response should have hits.*.files.*.url set to null for these files (just as for phantom files). Any manifest columns (fields) that would normally refer to a /repository/files or /fetch/repository/files endpoint must also be empty (null). We should add a hits.*.files.*.drs_uri to /index/files responses. All of the above for /index/projects and hits.*.projects.*.contributedAnalyses/contributorMatrices as well.

I think we should implement this by renaming the drs_path index field to drs_uri. We can pretend that DRS URI paths are still DRS URIs, just relative ones. A relative DRS URI is converted to an absolute one by the service, just as before, while absolute DRS URIs are passed through."

[melainalegaspi]

@hannes-ucsc to move the PR along.

[hannes-ucsc]

Updated list of reviewers on HumanCellAtlas/dcp2#55 and notified straggler on Slack.

[theathorn]

@hannes-ucsc to merge PR on 10/20/22 unless Amnon request changes by then.

[hannes-ucsc]

HumanCellAtlas/dcp2#55 was merged.

[theathorn]

Follow up with CCHMC for timeline for new LungMAP snapshots with these DRS URIs.

[bvizzier-ucsc]

LungMAP is working on a new release with updates to the existing datasets.

[hannes-ucsc]

Apparently, lm4 now uses non-null DRS URIs in file descriptors but we weren't informed and didn't implement this ahead of time. We had to back out the addition of lm4 which was slated to be promoted to prod today.

[hannes-ucsc]

Commit 58556e0 for #5824 disables the null requirement and converts it into a warning. Before we can implement this, we need schema PR HumanCellAtlas/metadata-schema#1437 to be approved and merged.

[hannes-ucsc]

For an example of what one of these external DRS URIs looks like see #5769 (comment)

[hannes-ucsc]

Assignee to work with LungMap team to restart progress on the schema PR.

[bvizzier-ucsc]

This should probably be moved to backlog.
At this time, there are no plans to allow for data download via the Data Explorer of the data located in BDC.

[achave11-ucsc]

Assignee to consider next steps.

[hannes-ucsc]

The workaround (58556e0 causes these external DRS URIs to be ignored by Azul and omitted from the index. There really isn't a point of putting them into the metadata if Azul simply ignores them. These DRS URIs are compact, e.g. drs://dg.4503:dg.4503/6282d0a2-732a-4949-a35d-e822581a705e. The prefix dg.4503 appears not to be registered in identifiers.org.

In order to determine how much work we put into this, it would be good to know, what the expected user journey is for these files, according to the LungMAP project leadership. It's fine to not support direct downloads for them, but surely we should at least include their DRS URIs in the manifest.

[achave11-ucsc]

Assignee to discuss with LungMAP leadership.