Index dcp34 in prod

[bvizzier-ucsc]

All projects with a "1' in the "dcp34" column of the HCA Prod tab in DCP/2 Datasets and snapshots.

Approvals are in this Slack thread.

dcp34 has 429 428 (see here) 427 (see here) snapshots total, with 33 updated, 16 new snapshots, and 0 re-released.

There are no soft deletes in this release.

• Security design review completed; the Resolution of this issue does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[achave11-ucsc]

Note that spreadsheet was updated to reflect DataBiosphere/azul-private#124.
The actual, total snapshot count is 428.

[bvizzier-ucsc]

If dcp34 is ready for wrangler review before the winter break, please update the Slack thread linked in the description. Let them know that it is ready for wrangler review and the URL to access it for review.

Please tag the following individuals in that message: @Oihane @Enrique Sapena Ventura @Rachel Schwartz @gabs @idazucchi @arsenios @Anu Shivalikanjli @Wei @parisanejad @WilliamGordanSullivan

[hannes-ucsc]

https://humancellatlas.slack.com/archives/C9XD6L0AD/p1703264879581679?thread_ts=1701288765.681329&cid=C9XD6L0AD

Indexing of dcp34 will complete in approximately 16h after I write this. We've already detected at least one issue with at least eight snapshots. They contain the disallowed string || in metadata:

• hca_prod_a2a2f324cf24409ea859deaee871269c__20220330_dcp2_20231213_dcp34 (project a2a2f324-cf24-409e-a859-deaee871269c)
• hca_prod_41fb1734a121461695c73b732c9433c7__20220113_dcp2_20231213_dcp34 (project 41fb1734-a121-4616-95c7-3b732c9433c7)
• hca_prod_9a23ac2d93dd4bac9bb8040e6426db9d__20220906_dcp2_20231213_dcp34 (project 9a23ac2d-93dd-4bac-9bb8-040e6426db9d)
• hca_prod_aecfd908674c4d4eb36e0c1ceab02245__20231101_dcp2_20231213_dcp34 (project aecfd908-674c-4d4e-b36e-0c1ceab02245)
• hca_prod_c302fe54d22d451fa130e24df3d6afca__20220606_dcp2_20231213_dcp34 (project c302fe54-d22d-451f-a130-e24df3d6afca)
• hca_prod_77780d5603c0481faade2038490cef9f__20220330_dcp2_20231213_dcp34 (project 77780d56-03c0-481f-aade-2038490cef9f)
• hca_prod_e255b1c611434fa683a8528f15b41038__20220330_dcp2_20231213_dcp34 (project e255b1c6-1143-4fa6-83a8-528f15b41038)
• hca_prod_aff9c3cd6b844fc2abf2b9c0b3038277__20220330_dcp2_20231213_dcp34 (project aff9c3cd-6b84-4fc2-abf2-b9c0b3038277)

https://humancellatlas.slack.com/archives/C9XD6L0AD/p1703358952358879?thread_ts=1701288765.681329&cid=C9XD6L0AD

Indexing of dcp34 is done. The release is ready for review. The list of problematic snapshots I posted above is exhaustive. The metadata for eight projects contain the disallowed string ||.

[hannes-ucsc]

I will push a commit that removes these eight snapshots.

[hannes-ucsc]

Snapshots are being removed. Since seven of the defective snapshots were updates and only one was a new project, there are now 427 snapshots in the release. The spreadsheet was updated.