GA4GH Cloud Interoperability Testing

The GA4GH Testbed Orchestrator is a system that brings together plugins that test implementations of services from the GA4GH Cloud (and eventually other) Work Stream. The orchestrator is designed to be a framework for running multiple tests within, and across services. For example, (1) the interoperability and integration tests across Workflow Execution Service (WES), Tool Registry Service (TRS), and Data Repository Service (DRS) APIs and also (2) specific compliance tests for implementations of individual APIs. By building the test infrastructure with a common Testbed Orchestrator, we can evolve how we test in the future while still leveraging a unified framework. This approach will not only make it much easier to aggregate results to a common GA4GH testing dashboard, but it will also reduce redundant code between testing efforts by factoring out orchestration to this effort.

The initial use case for this app will be to act as a workflow orchestrator and bridge between Tool Registry Service (TRS) and Workflow Execution Service (WES) endpoints for the Testbed Interoperability Platform, a core deliverable of the GA4GH Cloud Workstream for 2018. We're basing the design on the GA4GH Testbed Orchestrator Design document.

Note: this repo was originally forked from Sage-Bionetworks/workflow-interop and will be maintained here as the testbed scope expands.

Overview

This is an OpenAPI-enabled (and documented) Flask server. This app uses the Connexion library on top of Flask.

The (intended) logic for the testbed is further described in the in-progress OpenAPI specification.

Requirements

Python 3.6+

Usage

To install the server and dependencies, execute the following from the root directory:

pip install -r requirements.txt . && \
  pip install --no-deps -r dev-requirements.txt

To start the app, you can use this command:

testbed_server

and open your browser to here:

http://localhost:8080/ga4gh/testbed/v1/ui/

The OpenAPI definition lives here:

http://localhost:8080/ga4gh/testbed/openapi.json

You can also find the latest version of the OpenAPI specification in GitHub here.

Running with Docker

To run the server on a Docker container, please execute the following from the root directory:

# building the image
docker build -t ga4ghtest .

# starting up a container
docker run -p 8080:8080 ga4ghtest

Development

After editing the OpenAPI YAML spec, controller and model code can be regenerated with the following command:

bash codegen.sh

See instructions for creating new test plugins here.

Notes

When comparing the OpenAPI spec for the different services TRS, WES, TES and DRS the following inconsistencies were observed. They should be addressed to make the APIs specs uniform and ensure that implementers don't misinterpret the spec.

Base Path

The basePath is configured in the openAPi spec. This is the path used in the URL of the endpoint.

	WES	TRS	TES	DRS
BasePath	/ga4gh/wes/v1	/api/ga4gh/v1	(None)	/ga4gh/drs/v1
Implied BasePath (based on common prefix)	/ga4gh/wes/v1	/api/ga4gh/v1	/v1	/ga4gh/drs/v1

As you can see in the above table, they are different based on the service.

URL Conflict

The following URLs have the same resource-name, but are served from different path based on the service name. TRS uses /metadata, which is meant to provide similar information as /service-info

	WES	TRS	TES	DRS
BasePath	/ga4gh/wes/v1	/api/ga4gh/v1	/v1/tasks	/ga4gh/drs/v1
path	/service-info	/metadata	/service-info	/service-info

These URLs should be standardized across the services and potentially merged and made hostname specific instead of service specific.

Content-Type header misuse

Some services for example DRS, TRS and WES require the content-type header by default. Because of this any GET request also requires the content-type header even though there is no payload/body.

When using swagger-ui to send the GET request, it omits the content-type header. And this can result in HTTP 415 Unsupported Media Type errors. The swagger generated server must be modified manually to make the content-type header optional. It’s not typical to require the content-type in these cases and should be removed from the openApi spec.

Tags

The tags specified in the openApi spec are inconsistent.

	WES	TRS	TES	DRS
tags	WorkflowExecutionService	GA4GH	TaskService	DataRepositoryService

This is a minor and mostly cosmetic change. But will make the Swagger UI more usable out of the box.