ga4gh-node

Application for test GA4GH deployments. Current contains an example implementation of Data Connect and (under development) Data Repository Service servers. Client implementations aren't included at this time.

Usage Notes

Note that before starting any services with docker compose, you will need to create a ./secrets/ folder in the project root directory and propagate three files:

• POSTGRESROOT_PWD: Postgres password for the root "postgres" user.
• DC_POSTGRES_PWD: Postgres password for the Data Connect postgres user.
• DRS_POSTGRES_PWD: Postgres password for the DRS postgres user. For example, ./secrets/POSTGRESROOT_PWD will contain only the text for the desired postgres root password. These are the files that will be used by Docker Compose upon startup.

The .env.infra file in the root directory of the project contains default infrastructure configuration:

• DB_TYPE=postgresql # SQL server type (sqlalchemy address string)
• DB_HOST=postgres # SQL server hostname -- from docker-compose service name
• DB_PORT=5432 # SQL server port
• DC_POSTGRES_DB=data_connect
• DC_POSTGRES_USER=dc_user
• DRS_POSTGRES_DB=drs
• DRS_POSTGRES_USER=drs_user

Note that .env.test is used by the unit tests for all services. Do not set your ./secrets files to be the same as the passwords in .env.tests! Configuration parameters for individual services can be found in ./[SERVICE_NAME]/config.yaml.

The services are currently implemented as API containers that connect to backing services (e.g., Postgres), as defined by docker-compose.yml. To start the service, run

$ docker compose up --build

to start the two containers and expose the DC endpoint on the localhost of the host machine. To bring the containers down, use

$ docker compose down

Note that this will bring down the postgres (and any other running containers defined in docker-compose.yml) as well. To just bring down a single service (e.g., Data Connect), use $ docker compose down data_connect. Note that the API codebases are currently copied into the container at build time (as opposed to being mounted as a volume). So, containers will need to be rebuilt to reflect any code changes--simply restarting the services will not be effective.

Services are currently only accessible via SSL through nginx, which routes requests to the proper service through port 443 (requests to port 80 are redirected). nginx waits for the service ports to become available (./nginx/docker-entrypoint.sh) before finishing initialization, as it may take some time for the FastAPI services to come up. Note that any new services will need to update ./nginx/docker-entrypoint.sh to reflect the new port to wait upon.

Local (non-Dockerized) Execution

The installation environment for a service relies on poetry for Python package management. To install the required Python packages with a local env for use without Dockerization, run

$ poetry install

within the project root directory. To activate the installed poetry environment, run

$ source "$(poetry env info --path)/bin/activate"

from the project root directory. This has been fairly depreciated in favor of running with docker-compose, so your mileage may vary.

Testing

To run unit tests, execute

$ poetry run --directory [SERVICE_NAME] python -m pytest tests

from the root directory of the ga4gh-node project where SERVICE_NAME in {drs, data_connect}. For data_connect, this will test the service with sqlite, mysql, and postgres backends via the use of the testcontainers package. Running the tests from the docker containers is currently not tested. As noted previously, these unit tests rely on the .env.tests file in the project root for infrastructure configuration and secrets. There are also tests in the tests/common directory, which can be run with

$ poetry run --directory [SERVICE_NAME] python -m pytest ../tests/common

This ensures that the tests for the common code is run in the environment for each dependent service. This common code is also tested within the Github PR workflow (.github/workflows/test.yaml) as well.

The Github repository for this project, ga4gh-node, has been configured such that the test suite runs upon creation of a pull request into the main branch (see .github/workflows/test.yaml).

On Secrets Management

The current implementation instantiates postgres with a database for each service, and also creates a postgres user for each service that only has read-only access to tables within the database (see ./postgres/init/postgres-init.sql.template-initial). This requires the postgres service to have access to the root postgres password and passwords for each service. The postgres service currently relies on the POSTGRES_PASSWORD_FILE environment variable functionality for the root postgres password, and injects the per-service passwords into the postgres service environment via ./postgres/docker-entrypoint.sh. This is needed because the postgres image does not run the postgres service as root and can't read the /run/secrets files directly. Further, postgres initialization relies on envsubst to substitute the secrets (see ./postgres/postgres-init.sh) into SQL initialization scripts.

Each of the services themselves requires access to its own user postgres password. These are handled by the ./secrets files, and are loaded by the FastAPI application via the settings.py file that is in ./[SERVICE_NAME]/api/[API_VERSION]. This is handled by the read_docker_secret() function. Note that settings.py is a bit complicated in that is must handle secrets from either the environment (for unit tests and github workflows) or from /run/secrets files. This functionality is controlled via the USE_DOTENV environment variable. The FastAPI services are currently running as a "fastapi" user. To do this effectively, the entrypoint.sh scripts for the services copies the /run/secrets files to /tmp/secrets with permissions of fastapi:fastapi. The settings.py module then looks for secrets files in this directory. When the services ran as root, secrets were read directly from /run/secrets--which was interesting because docker compose creates secrets with ownership aligned with the host user by default (i.e., not root). When the services were running as root, they had permission to access these files. As the "fastapi" user, secrets need to be moved into /tmp/secrets. Also, for ease of use with the non-root user poetry installs packages globally (i.e., without creating a virtual environment in root's cache, which would'nt be accessible to the fastapi user). For github workflows, secrets (and infrastructure variables) are handled via repository secrets and variables, respectively.

Along the lines of security, the services are currently only accessible via SSL through nginx, which routes requests to the proper service through port 443 (requests to port 80 are redirected). At the moment, nginx uses self-signed certificates that are generated within ./nginx/docker-entrypoint.sh and saved within /etc/nginx/certs. This is currently a persistent volume within docker-compose.yml so that the keys are only regenerated when the volumes are blown away with $ docker compose down -v. A TODO item (issue #26) is to convert postgres to only use SSL for exchanges.

Data Connect

The database schema for Data Connect isn't fixed ahead of time due to the intention for the service to provide access to a flexible set of SQL tables. If you would like to have dummy data loaded into the Postgres database, execute the propagate_postgres.sql script on the Postgres container with the helper script data_connect/tests/run-sql-script.sh:

$ data_connect/scripts/run-sql-script.sh ga4gh-node-postgres-1 data_connect/scripts/propagate_postgres.sql 

Where ga4gh-node-postgres-1 is the name of the running Postgres container, and data_connect/tests/propagate_postgres.sql is the path to the script to be executed. The script relies on finding the .env.infra file, so run the script from the project root. Note that the postgres service defined within docker-compose mounts the pgdata volume, which preserves Postgres data between service runs. So, loading dummy data only needs to happen once. To clear out all Postgres data, run $ docker compose down -v to remove the pgdate volume.

Development Notes

• Implements Data Connect v1.0.0
• Service expects input SQL that roughly looks like postgres -- sqlglot is used to transpile the postgres input to a different backend.
  • Note that sqlglot is focused on interoperability, i.e., it will convert between dialects where there is an explicit conversion rule to follow. If a particular feature doesn't exist in the target dialect (i.e., no conversion rule exists), sqlglot will simply return the input tree--which would likely cause an error in the backend target SQL server. In other words, "SQLGlot ensures your SQL parses and can be expressed in another dialect’s grammar, not that it will actually run correctly on the target database." See common/database.py::truncate_table for an example of handling a feature that doesn't exist in SQLite. if needed (see common/database.py::execute), but this has some limitations such as passing arrays to mysql or sqllite backends.
• Using testcontainers to test with sqlite, postgressql, and mysql backends. Hopefully will test with sql server in future as well.
• Paginated queries expect the same type of request as the initial request -- e.g., paginated requests for /tables expect GET and paginated requests for /search expect POST (with the same body as the initial request)-
• The Postgres service mounts the pgdata volume, which preserves server data and configuration from the previous run unless the service is brought down with the $ docker compose down -v command. If pgdata is available to carry over data from the previous run, the postgres-init.sql script (which creates the data connect user with limited permissions) will not execute.
• /search endpoint
  • Infers table schema from data for /search endpoint from the first row of the result.
    • This may result in a malformed schema if there is missing data in the first row.
    • Furthermore, the implementation differs from the specification in that the schema can't be inferred if there are no results from the query--so including a table schema is now optional in the TableData response.
    • Columns involving inline functions (e.g., count(*) or max(foo)) should be aliased for consistency across underlying SQL server versions--as different SQL servers vary in the internal labels they assign to these columns.
  • The service attempts to preload the phenopacket schemas from https://schemablocks.org/schemas/sb-phenopackets/current/ to enable the ga4gh_type function. These are the only ga4gh_type references that will pass validation, and the server will throw an exception if the user attempts to pass a reference to a schema that hasn´t been preloaded.
  • /search endpoint also doesn't support queries that:
    • exclusively return sqlite builtin functions, e.g., "SELECT DATE('now')"
    • only have a user-supplied OFFSET without a preceeding LIMIT
    • have negative LIMIT values

Data Repostiory Service (DRS)

The database schema for DRS is currently being created by postgres/init/postgres-init.sql.template-drs. If you would like to have dummy data loaded into the Postgres database, execute the propagate_postgres.sql script on the Postgres container with the helper script drs/tests/run-sql-script.sh:

$ drs/scripts/run-sql-script.sh ga4gh-node-postgres-1 drs/scripts/propagate_postgres.sql 

Where ga4gh-node-postgres-1 is the name of the running Postgres container, and drs/tests/propagate_postgres.sql is the path to the script to be executed. The script relies on finding the .env.infra file, so run the script from the project root. Note that the postgres service defined within docker-compose mounts the pgdata volume, which preserves Postgres data between service runs. So, loading dummy data only needs to happen once. To clear out all Postgres data, run $ docker compose down -v to remove the pgdate volume.

To test the running service, one may want to obtain DRS ids to submit to the API. To do this, first log into the postgres container:

$ docker exec -it ga4gh-node-postgres-1 /bin/bash 

and then connect to the running postgres server's drs database via drs_user:

$ psql -U drs_user -d drs

From there, one can dump the drs_objects table:

$ select * from drs_objects;

Development Notes

• Targets DRS v1.5.0, but not completely implemented.
• DRS IDs are 32 characters in [A-Za-z0-9.-_~]
• Questions
  • What is the intention of the 204 (Authorizations not supported) response for the OPTIONS /objects/{object_id}? How is that "not supported" status indicated? How does this response differ from the 405 response?
  • What is the difference between the 401 (Request not authorized) and 403 (User is not authorized) response codes to GET /objects/{object_id}?
  • Why is the GET /objects endpoint missing? OPTIONs and POST are there. It feels strange to have the POST version, and not the GET.
  • What does it mean for OPTIONS /objects to return a 404 if a DRS object is missing? Shouldn't this be captured as part of the unresolved_drs_objects parameter?