Testing¶
We use pytest to create tests for Python components. Tests should be placed
in a top-level directory called tests/ in each repository.
Test workflows¶
The core functionality of Dyff is executing audit workflows. These workflows have data dependencies on one another, so our main API test suite consists of running an entire audit pipeline that exercises the API functions, with workflows later in the pipeline consuming data produced by workflows earlier in the pipeline.
The test pipeline is implemented in the dyff-audit repository. It can be run
against either the DyffLocalPlatform implemented in dyff-audit, or
against a remote Dyff instance via dyff-client.
Testing DyffLocalPlatform¶
From the dyff-audit repository root directory, execute:
$ pytest -rP
The -rP flag causes pytest to echo stdout and stderr from the
tests to the console. This lets you see the generated IDs of each workflow.
By default, the tests use a temporary directory to store data artifacts created by the workflows. To capture these data artifacts for inspection, specify a non-temporary directory for the output, e.g.:
$ pytest -rP --storage_root=tests/outputs
Testing a remote Dyff instance¶
You can also test a remote installation using the same test suite. This requires specifying client credentials. You can use either command line flags or environment variables.
DYFF_API_TOKEN=<token>/--api_token=<token>A Dyff API token with sufficient permissions to execute the tests.
DYFF_API_ENDPOINT=<endpoint>/--api_endpoint=<endpoint>The API endpoint to connect to. E.g.,
https://api.dyff.io/v0DYFF_API_INSECURE=1/--api_insecure[Optional] Don’t check SSL certificates. Use this only when connecting to a trusted local Dyff instance.
Invoke the test suite with the --test_remote flag to test a remote instance:
$ pytest -rP --test_remote
Note
When testing a remote instance, the tests will show success if the remote instance responded to the request with a success code. You will need to verify separately that the workflows actually ran successfully.