Testing

Note

Unit tests and integration tests are automatically executed with every PR to for both frontend and backend in their respective repositories.

Backend Testing

Tests can be run with tox automation: just run tox on project root (remember to install it first with pip install tox).

Unit Testing

In order to run the unit tests, security checks with bandit, Sphinx documentation check for links consistency and HTML output and flake8 (coding style guide) tox. To run the unit tests in parallel use:

$ tox -p auto

To run environments separately use:

$ # list environments
$ tox -l
$ # run flake8
$ tox -e flake8
$ # run bandit
$ tox -e bandit
$ # run docs
$ tox -e docs

Integration Testing

Integration tests are executed with pytest, and require a running backend. Follow the instructions in Build and Deployment for development setup of backend. Install

  • aiofiles
  • aiohttp
  • motor
  • pytest
  • pytest-asyncio

After the backend has been successfully set up, run the following in the backend repository root directory: pytest tests/integration. This command will run a series of integration tests.

Pytest fixtures

Be sure to familiarize yourself with pytest fixtures and their scope. They are pure magic.

Fixtures are defined in tests/integration/conftest.py.

Basically, a fixture is run before a new module/class/function/<fixture scope here> is executed, according to its defined scope. Statements before a yield statement function as a Setup, and statements after are Teardown. When there’s no yield, there’s no teardown either, only setup.

Running the tests without cleaning the database use the --nocleanup option, as follows:

  • pytest --nocleanup tests/integration for the whole test suite;
  • pytest --nocleanup tests/integration/test_module.py::TestClass::test_method for specific tests.

Note that this option might cause some tests to fail and it is only recommended for debugging the data from the integration tests.

Debugging tests

Depending on your setup, there are different ways to debug tests run with pytest. Without covering the possibilities that IDEs offer, here are the possibilities with the CLI and the Python debugger (PDB).

  • Run a single test with pytest tests/integration/test_module.py::TestClass::test_method. Or remove the method name to run a single class test.
  • Follow pytest docs, and run tests with pytest -x --pdb tests/integration to drop to the PDB on the first error.
  • Similarly, you can use ipdb to use the more feature-full pdb, using IPython.
$ pip install ipdb
$ pytest -x --pdb -pdbcls=IPython.terminal.debugger:TerminalPdb tests/integration
  • Together with --pdb, call breakpoint() in the code to pause the test execution at a predefined point.

Performance Testing

Performance tests utilize Locust load testing framework (install it first with pip install locust). Performance tests also require a running backend, similar to integration tests. After the backend has been set up, running the following commands in the repository root directory will run different performance related tests in headless mode (all test data printed to terminal).

$ # run tests that post objects/submissions
$ locust --tags post
$ # run tests that query for objects/submissions
$ locust --tags query

The configuration values for running performance tests are predefined in the locust.conf file in the repository root directory. All configuration options (as defined here) can be overridden and new options can be added by either editing the current locust.conf file or running the test with additional tags, e.g.:

$ # this will run the post test for 30 seconds
$ locust --tags post --run-time 30s

Frontend Testing

Run Jest-based tests with npm test. Check code formatting and style errors with npm run lint:check and fix them with npm run lint. Respectively for formatting errors in json/yaml/css/md -files, use npm run format:check or npm run format. Possible type errors can be checked with npm run flow.

We’re following recommended settings from eslint, react and prettier - packages with a couple of exceptions, which can be found in .eslintrc and .prettierrc. Linting, formatting and testing are also configured for you as a git pre-commit, which is recommended to use to avoid fails on CI pipeline.

End to End testing

End-to-end tests can be run on local host with npx cypress open in frontend repository. These tests required a running backend, follow the instructions in Build and Deployment for development setup of backend.

If the frontend is started with npm start no changes required in the setup.