preCICE system tests

The tutorials repository hosts cases that need multiple components from the preCICE ecosystem to run. Te system tests automatically run tutorials combining the required components and compare the numerical results to references. While the main purpose is to run complete tests in the continuous integration workflows of preCICE, you can also run these tests on your laptop.

Running

The main workflow for the user is executing the systemtests.py script, which is the same that the GitHub Actions workflow executes. Depending on the options given to the script, it reads in the respective metadata files and generates docker-compose.yaml files that can start a fully-defined coupled simulation. For arguments that are not provided, default values from components.yaml are used.

Manual and nightly runs on GitHub

The System tests (manual/nightly) workflow executes the release test suite nightly and can also be triggered manually.

On the workflow page, click Run workflow. The default values will execute the release test suite using the latest develop branches of every component. If you want to override the version of some component, specify it in the respective field. Commit hashes, branches, and tags are all accepted. Branches and tags will get automatically resolved to their current commit on GitHub before starting any test, and all tests will use the same version of any common component.

The available test suites are found in tests.yaml and common values are:

• quickstart, elastic-tube-1d, or any other tutorial (see exceptions)
• openfoam-adapter, micro-manager, fmi-runner, or similar test cases involving the respective component
• precice is a subset of cases that cover a range of preCICE features
• release is for all available but some very long or known to fail tests
• extra is for some longer tests
• selected and system-tests-dev for some special cases

The Use workflow from is a default option of GitHub Actions that concerns the GHA workflow file itself.

Running from a pull request

Several repositories include a workflow that allows triggering the system tests by adding the trigger-system-tests label to the pull request. The event is triggered only at the moment of adding the label, so you need to remove the label and add it again if needed.

See the system tests workflow in the OpenFOAM adapter for an example.

Running from the GitHub CLI

The GitHub CLI allows triggering workflows of a GitHub project. For example:

gh workflow run system-tests-latest-components.yml -f suites=release

More arguments are available, for example:

gh workflow run system-tests-latest-components.yml -f suites=release -f build_args="PLATFORM:ubuntu2404,PRECICE_REF:develop" -f log_level="DEBUG" --ref=develop

The build_args override the defaults set in tools/tests/components.yaml.

Running locally

To run locally, you will need Docker, Docker Compose, and Python 3.

Navigate into the directory tools/tests/ of the tutorials, make a Python virtual environment, and install the dependencies:

python -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt

To test a certain test-suite defined in tests.yaml, use:

python systemtests.py --suites=quickstart

This will build and connect Docker containers and run tutorials in the runs/ directory in the root of the tutorials.

To clean up at the end, you might want to run a docker system prune -a and remove the runs/ directory to save space.

There are also some auxiliary scripts (run without arguments):

• print_test_suites.py: Print all test suites defined in tests.yaml
• print_case_combinations.py: Print all possible combinations of participants in a tutorial, using its metadata.yaml.

Understanding the logs

For a local run, look into the tutorials/runs/ directory, where you will find a time-stamped directory for every test executed.

Each of these directories includes the usual tutorial case files and logs, as well as:

1. system-tests-build.log: The logs of building the respective components.
2. system-tests-run.log: The logs of running the simulation (intermixed, from all participants).
3. system-tests-compare.log: The logs for the comparison to the reference results.

In addition, in the directories of the cases executed, you can find system-tests-<case>.log files.

Numerical regressions

When the tests fail at the results comparison step, this typically means that there are numerical regressions (unless something wrong went undetected in a previous step). We use fieldcompare to compare all preCICE exports to reference results generated from a previous run. Relevant files:

• precice-exports/: The coupling meshes of the test run.
• reference-results/: The coupling meshes of the reference run, as stored on Git LFS, expanded into reference-results-unpacked. For test cases using implicit coupling, the reference .tar.gz also contains the reference precice-*-iterations.log files.
• diff-results/: Numerical difference of the results in the two directories (computed with fieldcompare dir --diff precice-exports/ reference/). These are only present on failed comparisons.
• iterations-logs/: The precice-*-iterations.log files of the test run. Only present in test cases using implicit coupling. The comparisons to references only take into account the file SHA-256 checksums.

To reproduce the comparison locally, use the same fieldcompare command:

fieldcompare dir precice-exports/ reference-results-unpacked/<case>/ \
             --ignore-missing-reference-files \
             --ignore-unsupported-file-formats \
             -rtol 3e-7

The default relative tolerance (-rtol) is 3e-7. Per-test overrides are possible in tests.yaml (e.g., tolerance: 1e-2). Set skip_compare: true to skip the comparison step and only verify that the build and run steps succeed.

The differences are only shown per file, and there is no global metric or other summary (see related discussion in fieldcompare).

Alternatively, visualize the precice-exports/diff_*.vtu in ParaView.

Re-running from CI artifacts

When a system test fails in CI, download the full artifact:

system_tests_run_<run_id>_<run_attempt>_full

(a smaller _logs archive contains only log files). The archive contains a shared runs/ directory:

runs/
├── tools/                              # Dockerfiles and helpers (shared)
└── <tutorial>_<cases>_<timestamp>/     # one folder per system test
    ├── docker-compose.tutorial.yaml
    ├── docker-compose.field_compare.yaml   # written at build time when compare is configured
    ├── rerun-system-test.sh
    ├── system-tests-build.log
    ├── system-tests-run.log
    ├── system-tests-compare.log
    └── …

To re-run one test locally:

1. Extract the zip and keep the runs/ layout (the test folder needs the sibling tools/ directory).
2. cd into the test folder.
3. Run ./rerun-system-test.sh (or sh rerun-system-test.sh).

The script rebuilds images, runs the tutorial, and (if present) runs fieldcompare with --exit-code-from field-compare, matching the Python runner. Compose paths are relative to the test folder (.. is the parent runs/ directory), so you can move the extracted tree elsewhere on a Linux host with Docker.

docker-compose.field_compare.yaml is written when the test is prepared for Docker build. The replay script fixes common permission issues from extracted archives.

Fieldcompare requires reference results in the artifact. If not already unpacked during the original CI run, unpack them manually first.

Extending

Adding new tests

Tests and test suites are defined in tests.yaml. By convention, every tutorial defines a test suite with the same name as its directory, and several test cases using combinations of the available participants. These test cases are later referenced by other test suites: these are typically the release and the test suites of different tested components.

The available cases are listed in the metadata.yaml of each tutorial. To add a new tutorial case as a test, add it to metadata.yaml and then define a test using it. Include that test in the relevant test suites.

Use the max_time or max_time_windows parameters to restrict the runtime of the test to the first few coupling time windows, to save time. Aim for a runtime of less than a minute (assuming cached components), if possible.

Some tutorials require setup before the simulation (e.g. switching configuration files). Use optional run-before and run-after fields in tests.yaml to run shell commands in the copied tutorial directory after copying and before Docker build (run-before), or after the simulation and before field comparison (run-after). Example:

run-before: ./set-case.sh 1d3d

You will need to define a reference results file. The reference results can and should be generated on GitHub using the Generate reference results (manual) workflow for the respective test suite. You might want to temporarily set the selected test suite for requesting results only for a subset of test cases.

Note that you will need to define the TUTORIALS_REF in the file reference_versions.yaml to match the respective branch. Restore that to develop after that. See a related issue.

The results will be added to a Git LFS, but you will need special push access: just use the aforementioned GitHub Actions workflow, instead.

Adding new components

To add a new component, a few changes are needed:

1. In the components.yaml, add a new component, defining all the parameters that it might need.
   • Add these parameters into the reference_versions.yaml.
   • Add fields for these parameters into the system-tests-latest-components.yml workflow.
2. In the dockerfiles/<platform>/Dockerfile, define a new stage for building your new component. Use the parameters you defined above.
   • Defining a <component>_PR variable will let you integrate the system tests into the respective component repository.
3. In the component_templates.yaml, define a component template. These are Jinja templates that are used by Docker Compose, and the main detail is how to run a simulation using that component.
4. Refer to the new component in the metadata.yaml of some tutorial and define some tests.

Adding new repositories that can trigger the tests

If you want to trigger the system tests from a new repository:

1. Add a workflow file to that repository (under .github/workflows/). Example: OpenFOAM adapter.
2. Create label as a trigger for workflow (under Issues->Labels->New label).
3. Give permissions: In the preCICE organization settings, you need to add the new repository to the action secret WORKFLOW_DISPATCH_TOKEN and to the default actions runner group. Adding the new label directly to the pull request in which you add the workflow should already trigger the system tests.

Implementation details

Each tutorial contains automation scripts (mainly run.sh and clean.sh), as well as metadata (metadata.yaml). The metadata file describes the available cases, how to run them, as well as their dependencies. A central tests.yaml file defines test suites, which execute different combinations of cases. The Python script systemtests.py executes the tests, filter for specific test suites.

Read more about the system tests in the publication System Regression Tests for the preCICE Coupling Ecosystem.

name: Elastic tube 3D
path: elastic-tube-3d
url: https://precice.org/tutorials-elastic-tube-3d.html

participants:
  - Fluid 
  - Solid 

cases:
  fluid-openfoam:
    participant: Fluid
    directory: ./fluid-openfoam
    run: ./run.sh 
    component: openfoam-adapter
  
  solid-calculix:
    participant: Solid
    directory: ./solid-calculix
    run: ./run.sh 
    component: calculix-adapter 
  
  solid-fenics:
    participant: Solid
    directory: ./solid-fenics
    run: ./run.sh
    component: fenics-adapter 

openfoam-adapter:
  template: component-templates/openfoam-adapter.yaml
  build_arguments:
    PLATFORM:
      default: "ubuntu_2404"
    PRECICE_REF:
      repository: https://github.com/precice/precice
      default: "develop"
    PRECICE_PRESET:
      default: "production-audit"
    TUTORIALS_REF:
      repository: https://github.com/precice/tutorials
      default: "develop"
    OPENFOAM_EXECUTABLE:
      default: "openfoam2512"
    OPENFOAM_ADAPTER_REF:
      repository: https://github.com/precice/openfoam-adapter
      default: "develop"

image: precice/fenics-adapter:
depends_on:    
  prepare:
    condition: service_completed_successfully
volumes:
  - :/runs
command: >
  /bin/bash -c "id && 
  cd '/runs//' &&
   | tee system-tests_.log 2>&1"