Skip to content
This repository was archived by the owner on Aug 14, 2025. It is now read-only.
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
1cc2552
Use TRAVIS_BRANCH to select branch of toolchain being tested #125
Dec 11, 2019
1d51b01
Add logic to also support pull requests
Dec 11, 2019
751a263
Fix syntax error
Dec 11, 2019
8b12ee9
Remove yaml-cpp install
Eder-K Feb 9, 2020
5e85037
Remove yaml-cpp install
Eder-K Feb 9, 2020
6f95b24
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
8e9775c
Add preciceDict
Eder-K Feb 9, 2020
8ebad2a
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
5c35e1b
Create preciceDict
Eder-K Feb 9, 2020
4804124
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
517e883
Create preciceDict
Eder-K Feb 9, 2020
42eaacb
Delete preciceDict
Eder-K Feb 9, 2020
318e7d7
Create preciceDict
Eder-K Feb 9, 2020
72d4825
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
6a97d85
Create preciceDict
Eder-K Feb 9, 2020
2527d55
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
b3f0f38
Add preciceDict
Eder-K Feb 9, 2020
0a2634e
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
cb4b17a
Add preciceDict
Eder-K Feb 9, 2020
1ca1e42
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
f03a6b1
Add preciceDict
Eder-K Feb 9, 2020
fa0f5c1
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
74c1c48
Add preciceDict
Eder-K Feb 9, 2020
f77f031
Delete precice-adapter-config.yml
Eder-K Feb 9, 2020
ffdcb1a
Add preciceDict
Eder-K Feb 9, 2020
efcdb99
Merge branch 'develop' into i_125
BenjaminRodenberg Feb 11, 2020
0ee0349
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
38b2e4a
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
4306042
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
88f3baa
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
3ecbf6d
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
c4fde6b
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
40885a2
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
6f3c43b
Update .travis.yml
BenjaminRodenberg Feb 11, 2020
981d36e
use correct build-arg for OF Adapter
BenjaminRodenberg Feb 11, 2020
f056056
remove if fork == false
BenjaminRodenberg Feb 11, 2020
2c9a9a8
Fix syntax
BenjaminRodenberg Feb 11, 2020
0260d1e
Merge branch 'i_125' into benjaminrueth_develop
BenjaminRodenberg Feb 11, 2020
f5d56b7
Remove preCICe builds
Eder-K Feb 11, 2020
66006ed
Print wmake.log
Eder-K Feb 11, 2020
06ddd7d
Merge branch 'EderK-update-ofadapter' into i_125
BenjaminRodenberg Feb 11, 2020
7d968b3
Merge branch 'i_125' into benjaminrueth_develop
BenjaminRodenberg Feb 11, 2020
6ee7bcf
remove depolyment for tests
BenjaminRodenberg Feb 12, 2020
bf822e9
Use --branch option for all tests
BenjaminRodenberg Feb 12, 2020
0da3836
Merge branch 'i_125' into benjaminrueth_develop
BenjaminRodenberg Feb 12, 2020
9d50b28
Merge branch 'develop' into i_125
BenjaminRodenberg Feb 12, 2020
d060bd7
Merge branch 'i_125' into benjaminrueth_develop
BenjaminRodenberg Feb 12, 2020
ae95543
Merge branch 'develop' into i_125
Eder-K Feb 15, 2020
a11358c
Merge branch 'develop' into i_125
BenjaminRodenberg Feb 22, 2020
d0c4f4e
Add jinja2 (accidentally removed in merge)
BenjaminRodenberg Feb 22, 2020
29ba320
Revert changes applied to reference output
BenjaminRodenberg Feb 22, 2020
67bbdcf
Reintroduce changes from #147 (accidentally removed in merge)
BenjaminRodenberg Feb 22, 2020
e38e2bf
Remove libyaml-cpp-dev
BenjaminRodenberg Feb 22, 2020
6463c10
Add missing line
BenjaminRodenberg Feb 22, 2020
37de72a
Merge branch 'i_125' into benjaminrueth_develop
BenjaminRodenberg Feb 22, 2020
56839a2
Add --docker-username to system_testing.py to define SYSTEST_REMOTE
BenjaminRodenberg Feb 22, 2020
c0f47b6
revert .travis.yml
BenjaminRodenberg Feb 22, 2020
65dfd83
Keep SYSTEST_REMOTE in .env files (will be used as default by docker-…
BenjaminRodenberg Feb 23, 2020
7259fcb
Fix generate_test.py w.r.t PRECICE_BASE in .env
BenjaminRodenberg Feb 23, 2020
74dd0cd
Extend documentation with respect to environment variables and runnin…
BenjaminRodenberg Feb 23, 2020
156acbc
Provide reasonable default for PRECICE_BASE in .env files
BenjaminRodenberg Feb 23, 2020
6f9631e
Merge branch 'develop' into provide_docker_username_via_system_testing
BenjaminRodenberg Feb 23, 2020
b05e7a9
Remove debugging output
BenjaminRodenberg Mar 2, 2020
ecd06a3
Merge branch 'develop' into provide_docker_username_via_system_testing
Eder-K Mar 12, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,9 @@ All adapters are built using user `precice` with `gid` and `uid` equal to 1000 (

This is done to ensure consistent user rights for writing and reading from mounted directories without a need to use root user on the host system. Running root within container will work, but will lead to output directory being owned by root on host. Running with different users in different directories will cause problems with access to e.g. `/home/precice/Data/Exchange` directory. More information and motivation behind the solution can be found [this blogs post](https://medium.com/@nielssj/docker-volumes-and-file-system-permissions-772c1aee23ca) and [this issue](https://github.com/moby/moby/issues/2259).

## Running tests
## The tests

At this stage, we spawn containers with the necessary solvers for the simulation. The execution is controlled using [docker-compose](https://docs.docker.com/compose/). Separate volumes are created for the input and the output of every adapter. An additional docker network is created and used for the communication between containers. Exchange of data is done using an additional, common volume. The general pattern across the tests for such volumes are paths
such as `/home/precice/Data/Exchange`, `/home/precice/Data/Input`, `/home/precice/Data/Output`, as explained above.
The actual tests are provided in `systemtests/tests`. For testing, we spawn containers with the necessary solvers for the simulation. The execution is controlled using [docker-compose](https://docs.docker.com/compose/). Separate volumes are created for the input and the output of every adapter. An additional docker network is created and used for the communication between containers. Exchange of data is done using an additional, common volume. The general pattern across the tests for such volumes are paths such as `/home/precice/Data/Exchange`, `/home/precice/Data/Input`, `/home/precice/Data/Output`, as explained above.

The tests are based on the corresponding tutorials from the [tutorials repository](https://github.com/precice/tutorials). Since tests run using independent containers, several modifications are necessary to split the input to solvers in different directories, to adjust the simulated time, or to potentially change a solver's version, as well as to modify the `precice-config.xml` for the communication pattern we use in docker. This is done using the `Dockerfile.tutorial_data` image, which clones the tutorial data and adjusts and splits the input to different solvers. It is then used as the first container spawned by `docker-compose`.

Expand All @@ -64,3 +63,18 @@ Since it is possible for the output of two identical tests to differ in floating
the files are also compared numerically using `compare_results.sh`. This script strips away non-numerical data and then compares files field by field with respect to a specified relative tolerance (default maximum allowed difference is 1 percent)

In either case after a test finishes, all the related volumes, networks and containers are removed.

### Required environment variables

The following environment variables are required for running the tests:

* `SYSTEST_REMOTE`: Specifies the user on Dockerhub from which the baseimage is pulled. Example: `precice/`
* `PRECICE_BASE`: Specifies the features of the baseimage used for running the test. Example: `ubuntu1604.home-develop` or `ubuntu1804.home-develop`
* `<PARTICIPANT>_TAG`: Specifies the tag that will be used for the baseimage for each of the participants. Example: `latest`

### Running the tests

There are two possibilities for running the tests:

1. Use `system_testing.py`: Go to `systemtests` and run `python3 system_testing.py -s <TESTCASE>` (e.g. `python3 system_testing.py -s fe-fe --base Ubuntu1804.home`). For more information on the script run `python3 system_testing.py --help`.
2. Use `docker compose up`: Go to the corresponding test (e.g. `systemtests/tests/TestCompose_fe-fe.Ubuntu1804.home`) and run `docker-compose up`. Note that `.env` defines the previously mentioned environment variables required for running the tests.
1 change: 1 addition & 0 deletions generate_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ def generate_test_structure(test_name = None, base_image = None,
solvers = [ a.replace('-adapter','') for a in adapters ]
participants = [ p.replace(':','_') for p in participants]
input_volumes = ['input_' + p for p in participants]
precice_base = '-' + base_image.lower() + '-develop'
# to pass zip to jinja
zipped_input = zip(participants, solvers, adapters)

Expand Down
29 changes: 15 additions & 14 deletions system_testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,16 @@
from subprocess import CalledProcessError
from common import call, ccall, get_test_variants, filter_tests, get_test_participants

def build(systest, tag, branch, local, force_rebuild):
def build(systest, tag, branch, docker_username, local, force_rebuild):
""" Builds a docker image for systest. """

baseimage_name = "precice-{tag}-{branch}:latest".format(tag = tag, branch=branch)
baseimage_name = "precice-{tag}-{branch}:latest".format(tag=tag, branch=branch)
test_tag = "-".join([systest, tag, branch])

docker.build_image(tag = test_tag,
build_args = {"from" : docker.get_namespace() +
baseimage_name if local
else 'precice/' + baseimage_name},
else docker_username + '/' + baseimage_name},
force_rebuild = force_rebuild)

def run(systest, tag, branch):
Expand All @@ -37,14 +37,14 @@ def run(systest, tag, branch):
shutil.rmtree("Logs", ignore_errors=True)
ccall("docker cp " + test_tag + ":Output . ")

def build_adapters(systest, tag, branch, local, force_rebuild):
def build_adapters(systest, tag, branch, local, docker_username, force_rebuild):
""" Builds a docker images for a preCICE adapter, participating in tests """
baseimage_name = "precice-{tag}-{branch}:latest".format(tag = tag, branch=branch)

participants = get_test_participants(systest)
docker_args = { 'tag': '',
'build_args': {"from": docker.get_namespace() + baseimage_name if local
else 'precice/' + baseimage_name },
else docker_username + '/' + baseimage_name },
'force_rebuild': force_rebuild,
'dockerfile': 'Dockerfile'}

Expand All @@ -58,7 +58,7 @@ def build_adapters(systest, tag, branch, local, force_rebuild):
if os.path.exists("Dockerfile.{}".format(participant)):
docker.build_image(**docker_args)

def run_compose(systest, branch, local, tag, force_rebuild, rm_all=False, verbose=False):
def run_compose(systest, branch, docker_username, local, tag, force_rebuild, rm_all=False, verbose=False):
""" Runs necessary systemtest with docker compose """

test_dirname = "TestCompose_{systest}".format(systest = systest)
Expand All @@ -69,7 +69,7 @@ def run_compose(systest, branch, local, tag, force_rebuild, rm_all=False, verbos
# set up environment variables, to detect precice base image, that we
# should run with and docker images location
export_cmd = "export PRECICE_BASE=-{}; ".format(adapter_base_name)
extra_cmd = "export SYSTEST_REMOTE={}; ".format(docker.get_namespace()) if local else ""
extra_cmd = "export SYSTEST_REMOTE={}; ".format(docker.get_namespace() if local else docker_username + "/")
compose_config_cmd = "docker-compose config && "
compose_exec_cmd = "bash ../../silent_compose.sh {}".format('debug' if verbose else "")
copy_cmd = "docker cp tutorial-data:/Output ."
Expand Down Expand Up @@ -159,22 +159,22 @@ def comparison(pathToRef, pathToOutput):
sys.exit(0)


def build_run_compare(test, tag, branch, local_precice, force_rebuild, rm_all=False, verbose=False):
def build_run_compare(test, tag, branch, docker_username, local_precice, force_rebuild, rm_all=False, verbose=False):
""" Runs and compares test, using precice branch. """
compose_tests = ["dealii-of", "of-of", "su2-ccx", "of-ccx", "of-of_np",
"fe-fe","nutils-of", "of-ccx_fsi"]
test_basename = test.split(".")[0]
if local_precice:
build_adapters(test_basename, tag, branch, local_precice, force_rebuild)
build_adapters(test_basename, tag, branch, docker_username, local_precice, force_rebuild)
if test_basename in compose_tests:
run_compose(test, branch, local_precice, tag, force_rebuild, rm_all, verbose)
run_compose(test, branch, docker_username, local_precice, tag, force_rebuild, rm_all, verbose)
else:
# remaining, non-compose tests
test_dirname = "Test_{systest}".format(systest=test)
test_path = os.path.join(os.getcwd(), 'tests', test_dirname)
with common.chdir(test_path):
# Build
build(test_basename, tag, branch, local_precice, force_rebuild)
build(test_basename, tag, branch, docker_username, local_precice, force_rebuild)
run(test_basename, tag, branch)
# Preparing string for path
pathToRef = os.path.join(os.getcwd(), "referenceOutput")
Expand Down Expand Up @@ -225,10 +225,11 @@ def compose_tag(docker_username, base, features, branch):
parser.add_argument('-l', '--local', action='store_true', help="Use local preCICE image (default: use remote image)")
parser.add_argument('-s', '--systemtest', type=str, help="Choose system tests you want to use",
choices = common.get_tests(), required = True)
parser.add_argument('-b', '--branch', help="preCICE branch to use", default=os.environ["TRAVIS_BRANCH"] if os.environ["TRAVIS_PULL_REQUEST"] == "false" else os.environ["TRAVIS_PULL_REQUEST_BRANCH"]) # make sure that branch corresponding to system tests branch is used, if no branch is explicitly specified. If we are testing a pull request, make sure to test agains branch from which PR originated.
parser.add_argument('-b', '--branch', help="preCICE branch to use", default=os.environ["TRAVIS_BRANCH"] if ("TRAVIS_BRANCH" in os.environ and os.environ["TRAVIS_PULL_REQUEST"] == "false") else (os.environ["TRAVIS_PULL_REQUEST_BRANCH"] if ("TRAVIS_PULL_REQUEST_BRANCH" in os.environ) else "develop")) # make sure that branch corresponding to system tests branch is used, if no branch is explicitly specified. If we are testing a pull request, make sure to test agains branch from which PR originated.
parser.add_argument('-u', '--docker-username', default=os.environ["DOCKER_USERNAME"] if "DOCKER_USERNAME" in os.environ else "precice", help="Name of the docker user on dockerhub that provides the preCICE and adapter base images for the tests. Defines the environment variable SYSTEST_REMOTE.")
parser.add_argument('-f', '--force_rebuild', nargs='+', help="Force rebuild of variable parts of docker image",
default = [], choices = ["precice", "tests"])
parser.add_argument('--base', type=str,help="Base preCICE image to use",
parser.add_argument('--base', type=str, help="Base preCICE image to use. Defines the environmental variable PRECICE_BASE.",
default= "Ubuntu1604.home")
parser.add_argument('-v', '--verbose', action='store_true', help="Verbose output of participant containers")
args = parser.parse_args()
Expand All @@ -241,5 +242,5 @@ def compose_tag(docker_username, base, features, branch):
else:
test = test[0]
tag = args.base.lower()
build_run_compare(test, tag, args.branch.lower(), args.local,
build_run_compare(test, tag, args.branch.lower(), args.docker_username, args.local,
args.force_rebuild, rm_all=False, verbose=args.verbose)
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
{%- endfor %}

SYSTEST_REMOTE=precice/
PRECICE_BASE=
PRECICE_BASE={{precice_base}}
2 changes: 1 addition & 1 deletion tests/TestCompose_fe-fe.Ubuntu1804.home/.env
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
FENICS_TAG=latest
SYSTEST_REMOTE=precice/
PRECICE_BASE=
PRECICE_BASE=-ubuntu1804.home-develop
2 changes: 1 addition & 1 deletion tests/TestCompose_nutils-of.Ubuntu1804.home/.env
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
OPENFOAM_TAG=latest
SYSTEST_REMOTE=precice/
PRECICE_BASE=
PRECICE_BASE=-ubuntu1804.home-develop
2 changes: 1 addition & 1 deletion tests/TestCompose_of-ccx.Ubuntu1604.home/.env
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ OPENFOAM_TAG=latest
CALCULIX_TAG=latest
SYSTEST_REMOTE=precice/

PRECICE_BASE=
PRECICE_BASE=-ubuntu1604.home-develop
3 changes: 2 additions & 1 deletion tests/TestCompose_of-ccx_fsi.Ubuntu1604.home.PETSc/.env
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
OPENFOAM_TAG=latest
CALCULIX_TAG=latest
SYSTEST_REMOTE=precice/
PRECICE_BASE=

PRECICE_BASE=-ubuntu1604.home.petsc-develop
2 changes: 1 addition & 1 deletion tests/TestCompose_of-of.Ubuntu1604.home/.env
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
OPENFOAM_TAG=latest
SYSTEST_REMOTE=precice/
PRECICE_BASE=
PRECICE_BASE=-ubuntu1604.home-develop
2 changes: 1 addition & 1 deletion tests/TestCompose_of-of_np.Ubuntu1604.home/.env
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
OPENFOAM_TAG=latest
SYSTEST_REMOTE=precice/
PRECICE_BASE=
PRECICE_BASE=-ubuntu1604.home-develop
2 changes: 1 addition & 1 deletion tests/TestCompose_su2-ccx.Ubuntu1604.home/.env
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
SU2_TAG=latest
CALCULIX_TAG=latest
SYSTEST_REMOTE=precice/
PRECICE_BASE=
PRECICE_BASE=-ubuntu1604.home-develop