Skip to content

Developer's guide: Continuous integration

Jump to bottom
MD edited this page Sep 26, 2025 · 2 revisions

Introduction

Continuous integration (CI), also known as nightly test or nightly regression, is a system consisted of a workflow manager, a series of testers, and a testing platform. When a commit is pushed to the GitHub repository, a pull request is created, or when a new release is published, CI runs the testers on the testing platform to make sure the new code does not cause anything unexpected.

Some testers have "gold data", which are results generated ion previous CI runs. CI compares the results generated by the tester in the current run with these gold data to make sure the results are consistent.

A failed CI run can be caused by one of the following 3 factors:

  • There is an issue in communicating with the testing platform, creating the test environment, and building the package;
  • One of the testers encounters a runtime exception;
  • The result of a tester is different from its gold data.

The latter 2 are related to the code change to the package itself. While runtime exception is almost always a problem, difference with test data is sometimes benign and intended. For example, an algorithmic improvement can cause the result to change. If you are certain that all the differences are expected and intended, you can update the gold data to allow CI to pass again.

Infrastructure

Workflow manager

We use Actions workflow in the GitHub repository to automate CI. The workflow is defined in .github/workflows. The workflow YAML file instruct Actions to create the test environment, install dependencies, and run tests.

Testing platform (runner)

The testing platform is also called the runner in GitHub Actions. We use self-hosted runners. For security purposes, contributors external to the APS will need approval to run CI workflows for their pull request.

Testers and data

A tester is a Python script that contains one or more functions named following the pattern test_*. This is the pattern recognized by pytest, where each function is considered as a test case. pytest marks a test case as failure when a exception, either raised exceptions or AssertionError, is encountered.

A typical tester used in our repository is structured as:

def test_testname(generate_gold: bool):
    data = load_test_data()
    
    # Build and run computation task
    task = Task(data)
    task.run()

    # Fetch data and run comparison
    result = task.get_result()
    if generate_gold:
        save_gold_data(result)
    else:
        gold_result = load_gold_data()
        assert allclose(result, gold_result)

The input data and gold data of all testers are stored separately on GitLab.

Adding testers

It is a good practice to provide a tester for a new feature you deliver. To create a tester, add a file starting with test_ under tests. In your test script, create a test class subclassing test_utils.BaseTester or any of its subclasses with custom data loading methods. Reference existing test scripts as templates. If you are contributing an end-to-end test that runs a complete reconstruction job and wish to compare the test result with gold data, use the @test_utils.TungstenDataTester.wrap_recon_tester decorator for your test method which compares the return of the decorated method with gold data. Gold data are loaded based on the name of the decorator. Contact the development team if you are adding a tester with gold data comparison. The team will generate the gold data in the test environment and deploy them to the CI data repository.

Running tests locally

It is always strongly recommended to run CI tests locally on your own machine before pushing any changes to GitHub. To set it up, first clone the CI data repository to your own machine:

https://git.aps.anl.gov/ptycho_software/ptychography-software.git

Prior to running tests, set the environment variable PTYCHO_CI_DATA_DIR to the ci_data directory of the data repository:

export PTYCHO_CI_DATA_DIR="<path-to-data-repo>/ci_data"

This informs the testers of the location to read and write data.

Then cd into the tests directory of the main repository, and run pytest.

  • To show tester outputs to stdout, add -s.
  • To accommodate the differences in results due to varying machine precision and low-level libraries, some testers can opt to use a higher tolerance when comparing current result with the gold data. This is enabled in the GitHub Actions workflow. To do the same on your local machine, add --high-tol.

Clone this wiki locally