From 3d318a4e9d0e303a0eaa89b009f8b27551c09f23 Mon Sep 17 00:00:00 2001 From: Caleb Schilly Date: Mon, 6 Jan 2025 14:54:09 -0500 Subject: [PATCH 1/5] #4: add README and sample workflows in docs folder --- README.md | 52 ++++++++++- docs/azure_build_and_test.yml | 159 ++++++++++++++++++++++++++++++++ docs/gh_build_and_test.yml | 165 ++++++++++++++++++++++++++++++++++ 3 files changed, 373 insertions(+), 3 deletions(-) create mode 100644 docs/azure_build_and_test.yml create mode 100644 docs/gh_build_and_test.yml diff --git a/README.md b/README.md index db4f9cae..bd3cc0e7 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,56 @@ -# DARMA-tasking template repository +# DARMA Workflows -Template repository with base configuration. +This repository unifies all CI/CD workflows and containers across the DARMA-tasking organization. + +Jump to: +- [Standard Workflows](standard-workflows) +- [Standard Containers](standard-containers) + +## Standard Workflows + +All DARMA-tasking repositories should include the following workflows: -Included workflows: * [*check-pr-fixes-issue*](https://github.com/DARMA-tasking/check-pr-fixes-issue) - checking if PR description contains phrase "Fixes #issue", and if PR title, description and branch mention the same issue number * [*find-unsigned-commits*](https://github.com/DARMA-tasking/find-unsigned-commits) - checking if there are any unsigned commits in PR * [*find-trailing-whitespace*](https://github.com/DARMA-tasking/find-trailing-whitespace) - checking if there are any trailing whitespaces in files * [*check-commit-format*](https://github.com/DARMA-tasking/check-commit-format) - checking if commit message is properly formatted - either starts with "*Merge ...*" or fullfils template: "*#issue_number: short commit description*" * [*action-git-diff-check*](https://github.com/joel-coffman/action-git-diff-check) - checking if changes introduce conflict markers or whitespace errors + +These workflows are included by default when a new repository is created based on [DARMA-tasking/template-repository](https://github.com/DARMA-tasking/template-repository). + +This repository periodically runs the `check_repository` action (`check-repositories.yml`), which locates all of the aforementioned workflows within every DARMA-tasking repository. +Any repositories that fail this check should be corrected as soon as possible to meet the requirements for DARMA-tasking. + +## Standard Containers + +This repository builds and pushes Docker containers that should be used throughout all DARMA-tasking repositories for CI. + +### General Overview + +All test environments are defined in `ci/config.yaml`, which also includes commented documentation. + +Some tools are provided as Python scripts: + +- `ci/build-matrix.py`: Constructs the list of available test environments as a JSON matrix file +- `ci/build-setup.py`: Generates setup shell scripts (one shell script for each test environment) +- `ci/build-docker-image.py`: Build a Docker image from the list of available images described in the configuration file + - This script includes interactive support for local builds as well as a non-interactive mode for CI + +### How To Use + +The following steps explain how to use the standardized Docker containers in the CI of other DARMA-tasking repositories. + +1. Copy the YAML workflow/pipeline + + - **GitHub**: Copy/paste the contents of `gh_build_and_test.yml` workflow from the `docs` directory into the `.github/workflows` directory of the desired repository. + + - **Azure**: Copy/paste the contents of `azure_build_and-test.yml` workflow from the `docs` directory into the `.azure/pipelines` directory of the desired repository. + +2. Update the definition of `CMD` in the "PR tests" step to reflect the correct build and test commands for the repository. + - Search for "PR tests" in the workflow or pipeline file to find the `CMD` definition + +3. Optional: You may also want to update other aspects of the file such as trigger events or selected test environments. + +### For Example... + +[PR 2](https://github.com/DARMA-tasking/test-ci-project/pull/2) from the DARMA-tasking/test-ci-project repository successfully implemented CI pipelines using the unified Docker containers. diff --git a/docs/azure_build_and_test.yml b/docs/azure_build_and_test.yml new file mode 100644 index 00000000..7660c732 --- /dev/null +++ b/docs/azure_build_and_test.yml @@ -0,0 +1,159 @@ +name: Sample Build and Test Pipeline + +trigger: + branches: + include: + - master + +pr: + drafts: false + autoCancel: true + branches: + include: + - '*' + +variables: + WF_REPO: DARMA-tasking/workflows + WF_BRANCH: master + +jobs: +- job: getMatrix + pool: + vmImage: 'ubuntu-latest' + displayName: 'Get matrix' + steps: + # Note: Filtering the test environments + # A jq query can be used to filter the matrix of test environments + # - All (default): `matrix=$(cat azure.json | jq '.matrix')` + # - CLang only: `matrix=$(cat azure.json | jq '.matrix | map(select(.label | contains("clang")))')` + # - gcc>=11: `matrix=$(cat azure.json | jq '.matrix | map(select(.name | test("gcc-[1-9][1-9]+")))')` + # - gcc>=11|clang>=14 `matrix=$(cat azure.json | jq '.matrix | map(select(.name | test("gcc-[1-9][1-9]+-|clang-[1-9][2-9]+")))')` + - bash: | + wget https://raw.githubusercontent.com/${{ variables.WF_REPO }}/refs/heads/${{ variables.WF_BRANCH }}/ci/shared/matrix/azure.json + matrix=$(cat azure.json | jq -c '.matrix') + echo "##vso[task.setvariable variable=matrix;isOutput=true]$matrix" + displayName: 'Get matrix' + name: getMatrixStep +- job: run + displayName: 'Run' + dependsOn: getMatrix + strategy: + matrix: $[ dependencies.getMatrix.outputs['getMatrixStep.matrix'] ] + pool: + vmImage: $(vmImage) + timeoutInMinutes: 180 + variables: + TS_YEAR: ~ + TS_MONTH: ~ + TS_DAY: ~ + TS: ~ + BUILD_DIR: /opt/foo/build + OUTPUT_DIR: /opt/foo/output + CCACHE_DIR: /opt/foo/build/ccache + COVERAGE_ENABLED: 0 + JUNIT_REPORT_PATH: /opt/foo/output/junit.xml + cache_name: $(name) + steps: + - checkout: self + fetchDepth: 0 + - bash: | + echo "Environment=$(label)" + echo "Runner=$(vmImage)" + if [ "$(image)" != "" ] + then + echo "> With Docker Image=$(image)" + else + echo "> With Runner Setup=$(setup)" + fi + displayName: Display configuration + - bash: | + if [ "$(name)" == "wf-amd64-ubuntu-22.04-gcc-12-cpp" ] + then + echo "##vso[task.setvariable variable=COVERAGE_ENABLED]1" + fi + if [[ "$(image)" == "" ]]; then + echo "::group::Setup in runner" + echo "Set setup permissions..." + sudo mkdir -p /opt + sudo chown $(whoami) /opt + wget -O setup.sh https://raw.githubusercontent.com/${{ variables.CI_REPO }}/refs/heads/${{ variables.CI_BRANCH }}/ci/shared/scripts/setup-$(setup).sh + chmod +x setup.sh + export WF_SETUP_ID=$(setup) + ./setup.sh + echo "::endgroup::" + elif [[ "$(image)" != "" ]]; then + echo "::group::Pull Docker image" + docker image pull $(image) + echo "::endgroup::" + fi + displayName: Set up dependencies + - bash: | + echo "##vso[task.setvariable variable=TS_YEAR]$(date -u +%Y)" + echo "##vso[task.setvariable variable=TS_MONTH]$(date -u +%m)" + echo "##vso[task.setvariable variable=TS_DAY]$(date -u +%d)" + echo "##vso[task.setvariable variable=TS]$(date -u +%H%M%S)" + displayName: Build timestamp for caching + - task: Cache@2 + displayName: Update cache + continueOnError: true + inputs: + securityNamespace: cache + key: $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) | $(TS_MONTH) | $(TS_DAY) | $(TS) + path: $(CCACHE_DIR) + restoreKeys: | + $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) | $(TS_MONTH) | $(TS_DAY) + $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) | $(TS_MONTH) + $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) + $(Agent.OS) | "$(cache_name)" + - bash: | + mkdir -p $(CCACHE_DIR) + WORKSPACE=$(Pipeline.Workspace)/s + if [[ "$(image)" != "" ]]; then + WORKSPACE=/workspace + fi + CMD=' + cd '${WORKSPACE}'; \ + ls -l; \ + chmod +x ./build.sh; \ + \ + FOO_COVERAGE_ENABLED="$(COVERAGE_ENABLED)" \ + FOO_TEST_REPORT="$(JUNIT_REPORT_PATH)" \ + ./build.sh' + echo "Running ${CMD}" + if [[ "$(image)" == "" ]]; then + bash -c "export $(cat .env | sed '/^[[:blank:]]*#/d;s/#.*//' | xargs) && $CMD"; + else + docker run \ + --name test-container \ + --env-file .env \ + -w $WORKSPACE \ + -v $(Pipeline.Workspace)/s:/workspace \ + -v $(BUILD_DIR):/opt/foo/build \ + -v $(OUTPUT_DIR):/opt/foo/output \ + -e CI="1" \ + -e https_proxy="" \ + -e http_proxy="" \ + $(image) \ + bash -c "$CMD" + exit $(docker container inspect --format '{{.State.ExitCode}}' test-container) + fi + if [ -f "$FOO_TEST_REPORT" ]; then + echo "##vso[task.setvariable variable=JUNIT_REPORT_PATH]${FOO_TEST_REPORT}" + fi + displayName: PR tests + - task: PublishTestResults@2 + displayName: Tests ($(name)) + inputs: + testResultsFormat: 'JUnit' + testResultsFiles: $(JUNIT_REPORT_PATH) + + - task: PublishPipelineArtifact@1 + displayName: Upload artifacts + continueOnError: true + inputs: + targetPath: $(OUTPUT_DIR) + artifact: foo-output-$(name) + publishLocation: 'pipeline' + + # FEAT: include also coverage reporting for Azure (require cobertura or JaCoCo format) + # task: PublishCodeCoverageResults@1 diff --git a/docs/gh_build_and_test.yml b/docs/gh_build_and_test.yml new file mode 100644 index 00000000..32bca90e --- /dev/null +++ b/docs/gh_build_and_test.yml @@ -0,0 +1,165 @@ +name: Sample Build And Test Workflow + +on: + push: + branches: + - '*' + pull_request: + branches: + - '*' + +env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + WF_REPO: DARMA-tasking/workflows + WF_BRANCH: master + +jobs: + get-matrix: + name: Get matrix + runs-on: ubuntu-latest + steps: + # Note: Filtering the test environments + # A jq query can be used to filter the matrix of test environments + # - All (default): `matrix=$(cat github.json | jq '.matrix')` + # - CLang only: `matrix=$(cat github.json | jq '.matrix | map(select(.label | contains("clang")))')` + # - gcc>=11: `matrix=$(cat github.json | jq '.matrix | map(select(.name | test("gcc-[1-9][1-9]+")))')` + # - gcc>=11|clang>=14 `matrix=$(cat github.json | jq '.matrix | map(select(.name | test("gcc-[1-9][1-9]+-|clang-[1-9][2-9]+")))')` + - name: Get matrix + id: get-matrix + run: | + wget https://raw.githubusercontent.com/${{ env.WF_REPO }}/refs/heads/${{ env.WF_BRANCH }}/ci/shared/matrix/github.json + matrix=$(cat github.json | jq '.matrix') + echo "runner=$(echo $matrix)" >> $GITHUB_OUTPUT + outputs: + matrix: ${{ steps.get-matrix.outputs.runner }} + + run: + name: Run ${{ matrix.runner.name }} + runs-on: ${{ matrix.runner.runs-on }} + needs: get-matrix + env: + TS_YEAR: ~ + TS_MONTH: ~ + TS_DAY: ~ + TS: ~ + BUILD_DIR: /opt/foo/build + OUTPUT_DIR: /opt/foo/output + CCACHE_DIR: /opt/foo/build/ccache + COVERAGE_ENABLED: ${{ matrix.runner.name == 'wf-amd64-ubuntu-22.04-gcc-12-cpp' && 1 || 0 }} + JUNIT_REPORT_PATH: /opt/foo/output/junit.xml + strategy: + fail-fast: false + matrix: + runner: ${{ fromJson(needs.get-matrix.outputs.matrix ) }} + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Display configuration + run: | + echo "Environment=${{ matrix.runner.name }}" + echo "Runner=${{ matrix.runner.runs-on }}" + if [ "${{ matrix.runner.image }}" != "" ] + then + echo "> With Docker Image=${{ matrix.runner.image }}" + else + echo "> With Runner Setup=${{ matrix.runner.setup }}" + fi + - name: Set up dependencies + run: | + if [[ "${{ matrix.runner.image }}" == "" ]]; then + echo "::group::Setup in runner" + echo "Set setup permissions..." + sudo mkdir -p /opt + sudo chown $(whoami) /opt + wget -O setup.sh https://raw.githubusercontent.com/${{ env.WF_REPO }}/refs/heads/${{ env.WF_BRANCH }}/ci/shared/scripts/setup-${{ matrix.runner.setup }}.sh + chmod +x setup.sh + export WF_SETUP_ID=${{ matrix.runner.setup }} + ./setup.sh + echo "::endgroup::" + elif [ "${{ matrix.runner.image }}" != "" ]; then + echo "::group::Pull Docker image" + docker image pull ${{ matrix.runner.image }} + echo "::endgroup::" + fi + - name: Display System Information + run: | + CMD="uname -a" + if [[ "${{ matrix.runner.image }}" == "" ]] + then + echo "Runner System Information:" + $CMD + else + echo "Docker System Information:" + docker run ${{ matrix.runner.image }} $CMD + fi + - name: Build timestamp for caching + continue-on-error: true + run: | + echo "TS_YEAR=$(date -u +%Y)" >> $GITHUB_ENV + echo "TS_MONTH=$(date -u +%m)" >> $GITHUB_ENV + echo "TS_DAY=$(date -u +%d)" >> $GITHUB_ENV + echo "TS=$(date -u +%H%M%S)" >> $GITHUB_ENV + - name: Update cache + continue-on-error: true + uses: actions/cache@v3 + env: + cache_name: ${{ matrix.runner.name }} + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ runner.os }}-${{ env.cache_name }}-${{ env.TS_YEAR }}-${{ env.TS_MONTH }}-${{ env.TS_DAY }}-${{ env.TS }} + restore-keys: | + ${{ runner.os }}-${{ env.cache_name }}-${{ env.TS_YEAR }}-${{ env.TS_MONTH }}-${{ env.TS_DAY }}- + ${{ runner.os }}-${{ env.cache_name }}-${{ env.TS_YEAR }}-${{ env.TS_MONTH }}- + ${{ runner.os }}-${{ env.cache_name }}-${{ env.TS_YEAR }}- + ${{ runner.os }}-${{ env.cache_name }}- + - name: PR tests + run: | + WORKSPACE=${{ github.workspace }} + if [[ "${{ matrix.runner.image }}" != "" ]]; then + WORKSPACE=/workspace + fi + CMD=' + cd '${WORKSPACE}'; \ + ls -l; \ + chmod +x ./build.sh; \ + \ + FOO_COVERAGE_ENABLED="${{ env.COVERAGE_ENABLED }}" \ + FOO_TEST_REPORT="${{ env.JUNIT_REPORT_PATH }}" \ + ./build.sh' + echo "Running ${CMD}" + if [[ "${{ matrix.runner.image }}" == "" ]]; then + bash -c "export $(cat .env | sed '/^[[:blank:]]*#/d;s/#.*//' | xargs) && $CMD"; + else + docker run \ + --name test-container \ + --env-file .env \ + -w $WORKSPACE \ + -v ${{ github.workspace }}:/workspace \ + -v ${{ env.BUILD_DIR }}:/opt/foo/build \ + -v ${{ env.OUTPUT_DIR }}:/opt/foo/output \ + -e CI="1" \ + -e https_proxy="" \ + -e http_proxy="" \ + ${{ matrix.runner.image }} \ + bash -c "$CMD" + exit $(docker container inspect --format '{{.State.ExitCode}}' test-container) + fi + if [ -f "$FOO_TEST_REPORT" ]; then + echo "JUNIT_REPORT_PATH=${FOO_TEST_REPORT}" >> "$GITHUB_OUTPUT" + fi + - name: Unit tests + if: (success() || failure()) && ${{ env.JUNIT_REPORT_PATH != '' }} + uses: phoenix-actions/test-reporting@v15 + with: + name: Tests report + path: ${{ env.JUNIT_REPORT_PATH }} + reporter: java-junit + output-to: step-summary + + - name: Upload artifacts + uses: actions/upload-artifact@v4 + if: always() + with: + name: foo-output-${{ matrix.runner.name }} + path: ${{ env.OUTPUT_DIR }} From 99fdc2bcd20a8eaea61a563d5eb3609b3e996ad7 Mon Sep 17 00:00:00 2001 From: Caleb Schilly Date: Tue, 7 Jan 2025 10:36:35 -0500 Subject: [PATCH 2/5] #4: improve documentation --- README.md | 44 +++++++++++++++++++++++++++++++++------- ci/build-docker-image.py | 2 +- ci/config.yaml | 2 +- 3 files changed, 39 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index bd3cc0e7..7d92aadf 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DARMA Workflows -This repository unifies all CI/CD workflows and containers across the DARMA-tasking organization. +This repository unifies all CI/CD workflows and containers across the [DARMA-tasking](https://github.com/DARMA-tasking) organization. Jump to: - [Standard Workflows](standard-workflows) @@ -16,16 +16,24 @@ All DARMA-tasking repositories should include the following workflows: * [*check-commit-format*](https://github.com/DARMA-tasking/check-commit-format) - checking if commit message is properly formatted - either starts with "*Merge ...*" or fullfils template: "*#issue_number: short commit description*" * [*action-git-diff-check*](https://github.com/joel-coffman/action-git-diff-check) - checking if changes introduce conflict markers or whitespace errors -These workflows are included by default when a new repository is created based on [DARMA-tasking/template-repository](https://github.com/DARMA-tasking/template-repository). +> [!TIP] +> These workflows are included by default when a new repository is created based on [DARMA-tasking/template-repository](https://github.com/DARMA-tasking/template-repository). -This repository periodically runs the `check_repository` action (`check-repositories.yml`), which locates all of the aforementioned workflows within every DARMA-tasking repository. +This repository periodically runs a check ([`DARMA repositories check`](https://github.com/DARMA-tasking/workflows/actions/workflows/check-repositories.yml)) to locate all of the above workflows within every DARMA-tasking repository. Any repositories that fail this check should be corrected as soon as possible to meet the requirements for DARMA-tasking. +> [!NOTE] +> To ignore a repository during this check, add it to the `EXCLUDE` list in `ci/list_repositories.sh`. + ## Standard Containers This repository builds and pushes Docker containers that should be used throughout all DARMA-tasking repositories for CI. -### General Overview +This section explains both 1) how to create a container within this repository and 2) how to use any created containers throughout the DARMA-tasking organization. + +### 1. Creating New Containers + +#### Overview All test environments are defined in `ci/config.yaml`, which also includes commented documentation. @@ -36,15 +44,37 @@ Some tools are provided as Python scripts: - `ci/build-docker-image.py`: Build a Docker image from the list of available images described in the configuration file - This script includes interactive support for local builds as well as a non-interactive mode for CI -### How To Use +#### Step By Step + +To create a new container, you only need to edit the `ci/config.yaml` file and run the `build` helper scripts above. + +1. **Setup**: Add a new configuration to the `setup` section of `ci/config.yaml`. + +2. **Image**: Add a new Docker image to the `images` section of `ci/config.yaml`, referencing the setup that you just created. + +3. **Runner**: Add a new runner to the `runners` section of `ci/config.yaml`, referencing the docker image tag you just created. + +4. **Generate**: Build the setup and matrix files with these commands: + +```sh +python ci/build-setup.py +python ci/build-matrix.py +``` + +> [!NOTE] +> If you wish to build the new image locally, run `python ci/build-docker-image.py` and select the new image when prompted. Otherwise, the next step will build and push the image in CI. + +5. **Merge**: The `build-docker-image.yml` workflow is triggered by a merge to `master`. This will 1) build all Docker containers defined by the matrices in the `shared/matrix` directory and 2) push to Dockerhub. + +### 2. Using Containers The following steps explain how to use the standardized Docker containers in the CI of other DARMA-tasking repositories. 1. Copy the YAML workflow/pipeline - - **GitHub**: Copy/paste the contents of `gh_build_and_test.yml` workflow from the `docs` directory into the `.github/workflows` directory of the desired repository. + - **GitHub**: Copy/paste the contents of `docs/gh_build_and_test.yml` workflow into the `.github/workflows` directory of the desired repository. - - **Azure**: Copy/paste the contents of `azure_build_and-test.yml` workflow from the `docs` directory into the `.azure/pipelines` directory of the desired repository. + - **Azure**: Copy/paste the contents of `docs/azure_build_and-test.yml` workflow into the `.azure/pipelines` directory of the desired repository. 2. Update the definition of `CMD` in the "PR tests" step to reflect the correct build and test commands for the repository. - Search for "PR tests" in the workflow or pipeline file to find the `CMD` definition diff --git a/ci/build-docker-image.py b/ci/build-docker-image.py index afee000f..ba93949d 100755 --- a/ci/build-docker-image.py +++ b/ci/build-docker-image.py @@ -10,7 +10,7 @@ class DockerBuilder: """Dockerfile generator class""" def build(self, args: list): - """Build an image using a given docker configuration fro the config file""" + """Build an image using a given docker configuration from the config file""" raw_config: dict = {} with open(os.path.dirname(__file__) + "/config.yaml", 'r', encoding="utf-8") as file: diff --git a/ci/config.yaml b/ci/config.yaml index 1f78df72..60077908 100644 --- a/ci/config.yaml +++ b/ci/config.yaml @@ -21,7 +21,7 @@ # Dependencies: # The dependencies are command or scripts to run during a setup process -# - dependencies scripts: located under the `deps` directory. Use script same (no prefix) as key +# - dependencies scripts: located under the `deps` directory. Use script name (no prefix) as key # - simple commands: use `cmd` as key. It will be added as is in the setup script. # List of defaults and/or re-usable configuration nodes From a5517d34aea13a472657857bb11c06a43ea21885 Mon Sep 17 00:00:00 2001 From: Caleb Schilly Date: Tue, 7 Jan 2025 11:02:45 -0500 Subject: [PATCH 3/5] #4: add to toc --- README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 7d92aadf..310f799b 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,8 @@ This repository unifies all CI/CD workflows and containers across the [DARMA-tas Jump to: - [Standard Workflows](standard-workflows) - [Standard Containers](standard-containers) + - [Creating Containers](creating-containers) + - [Using Containers](using-containers) ## Standard Workflows @@ -39,8 +41,8 @@ All test environments are defined in `ci/config.yaml`, which also includes comme Some tools are provided as Python scripts: -- `ci/build-matrix.py`: Constructs the list of available test environments as a JSON matrix file - `ci/build-setup.py`: Generates setup shell scripts (one shell script for each test environment) +- `ci/build-matrix.py`: Constructs the list of available test environments as a JSON matrix file - `ci/build-docker-image.py`: Build a Docker image from the list of available images described in the configuration file - This script includes interactive support for local builds as well as a non-interactive mode for CI @@ -50,11 +52,11 @@ To create a new container, you only need to edit the `ci/config.yaml` file and r 1. **Setup**: Add a new configuration to the `setup` section of `ci/config.yaml`. -2. **Image**: Add a new Docker image to the `images` section of `ci/config.yaml`, referencing the setup that you just created. +2. **Image**: Add a new Docker image to the `images` section of `ci/config.yaml`, referencing the setup that you just defined. -3. **Runner**: Add a new runner to the `runners` section of `ci/config.yaml`, referencing the docker image tag you just created. +3. **Runner**: Add a new runner to the `runners` section of `ci/config.yaml`, referencing the docker image tag you just defined. -4. **Generate**: Build the setup and matrix files with these commands: +4. **Generate**: Build the setup script and matrix file with: ```sh python ci/build-setup.py @@ -81,6 +83,6 @@ The following steps explain how to use the standardized Docker containers in the 3. Optional: You may also want to update other aspects of the file such as trigger events or selected test environments. -### For Example... +#### For Example... -[PR 2](https://github.com/DARMA-tasking/test-ci-project/pull/2) from the DARMA-tasking/test-ci-project repository successfully implemented CI pipelines using the unified Docker containers. +The [test-ci-project](https://github.com/DARMA-tasking/test-ci-project) repository successfully implemented CI pipelines using the common Docker containers. From 9db3ead151e7b736677b5ba67472dae96ad13f54 Mon Sep 17 00:00:00 2001 From: Caleb Schilly Date: Tue, 7 Jan 2025 12:01:08 -0500 Subject: [PATCH 4/5] #4: small fixes --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 310f799b..ff7eda31 100644 --- a/README.md +++ b/README.md @@ -81,8 +81,10 @@ The following steps explain how to use the standardized Docker containers in the 2. Update the definition of `CMD` in the "PR tests" step to reflect the correct build and test commands for the repository. - Search for "PR tests" in the workflow or pipeline file to find the `CMD` definition -3. Optional: You may also want to update other aspects of the file such as trigger events or selected test environments. +3. Update the rest of the workflow as needed. For example: + - Trigger Events + - Environment Variables #### For Example... -The [test-ci-project](https://github.com/DARMA-tasking/test-ci-project) repository successfully implemented CI pipelines using the common Docker containers. +The [test-ci-project](https://github.com/DARMA-tasking/test-ci-project) repository uses the common Docker containers in its CI pipelines. From 5925255b822616ef5967b37a4618a10575b4f9d1 Mon Sep 17 00:00:00 2001 From: Caleb Schilly Date: Wed, 8 Jan 2025 10:36:26 -0500 Subject: [PATCH 5/5] #4: add details on build-docker-image pipeline --- README.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index ff7eda31..d852c3e3 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,9 @@ All DARMA-tasking repositories should include the following workflows: > [!TIP] > These workflows are included by default when a new repository is created based on [DARMA-tasking/template-repository](https://github.com/DARMA-tasking/template-repository). -This repository periodically runs a check ([`DARMA repositories check`](https://github.com/DARMA-tasking/workflows/actions/workflows/check-repositories.yml)) to locate all of the above workflows within every DARMA-tasking repository. -Any repositories that fail this check should be corrected as soon as possible to meet the requirements for DARMA-tasking. +This repository periodically (once a month) runs a check ([`DARMA repositories check`](https://github.com/DARMA-tasking/workflows/actions/workflows/check-repositories.yml)) to locate all of the above workflows within every **non-forked** DARMA-tasking repository. + +Any repositories that fail this check should be corrected as soon as possible to meet the requirements for DARMA-tasking. An issue will be automatically created in any non-compliant repositories. > [!NOTE] > To ignore a repository during this check, add it to the `EXCLUDE` list in `ci/list_repositories.sh`. @@ -48,15 +49,15 @@ Some tools are provided as Python scripts: #### Step By Step -To create a new container, you only need to edit the `ci/config.yaml` file and run the `build` helper scripts above. - -1. **Setup**: Add a new configuration to the `setup` section of `ci/config.yaml`. +To create a new container, you will only need to edit the main `ci/config.yaml` file, as explained below. -2. **Image**: Add a new Docker image to the `images` section of `ci/config.yaml`, referencing the setup that you just defined. +1. **Define**: Define the new configuration in `ci/config.yaml` by updating the following sections: -3. **Runner**: Add a new runner to the `runners` section of `ci/config.yaml`, referencing the docker image tag you just defined. + - **Setup**: Add a new configuration to the `setup` section of `ci/config.yaml`. + - **Image**: Add a new Docker image to the `images` section of `ci/config.yaml`, referencing the setup that you just defined. + - **Runner**: Add a new runner to the `runners` section of `ci/config.yaml`, referencing the docker image tag you just defined. -4. **Generate**: Build the setup script and matrix file with: +2. **Generate**: Build the setup script and matrix file with: ```sh python ci/build-setup.py @@ -66,7 +67,10 @@ python ci/build-matrix.py > [!NOTE] > If you wish to build the new image locally, run `python ci/build-docker-image.py` and select the new image when prompted. Otherwise, the next step will build and push the image in CI. -5. **Merge**: The `build-docker-image.yml` workflow is triggered by a merge to `master`. This will 1) build all Docker containers defined by the matrices in the `shared/matrix` directory and 2) push to Dockerhub. +3. **Push**: The `build-docker-image.yml` workflow runs on every pull request and every push to `master`. It comprises three main steps: + - **Build**: Builds all Docker containers defined by the matrices in the `shared/matrix` directory. + - **Test**: Runs the `vt` testing suite inside every container. + - **Push**: If all tests pass and the workflow is running on the `master` branch, the new images are pushed to Dockerhub. ### 2. Using Containers @@ -81,10 +85,8 @@ The following steps explain how to use the standardized Docker containers in the 2. Update the definition of `CMD` in the "PR tests" step to reflect the correct build and test commands for the repository. - Search for "PR tests" in the workflow or pipeline file to find the `CMD` definition -3. Update the rest of the workflow as needed. For example: +3. Update the rest of the workflow as needed. For example, you will likely want to tailor the following to the current repository: - Trigger Events - Environment Variables -#### For Example... - -The [test-ci-project](https://github.com/DARMA-tasking/test-ci-project) repository uses the common Docker containers in its CI pipelines. +**For Example**: The [test-ci-project](https://github.com/DARMA-tasking/test-ci-project) repository uses the common Docker containers in its CI pipelines.