#4: Add documentation for configuration and docker images

README.md

@@ -1,10 +1,92 @@
-# DARMA-tasking template repository
+# DARMA Workflows
 
-Template repository with base configuration.
+This repository unifies all CI/CD workflows and containers across the [DARMA-tasking](https://github.com/DARMA-tasking) organization.
+
+Jump to:
+- [Standard Workflows](standard-workflows)
+- [Standard Containers](standard-containers)
+    - [Creating Containers](creating-containers)
+    - [Using Containers](using-containers)
+
+## Standard Workflows
+
+All DARMA-tasking repositories should include the following workflows:
 
-Included workflows:
 * [*check-pr-fixes-issue*](https://github.com/DARMA-tasking/check-pr-fixes-issue) - checking if PR description contains phrase "Fixes #issue", and if PR title, description and branch mention the same issue number
 * [*find-unsigned-commits*](https://github.com/DARMA-tasking/find-unsigned-commits) - checking if there are any unsigned commits in PR
 * [*find-trailing-whitespace*](https://github.com/DARMA-tasking/find-trailing-whitespace) - checking if there are any trailing whitespaces in files
 * [*check-commit-format*](https://github.com/DARMA-tasking/check-commit-format) - checking if commit message is properly formatted - either starts with "*Merge ...*" or fullfils template: "*#issue_number: short commit description*"
 * [*action-git-diff-check*](https://github.com/joel-coffman/action-git-diff-check) - checking if changes introduce conflict markers or whitespace errors
+
+> [!TIP]
+> These workflows are included by default when a new repository is created based on [DARMA-tasking/template-repository](https://github.com/DARMA-tasking/template-repository).
+
+This repository periodically (once a month) runs a check ([`DARMA repositories check`](https://github.com/DARMA-tasking/workflows/actions/workflows/check-repositories.yml)) to locate all of the above workflows within every **non-forked** DARMA-tasking repository.
+
+Any repositories that fail this check should be corrected as soon as possible to meet the requirements for DARMA-tasking. An issue will be automatically created in any non-compliant repositories.
+
+> [!NOTE]
+> To ignore a repository during this check, add it to the `EXCLUDE` list in `ci/list_repositories.sh`.
+
+## Standard Containers
+
+This repository builds and pushes Docker containers that should be used throughout all DARMA-tasking repositories for CI.
+
+This section explains both 1) how to create a container within this repository and 2) how to use any created containers throughout the DARMA-tasking organization.
+
+### 1. Creating New Containers
+
+#### Overview
+
+All test environments are defined in `ci/config.yaml`, which also includes commented documentation.
+
+Some tools are provided as Python scripts:
+
+- `ci/build-setup.py`: Generates setup shell scripts (one shell script for each test environment)
+- `ci/build-matrix.py`: Constructs the list of available test environments as a JSON matrix file
+- `ci/build-docker-image.py`: Build a Docker image from the list of available images described in the configuration file
+    - This script includes interactive support for local builds as well as a non-interactive mode for CI
+
+#### Step By Step
+
+To create a new container, you will only need to edit the main `ci/config.yaml` file, as explained below.
+
+1. **Define**: Define the new configuration in `ci/config.yaml` by updating the following sections:
+
+    - **Setup**: Add a new configuration to the `setup` section of `ci/config.yaml`.
+    - **Image**: Add a new Docker image to the `images` section of `ci/config.yaml`, referencing the setup that you just defined.
+    - **Runner**: Add a new runner to the `runners` section of `ci/config.yaml`, referencing the docker image tag you just defined.
+
+2. **Generate**: Build the setup script and matrix file with:
+
+```sh
+python ci/build-setup.py
+python ci/build-matrix.py
+```
+
+> [!NOTE]
+> If you wish to build the new image locally, run `python ci/build-docker-image.py` and select the new image when prompted. Otherwise, the next step will build and push the image in CI.
+
+3. **Push**: The `build-docker-image.yml` workflow runs on every pull request and every push to `master`. It comprises three main steps:
+    - **Build**: Builds all Docker containers defined by the matrices in the `shared/matrix` directory.
+    - **Test**: Runs the `vt` testing suite inside every container.
+    - **Push**: If all tests pass and the workflow is running on the `master` branch, the new images are pushed to Dockerhub.
+
+### 2. Using Containers
+
+The following steps explain how to use the standardized Docker containers in the CI of other DARMA-tasking repositories.
+
+1. Copy the YAML workflow/pipeline
+
+    - **GitHub**: Copy/paste the contents of `docs/gh_build_and_test.yml` workflow into the `.github/workflows` directory of the desired repository.
+
+    - **Azure**: Copy/paste the contents of `docs/azure_build_and-test.yml` workflow into the `.azure/pipelines` directory of the desired repository.
+
+2. Update the definition of `CMD` in the "PR tests" step to reflect the correct build and test commands for the repository.
+    - Search for "PR tests" in the workflow or pipeline file to find the `CMD` definition
+
+3. Update the rest of the workflow as needed. For example, you will likely want to tailor the following to the current repository:
+    - Trigger Events
+    - Environment Variables
+
+**For Example**: The [test-ci-project](https://github.com/DARMA-tasking/test-ci-project) repository uses the common Docker containers in its CI pipelines.

ci/build-docker-image.py

@@ -10,7 +10,7 @@ class DockerBuilder:
     """Dockerfile generator class"""
 
     def build(self, args: list):
-        """Build an image using a given docker configuration fro the config file"""
+        """Build an image using a given docker configuration from the config file"""
 
         raw_config: dict = {}
         with open(os.path.dirname(__file__) + "/config.yaml", 'r', encoding="utf-8") as file:

ci/config.yaml

@@ -21,7 +21,7 @@
 
 # Dependencies:
 # The dependencies are command or scripts to run during a setup process
-# - dependencies scripts: located under the `deps` directory. Use script same (no prefix) as key
+# - dependencies scripts: located under the `deps` directory. Use script name (no prefix) as key
 # - simple commands: use `cmd` as key. It will be added as is in the setup script.
 
 # List of defaults and/or re-usable configuration nodes

docs/azure_build_and_test.yml

@@ -0,0 +1,159 @@
+name: Sample Build and Test Pipeline
+
+trigger:
+  branches:
+    include:
+      - master
+
+pr:
+  drafts: false
+  autoCancel: true
+  branches:
+    include:
+      - '*'
+
+variables:
+  WF_REPO: DARMA-tasking/workflows
+  WF_BRANCH: master
+
+jobs:
+- job: getMatrix
+  pool:
+    vmImage: 'ubuntu-latest'
+  displayName: 'Get matrix'
+  steps:
+    # Note: Filtering the test environments
+    # A jq query can be used to filter the matrix of test environments
+    # - All (default):           `matrix=$(cat azure.json | jq '.matrix')`
+    # - CLang only:              `matrix=$(cat azure.json | jq '.matrix | map(select(.label | contains("clang")))')`
+    # - gcc>=11:                 `matrix=$(cat azure.json | jq '.matrix | map(select(.name | test("gcc-[1-9][1-9]+")))')`
+    # - gcc>=11|clang>=14        `matrix=$(cat azure.json | jq '.matrix | map(select(.name | test("gcc-[1-9][1-9]+-|clang-[1-9][2-9]+")))')`
+    - bash: |
+        wget https://raw.githubusercontent.com/${{ variables.WF_REPO }}/refs/heads/${{ variables.WF_BRANCH }}/ci/shared/matrix/azure.json
+        matrix=$(cat azure.json | jq -c '.matrix')
+        echo "##vso[task.setvariable variable=matrix;isOutput=true]$matrix"
+      displayName: 'Get matrix'
+      name: getMatrixStep
+- job: run
+  displayName: 'Run'
+  dependsOn: getMatrix
+  strategy:
+    matrix: $[ dependencies.getMatrix.outputs['getMatrixStep.matrix'] ]
+  pool:
+    vmImage: $(vmImage)
+  timeoutInMinutes: 180
+  variables:
+    TS_YEAR: ~
+    TS_MONTH: ~
+    TS_DAY: ~
+    TS: ~
+    BUILD_DIR: /opt/foo/build
+    OUTPUT_DIR: /opt/foo/output
+    CCACHE_DIR: /opt/foo/build/ccache
+    COVERAGE_ENABLED: 0
+    JUNIT_REPORT_PATH: /opt/foo/output/junit.xml
+    cache_name: $(name)
+  steps:
+    - checkout: self
+      fetchDepth: 0
+    - bash: |
+        echo "Environment=$(label)"
+        echo "Runner=$(vmImage)"
+        if [ "$(image)" != "" ]
+        then
+          echo "> With Docker Image=$(image)"
+        else
+          echo "> With Runner Setup=$(setup)"
+        fi
+      displayName: Display configuration
+    - bash: |
+        if [ "$(name)" == "wf-amd64-ubuntu-22.04-gcc-12-cpp" ]
+        then
+          echo "##vso[task.setvariable variable=COVERAGE_ENABLED]1"
+        fi
+        if [[ "$(image)" == "" ]]; then
+          echo "::group::Setup in runner"
+          echo "Set setup permissions..."
+          sudo mkdir -p /opt
+          sudo chown $(whoami) /opt
+          wget -O setup.sh https://raw.githubusercontent.com/${{ variables.CI_REPO }}/refs/heads/${{ variables.CI_BRANCH }}/ci/shared/scripts/setup-$(setup).sh
+          chmod +x setup.sh
+          export WF_SETUP_ID=$(setup)
+          ./setup.sh
+          echo "::endgroup::"
+        elif [[ "$(image)" != "" ]]; then
+          echo "::group::Pull Docker image"
+          docker image pull $(image)
+          echo "::endgroup::"
+        fi
+      displayName: Set up dependencies
+    - bash: |
+        echo "##vso[task.setvariable variable=TS_YEAR]$(date -u +%Y)"
+        echo "##vso[task.setvariable variable=TS_MONTH]$(date -u +%m)"
+        echo "##vso[task.setvariable variable=TS_DAY]$(date -u +%d)"
+        echo "##vso[task.setvariable variable=TS]$(date -u +%H%M%S)"
+      displayName: Build timestamp for caching
+    - task: Cache@2
+      displayName: Update cache
+      continueOnError: true
+      inputs:
+        securityNamespace: cache
+        key: $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) | $(TS_MONTH) | $(TS_DAY) | $(TS)
+        path: $(CCACHE_DIR)
+        restoreKeys: |
+          $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) | $(TS_MONTH) | $(TS_DAY)
+          $(Agent.OS) | "$(cache_name)" | $(TS_YEAR) | $(TS_MONTH)
+          $(Agent.OS) | "$(cache_name)" | $(TS_YEAR)
+          $(Agent.OS) | "$(cache_name)"
+    - bash: |
+        mkdir -p $(CCACHE_DIR)
+        WORKSPACE=$(Pipeline.Workspace)/s
+        if [[ "$(image)" != "" ]]; then
+          WORKSPACE=/workspace
+        fi
+        CMD='
+          cd '${WORKSPACE}'; \
+          ls -l; \
+          chmod +x ./build.sh; \
+          \
+          FOO_COVERAGE_ENABLED="$(COVERAGE_ENABLED)" \
+          FOO_TEST_REPORT="$(JUNIT_REPORT_PATH)" \
+          ./build.sh'
+        echo "Running ${CMD}"
+        if [[ "$(image)" == "" ]]; then
+          bash -c "export $(cat .env | sed '/^[[:blank:]]*#/d;s/#.*//' | xargs) && $CMD";
+        else
+          docker run \
+            --name test-container \
+            --env-file .env \
+            -w $WORKSPACE \
+            -v $(Pipeline.Workspace)/s:/workspace \
+            -v $(BUILD_DIR):/opt/foo/build \
+            -v $(OUTPUT_DIR):/opt/foo/output \
+            -e CI="1" \
+            -e https_proxy="" \
+            -e http_proxy="" \
+            $(image) \
+            bash -c "$CMD"
+          exit $(docker container inspect --format '{{.State.ExitCode}}' test-container)
+        fi
+        if [ -f "$FOO_TEST_REPORT" ]; then
+          echo "##vso[task.setvariable variable=JUNIT_REPORT_PATH]${FOO_TEST_REPORT}"
+        fi
+      displayName: PR tests
+    - task: PublishTestResults@2
+      displayName: Tests ($(name))
+      inputs:
+        testResultsFormat: 'JUnit'
+        testResultsFiles: $(JUNIT_REPORT_PATH)
+
+    - task: PublishPipelineArtifact@1
+      displayName: Upload artifacts
+      continueOnError: true
+      inputs:
+        targetPath: $(OUTPUT_DIR)
+        artifact: foo-output-$(name)
+        publishLocation: 'pipeline'
+
+    # FEAT: include also coverage reporting for Azure (require cobertura or JaCoCo format)
+    # task: PublishCodeCoverageResults@1