Skip to content

affected.md

Latest commit

 

History

History
302 lines (228 loc) · 12.5 KB

affected.md

File metadata and controls

302 lines (228 loc) · 12.5 KB

Affected Action

This task is designed for projects in mono repos that are not fully covered by build tools similar to Make, Bazel, or Nx. It helps track the dependency graph and streamline your pipeline by identifying and executing only the steps impacted by recent changes.

Key Features

  • Dependency Graph Optimization: Generates a JSON object to identify dependencies impacted by changes, allowing you to skip unnecessary steps and focus only on what needs to be executed.
  • Commit Alignment: Aligns Git commits with images using recommended_imagetags and shas. These hashes represent the state of the dependency graph, based on defined rules, ensuring consistency across your workflow.

Recommendations:

  • Use changes for pull requests to detect and act upon specific updates.
  • Use shas for core branches like main, develop, and prod as a key for caching purposes, improving build speed and efficiency.

This approach helps optimize pipelines, reduce execution time, and maintain reliable caching across your development workflow.

jobs:
  init:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0 # fetch all history for accurate change detection
          # If you have multi-job workflow add affected task to an init step to avoid redundant checkouts.
          # If you are using path triggers the diff is limited to 300 files.
          # @see: https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#git-diff-comparisons
          # With this task you can get all the changes.

      - name: calculate affected
        id: affected
        uses: leblancmeneses/actions/apps/affected@main
        with:
          verbose: false # optional
          recommended-imagetags-tag-format: '{sha}' # optional
          recommended-imagetags-tag-format-whenchanged: ${{ github.event_name == 'pull_request' && format('pr-{0}-{1}', github.event.number, '{sha|10}') || '{sha}' }} # optional to add prefix, suffix to the image tag.
          recommended-imagetags-registry: '${{ env.ARTIFACT_REGISTRY_PUBLIC }}/' # optional; used in recommended_imagetags.
          recommended-imagetags-registry-if: | # optional; allows specifying different registry for specific targets.
            project-dbmigrations: ${{ env.ARTIFACT_REGISTRY_PRIVATE }}/;
          changed-files-output-file: '' # optional; The path to write the file containing the list of changed files.
          rules-file: '' # optional; The path to the file containing the rules if you perfer externalizing the rules for husky integration.
          rules: |
            peggy-parser: 'apps/affected/src/parser.peggy';
            peggy-parser-checkIf-incomplete: peggy-parser AND (!'apps/affected/src/parser.ts' OR !'apps/affected/src/parser.spec.ts');
              # peggy was updated but not the generated parser file or its tests.

            markdown: '**/*.md';

            third-party-deprecated: 'libs/third-party-deprecated/**';
            ui-core: 'libs/ui-core/**';
            ui-libs: ui-core third-party-deprecated;

            <project-ui>: ui-libs 'project-ui/**' EXCEPT (markdown '**/*.spec.ts');
            <project-api>: 'project-api/**' EXCEPT ('**/README.md');
            <project-dbmigrations>: './databases/project/**';

Registry Configuration

The recommended-imagetags-registry parameter sets the default registry for all recommended image tags. However, you can override this for specific targets using recommended-imagetags-registry-if. This is useful when different projects need to be pushed to different registries (e.g., public vs private registries).

The format is target-name: registry-url; where each target override is on its own line.

Rule DSL

These rules map a project name and the expression to check for changes and to generate an sha1 hash of the dependency graph.

  • The left side of the colon : is the rule key, while the right side specifies the expression to match files.
  • Rule keys with brackets <> will output a JSON object containing recommended_imagetags, shas, and changes.
  • Rule keys without brackets will output a JSON object containing shas, and changes but not recommended_imagetags.
  • Glob expressions use picomatch for matching.

Composing Rules

The project-ui rule is composed of ui-libs and project-ui's definition, enabling you to reference and combine multiple expressions. For example, project-ui runs when files change in any of these projects but excludes runs triggered by markdown or test only changes.

Expressions can combine multiple conditions using AND or OR operators. If no operator is specified, OR is used by default.

Literal Expression

Literal expressions are string-based and can be enclosed in single or double quotes. For example:

  • 'file.ts' OR "file.ts"

By default, literal expressions are case-sensitive. To make them case-insensitive, append the i flag:

  • Example: "readme.md"i will match README.md, readme.md, or rEaDme.mD.

Regex Expression

Regex expressions allow for more flexible matching and are defined using the standard JavaScript regex syntax. For example:

  • /readme\.md/i

This regex will match README.md, readme.md, or rEaDme.mD. Internally, the expression is converted to a JavaScript RegExp object, ensuring full compatibility with JavaScript’s native regex functionality.

Suffix for Literal and Regex Expressions

By default, all expressions match files regardless of their Git status code. However, you can add a suffix to the expression to filter matches based on specific Git status codes. The suffixes are A for added, M for modified, D for deleted, R for renamed, C for copied, U for unmerged, T for typechange, X for unknown, B for broken.

Usage with Literal Expressions:

  • Default behavior: 'file.ts' matches files with any Git status code.
  • With status suffix: 'file.ts':M matches only files with the "modified" status.
  • Case-insensitive matching: 'file.ts'i:A matches "added" files, ignoring case.

Usage with Regular Expressions:

  • Default behavior: /readme\.md/ matches files with any Git status code.
  • With status suffix: /readme\.md/:M matches only "modified" files.
  • Case-insensitive matching: /readme\.md/i:A matches "added" files, ignoring case.

Key Notes:

  1. Suffix Syntax: Add a colon : followed by the desired status code to filter matches.
  2. Case Insensitivity: Use the i flag before the colon to make the match case-insensitive.

Negate Expression

The ! operator is used to exclude specific files or directories from matching criteria. This ensures that certain files or directories are not modified in a pull request.

  • Example: !'dir/file.js' ensures that changes to dir/file.js are not allowed in a pull request.

Except Expression

The EXCEPT operator removes files or directories from the expression.

  markdown: '**/*.md';
  <project-ui>: 'project-ui/**' EXCEPT (markdown '**/*.spec.ts');

Wrapping up example

Assuming a changelist contains the following files:

[
  "project-ui/file1.js",
  "project-api/README.md",
]

The affected action will generate the following JSON objects:

{
  "peggy-parser": {
    "changes": false,
    "sha": "d165064e5d3e4b0a21b867fa02561e37b2cf7f01"
  },
  "peggy-parser-checkIf-incomplete": {
    "changes": false,
    "sha": "d265064e5d3e4b0a21b867fa02561e37b2cf7f01"
  },
  "markdown": {
    "changes": true,
    "sha": "d365064e5d3e4b0a21b867fa02561e37b2cf7f01"
  },
  "project-api": {
    "changes": false,
    "sha": "dd65064e5d3e4b0a21b867fa02561e37b2cf7f01",
    "recommended_imagetags": [
      "project-api:dd65064e5d3e4b0a21b867fa02561e37b2cf7f01",
      "project-api:pr-6"
    ]
  },
  "project-ui": {
    "changes": true,
    "sha": "38aabc2d6ae9866f3c1d601cba956bb935c02cf5",
    "recommended_imagetags": [
      "project-ui:38aabc2d6ae9866f3c1d601cba956bb935c02cf5",
      "project-ui:pr-6"
    ]
  },
  "project-dbmigrations": {
    "changes": false,
    "sha": "7b367954a3ca29a02e2b570112d85718e56429c9",
    "recommended_imagetags": [
      "project-dbmigrations:7b367954a3ca29a02e2b570112d85718e56429c9"
    ]
  },
  "third-party-deprecated": {
    "changes": false,
    "sha": "d465064e5d3e4b0a21b867fa02561e37b2cf7f01"
  },
  "ui-core": {
    "changes": false,
    "sha": "d565064e5d3e4b0a21b867fa02561e37b2cf7f01"
  },
  "ui-libs": {
    "changes": false,
    "sha": "d665064e5d3e4b0a21b867fa02561e37b2cf7f01"
  }
}

Consuming the JSON object

      - name: example affected output
        run: |
          echo "affected: "
          echo '${{ steps.affected.outputs.affected }}' | jq .

          # You can use env values for naming complex expressions.
          HAS_CHANGED_PROJECT_UI=$(echo '${{ steps.affected.outputs.affected }}' | jq -r '.["project-ui"].changes')
          echo "HAS_CHANGED_PROJECT_UI=$HAS_CHANGED_PROJECT_UI" >> $GITHUB_ENV

      - name: ui tests
        if: ${{ !failure() && !cancelled() && fromJson(steps.affected.outputs.affected).project-ui.changes }}
        run: npx nx run project-ui:test

Real world usage

jobs:
  vars:
    uses: ./.github/workflows/template.job.init.yml # [README.md](../README.md#recommendations-for-multi-job-pipeline)
    secrets:
      GCP_GITHUB_SERVICE_ACCOUNT: ${{secrets.GCP_GITHUB_SERVICE_ACCOUNT}}

  build-api:
    needs: [vars]
    uses: ./.github/workflows/template.job.build.yml
    if: |
      !failure() && !cancelled() && (
        inputs.MANUAL_FORCE_BUILD == 'true' || (
          fromJson(needs.vars.outputs.affected).build-api.changes == true &&
          fromJson(needs.vars.outputs.cache).build-api.cache-hit == false
        )
      )
    with:
      CACHE: ${{toJson(fromJson(needs.vars.outputs.cache).build-api)}}
      DOCKER_FILE: "./build-api/Dockerfile"
      DOCKER_BUILD_ARGS: "IS_PULL_REQUEST=${{github.event_name == 'pull_request'}}"
      DOCKER_CONTEXT: "./build-api"
      DOCKER_LABELS: ${{needs.vars.outputs.IMAGE_LABELS}}
      DOCKER_IMAGE_TAGS: ${{ fromJson(needs.vars.outputs.affected).build-api.recommended_imagetags &&
           toJson(fromJson(needs.vars.outputs.affected).build-api.recommended_imagetags) || '[]' }}
    secrets:
      GCP_GITHUB_SERVICE_ACCOUNT: ${{secrets.GCP_GITHUB_SERVICE_ACCOUNT}}

  # ...

Run locally for Husky integration

After installing Husky in your project, you can integrate the affected action.

Benefits of This Approach

  • Speed: Only runs checks on changed files, making pre-commit hooks faster.
  • Efficiency: Avoids running checks on the entire codebase unnecessarily.
  • Automation: Automatically adds fixed files back to the staging area, streamlining the commit process.

Our rule-based approach standardizes the process to identify which targets have changed, making it adaptable to diverse tech stacks and monorepo structures.

See Husky Example.

Explore the latest version tags of the built container on Docker Hub: leblancmeneses/actions-affected

Run the cli version of this tool outside of the GitHub Actions environment:

docker run --rm -v ./:/app -w /app leblancmeneses/actions-affected:v4.0.5-7a8ab79 calculate --rules-file ./.github/affected.rules > affected.json

How to debug

List all files that a specific rule would match:

docker run --rm -v ./:/app -w /app leblancmeneses/actions-affected:v4.0.5-7a8ab79 ls --rule-name=pragma --rule-name=affected --rules-file ./.github/affected.rules