Devel

.github/workflows/README.md

@@ -0,0 +1,85 @@
+# GitHub Actions Workflows
+These workflows define a CI/CD pipeline that is intended to work both for the
+main repo and for forks.
+
+GitHub Actions workflows overview:
+- Run unit tests and style checks against PR's
+- Run end-to-end tests whenever a PR is merged
+- Build and publish zip/yaml files needed to spin up a TEA stack on releases
+
+In order for the end-to-end tests to run you need to set the `RUN_TESTS` secret
+in your repository secrets to `true`. However, currently these tests assume
+they are running in ASF's environment, so enabling them is probably not desired
+on forked repos, hence they are disabled by default. All other secrets are
+stored in GitHub environments.
+
+## Environments
+There are two environments used to control which AWS resources will be modified
+by GitHub Actions. The `prod` environment is used for pushing finished build
+artifacts and build status files to a public bucket. The `test` environment is
+used for deploying a test stack and running the end-to-end tests against it.
+Unless you enable the end-to-end tests with the `RUN_TESTS` repository secret,
+you don't need to configure any secrets in the `test` environment.
+
+- prod
+  - AWS Credentials (see below)
+- test
+  - AWS Credentials (see below)
+  - `URS_USERNAME` URS username used by end-to-end tests for authenticated
+    downloads
+  - `URS_PASSWORD` URS password used by end-to-end tests for authenticated
+    downloads
+  - If the config file doesn't specify a value for `URS_AUTH_CREDS_SECRET_NAME`,
+    the following secrets are also needed:
+    - `URS_CLIENT_ID`
+    - `EDL_APP_UID`
+    - `EDL_APP_PASSWORD`
+
+### Setting up AWS Credentials
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+- (optional) `AWS_ROLE_ARN`
+- (optional) `AWS_REGION`
+
+## Config file
+Unfortunately, GitHub currently doesn't support non-secret configuration
+variables. Therefore all non-secret configuration is stored in `.env`
+files located in the `config-public` directory. The file names correspond to
+the environment names that the configuraiton belongs to. After forking the repo,
+you can commit changes to these files to adjust the setup for your use case.
+
+### Build Configuration
+The build configuration is located in `prod.env` and includes the following
+options:
+
+- `CODE_BUCKET` Bucket to upload build results to. Needs to allow public ACL.
+- (optional) `CODE_PREFIX` All objects uploaded to the code bucket get
+  this prefix
+
+### Test Configuration
+The end-to-end test configuration is located in `test.env` and includes the
+following options:
+
+- `BUCKET_MAP_FILE` Name of the bucket map file to use from the config
+bucket.
+- `BUCKETNAME_PREFIX`
+- `CODE_BUCKET` Bucket to upload build results to for testing. Probably a
+  private bucket.
+- `CODE_PREFIX` All objects uploaded to the code bucket get this prefix.
+- `CONFIG_BUCKET` Bucket containing configuration files such as
+  the bucket map. Defaults to `CODE_BUCKET`
+- `COOKIE_DOMAIN`
+- `DOMAIN_CERT_ARN`
+- `DOMAIN_NAME`
+- `JWT_KEY_SECRET_NAME` Name of the AWS SecretsManager secret
+  containing the JWT public and private keys. This can be omitted and a
+  secret will be created automatically with a newly generated key pair.
+- `STACK_NAME` Name of the CloudFormation stack to update
+- `URS_AUTH_CREDS_SECRET_NAME` Name of the AWS SecretsManager
+  secret containing URS client id and client secret. This can be omitted
+  and a secret will be created automatically using the following github
+  environment secrets:
+  - `URS_CLIENT_ID`
+  - `EDL_APP_UID`
+  - `EDL_APP_PASSWORD`
+- `URS_URL` URL to use for Earthdata login.

.github/workflows/config-public/prod.env

@@ -0,0 +1,2 @@
+CODE_BUCKET=bbarton-dev
+CODE_PREFIX=thin-egress-app-prodbuild/

.github/workflows/config-public/test.env

@@ -0,0 +1,13 @@
+BUCKET_MAP_FILE=bucket_map_customheaders.yaml
+BUCKETNAME_PREFIX=rain-uw2-t-
+CODE_BUCKET=bbarton-dev
+CODE_PREFIX=thin-egress-app-benbart/
+CONFIG_BUCKET=rain-uw2-t-config
+COOKIE_DOMAIN=.asf.alaska.edu
+DOMAIN_CERT_ARN=arn:aws:acm:us-east-1:117169578524:certificate/1f3945cf-cac7-4d75-ad8d-b8920534fea2
+DOMAIN_NAME=tea-test-jenk-0.asf.alaska.edu
+JWT_KEY_SECRET_NAME=
+STACK_NAME=teatest-jenk-same
+URS_AUTH_CREDS_SECRET_NAME=URS_creds_ASF_DATA_ACCESS_EGRESS_CONTROL
+URS_URL=https://urs.earthdata.nasa.gov
+FOO=BAR

.github/workflows/lint.yml

@@ -0,0 +1,18 @@
+name: Lint
+on:
+  pull_request:
+
+jobs:
+  flake8:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+
+      - run: pip install -r requirements.txt
+
+      - uses: TrueBrain/actions-flake8@v2
+        with:
+          plugins: flake8-isort

.github/workflows/re-build.yml

@@ -0,0 +1,96 @@
+# Reusable workflow for building artifacts
+name: Build
+
+on:
+  workflow_call:
+    inputs:
+      environment:
+        required: true
+        type: string
+
+    outputs:
+      version:
+        value: ${{ jobs.variables.outputs.version }}
+      code-zip:
+        value: ${{ jobs.variables.outputs.code-zip }}
+      dependency-zip:
+        value: ${{ jobs.variables.outputs.dependency-zip }}
+      cloudformation-yaml:
+        value: ${{ jobs.variables.outputs.cloudformation-yaml }}
+      terraform-zip:
+        value: ${{ jobs.variables.outputs.terraform-zip }}
+
+
+jobs:
+  # Generate some names from the version info in the tag name
+  variables:
+    runs-on: ubuntu-latest
+    outputs:
+      version: ${{ steps.step1.outputs.version }}
+      code-zip: ${{ steps.step1.outputs.code-zip }}
+      dependency-zip: ${{ steps.step1.outputs.dependency-zip }}
+      cloudformation-yaml: ${{ steps.step1.outputs.cloudformation-yaml }}
+      terraform-zip: ${{ steps.step1.outputs.terraform-zip }}
+
+    steps:
+      - id: step1
+        run: |
+          VERSION=${GITHUB_REF_NAME#*.}
+          echo "::set-output name=version::$VERSION"
+          echo "::set-output name=code-zip::tea-code-build.$VERSION.zip"
+          echo "::set-output name=dependency-zip::tea-dependencylayer-build.$VERSION.zip"
+          echo "::set-output name=cloudformation-yaml::tea-cloudformation-build.$VERSION.yaml"
+          echo "::set-output name=terraform-zip::tea-terraform-build.$VERSION.zip"
+
+  # Build everything
+  build-all:
+    runs-on: ubuntu-latest
+    environment: ${{ inputs.environment }}
+    needs: variables
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Load environment defaults
+        run: cat .github/workflows/config-public/${{ inputs.environment }}.env >> $GITHUB_ENV
+
+      - name: Set Makefile.config
+        run: |
+          TIMESTAMP=$(TZ=America/Anchorage date)
+
+          cat > Makefile.config << EOF
+          BUILD_ID := ${{ needs.variables.outputs.version }}
+
+          CF_DEFAULT_CODE_BUCKET := ${{ env.CODE_BUCKET }}
+          CF_DEFAULT_DEPENDENCY_ARCHIVE_KEY := ${{ env.CODE_PREFIX }}${{ needs.variables.outputs.code-zip }}
+          CF_DEFAULT_CODE_ARCHIVE_KEY := ${{ env.CODE_PREFIX }}${{ needs.variables.outputs.dependency-zip }}
+          CF_BUILD_VERSION := \$(BUILD_ID)
+          CF_DESCRIPTION := TEA version ${{ needs.variables.outputs.version }} (${GITHUB_SHA:0:7}) built by GitHub Actions on $TIMESTAMP.
+          EOF
+
+      - name: Build artifacts
+        run: make build
+
+      - name: Save dependency layer zip
+        uses: actions/upload-artifact@v2
+        with:
+          name: dependency-layer
+          path: dist/thin-egress-app-dependencies.zip
+
+      - name: Save Lambda code
+        uses: actions/upload-artifact@v2
+        with:
+          name: code
+          path: dist/thin-egress-app-code.zip
+
+      - name: Save CloudFormation yaml
+        uses: actions/upload-artifact@v2
+        with:
+          name: cloudformation
+          path: dist/thin-egress-app.yaml
+
+      - name: Save Terraform zip
+        uses: actions/upload-artifact@v2
+        with:
+          name: terraform
+          path: dist/thin-egress-app-terraform.zip

.github/workflows/re-status.yml

@@ -0,0 +1,91 @@
+# Reusable workflow for reporting build status
+name: Status
+
+on:
+  workflow_call:
+    inputs:
+      environment:
+        required: true
+        type: string
+      build_tag:
+        required: true
+        type: string
+      success:
+        required: true
+        type: boolean
+
+    secrets:
+      AWS_ACCESS_KEY_ID:
+        required: true
+      AWS_SECRET_ACCESS_KEY:
+        required: true
+      # Optional
+      AWS_ROLE_ARN:
+        required: false
+      AWS_REGION:
+        required: false
+
+
+jobs:
+  report-success:
+    if: inputs.success
+    runs-on: ubuntu-latest
+    environment: ${{ inputs.environment }}
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }}
+      AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Load environment defaults
+        run: cat .github/workflows/config-public/${{ inputs.environment }}.env >> $GITHUB_ENV
+
+      - uses: actions/download-artifact@v2
+
+      - name: Upload success labels
+        run: |
+          mkdir -p buildreport
+          echo '{"schemaVersion": 1, "label": "Build Status", "message": "Success", "color": "success"}' > buildreport/buildstatus.json
+          echo '{"schemaVersion": 1, "label": "Last Successful Build", "message": "${{ inputs.build_tag }}", "color": "success"}'  > buildreport/lastgoodbuild.json
+          echo '{"schemaVersion": 1, "label": "Last Build", "message": "${{ inputs.build_tag }}", "color": "success"}' > buildreport/lastbuild.json
+          aws s3 cp buildreport/ "s3://${CODE_BUCKET}/thin-egress-app/" \
+              --recursive \
+              --metadata-directive REPLACE \
+              --cache-control no-cache \
+              --expires '2016-06-14T00:00:00Z' \
+              --content-type 'application/json' \
+              --acl public-read
+
+  report-failures:
+    if: ${{ !inputs.success }}
+    runs-on: ubuntu-latest
+    environment: ${{ inputs.environment }}
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }}
+      AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Load environment defaults
+        run: cat .github/workflows/config-public/${{ inputs.environment }}.env >> $GITHUB_ENV
+
+      - uses: actions/download-artifact@v2
+
+      - name: Upload failure labels
+        run: |
+          mkdir -p buildreport
+          echo '{"schemaVersion": 1, "label": "Build Status", "message": "Failed!", "color": "critical"}' > buildreport/buildstatus.json
+          echo '{"schemaVersion": 1, "label": "Last Build", "message": "${{ inputs.build_tag }}", "color": "critical"}' > buildreport/lastbuild.json
+          aws s3 cp buildreport/ "s3://${CODE_BUCKET}/thin-egress-app/" \
+              --recursive \
+              --metadata-directive REPLACE \
+              --cache-control no-cache \
+              --expires '2016-06-14T00:00:00Z' \
+              --content-type 'application/json' \
+              --acl public-read