Test failures send too much email

[gvanrossum]

When tests fail, a separate email is sent for each of the 3x4 matrix items. This is rather verbose, especially since every time a 3rd party (without repo write access) the entire matrix fails. It's gotten worse with the latest security tightening.

@robgruen Could we have a lighter-weight approach? Or maybe test the matrix items in a nested loop in a single action rather than making 3x4 actions for them?

[bmerkle]

there seems not to be a OOTB solution in ci.yml but we could go with the following approach:

1. Matrix jobs always succeed (or are allowed to fail)
2. Each job writes its result (pass/fail + metadata) as an artifact
3. A single follow-up job:

• downloads artifacts
• summarizes failures
• fails once

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu, windows, macos]
        python: ["3.10", "3.11", "3.12"]
    runs-on: ${{ matrix.os }}
    continue-on-error: true
    steps:
      - run: pytest || echo "failed" > result.txt
      - uses: actions/upload-artifact@v4
        with:
          name: result-${{ matrix.os }}-${{ matrix.python }}
          path: result.txt

  summarize:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
      - run: |
          failures=$(grep -R "failed" . || true)
          if [ -n "$failures" ]; then
            echo "$failures"
            exit 1
          fi`

[gvanrossum]

I think we could cut the email barrage in half by not running CI again after a PR is merged. For this we could just remove these two lines from ci.yml:

  push:
    branches: [ "main" ] 

@robgruen What do you think?

Not a full solution -- I am interested in @bmerkle's solution as well. But let's finish #125 first.

[robgruen]

I think we could cut the email barrage in half by not running CI again after a PR is merged. For this we could just remove these two lines from ci.yml:

push:
branches: [ "main" ]
@robgruen What do you think?

Not a full solution -- I am interested in @bmerkle's solution as well. But let's finish #125 first.

We certainly could. That would cut it in half. Another option would be to have a workflow chain where the first one is the perms check (or something similar) with no matrix and then have the current CI be dependent on that one. That should effectively just give us one notification per PR. Thoughts?

[gvanrossum]

The latter is what I had hoped we could do, but I don't know GH Actions well enough. Maybe @KRRT7 has a suggestion?

[robgruen]

I mean I would move the perms check and the approval gate to the "Policy Check" action, and then have the current CI action be dependent on that one. Should be straight forward to setup: https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows#nesting-reusable-workflows.

[gvanrossum]

Sure. Give it a try.

[robgruen]

[like] Robert Gruen reacted to your message:

…

________________________________ From: Guido van Rossum ***@***.***> Sent: Wednesday, December 31, 2025 7:48:55 PM To: microsoft/typeagent-py ***@***.***> Cc: Comment ***@***.***> Subject: Re: [microsoft/typeagent-py] Test failures send too much email (Issue #134) Sure. Give it a try. — Reply to this email directly, view it on GitHub<#134 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGBS6WKYUEJ5YQV6KGV2CVD4EQSCPBFHORZGSZ3HMVZKMY3SMVQXIZNMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZFAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWVDCMBVGM3DSNJQG44YFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZTONRYGQ4TCOJTHCVGC5DUOJUWE5LUMVZZJAVFOZQWY5LFUR2HE5LFURXGC3LFXN2GQ4TFMFSF64DBOJ2GSY3JOBQW45C7MFRXI2LWNF2HTAVFOZQWY5LFUR2HE5LFURXGC3LFVZ3WC5DDNBPWCY3UNF3GS5DZQKSXMYLMOVS2K2LTON2WLJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2OMRYHE2DMNBSURXGC3LFVBQWG5DPOJPWSZA>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[KRRT7]

I don't know much about GHA but at work what I did was a final "status" job that depends on all matrix jobs and reports a single pass/fail — then set that as the required check. That plus removing the push trigger for main should cut notifications way down.

[bmerkle]

if we get too much mails, the best practice would be: split execution from notification

so run all the jobs (from the matrix) and then sum up the results in one step and send out the mail.

maybe we should run the test also no on the source status but on the fully packaged pip typeagent-py library ?
so then we would have the following steps:

• make format check
• make test (Test against src/ fast feedback, developer correctness)
• create typeagent-py pip package
• deploy the package and make test (Test real usage secenario, missing init.py or broken depdencies can be catched)

[bmerkle]

and another observation:
it seems that (as an external) i am not allowed to run any unit tests (or ci at all ?)
if would be cool, if the tests could be run on my PR branch and then display the status (green or red)
using this workflow, we could avoid a lot of "build failure because of permission denied" and we would see if my PR is green and can be integrated.
This is the workflow i know from other OS projects IIRC.

[gvanrossum]

it seems that (as an external) i am not allowed to run any unit tests (or ci at all ?)

Yeah, this is exactly the problem since we (Rob) added secrets for the tests to be able to actually reach out to our Azure OpenAI instance.

The plan is to restore the old situation where the tests run without secrets. This means that about 24 tests will be skipped because they require a secret. But skipping doesn't fail so you will get a useful signal from actual failures.

We will continue to run one set of tests with secrets (my proposal is Python 3.14 on Ubuntu), and those must be triggered and approved by a project owner.

In #148 we're halfway: the tests still all require secrets, but 'make check' and 'make format' don't. We'll get there though.

[gvanrossum]

Fixed by #151.