diff --git a/docs/case-studies/issue-115/README.md b/docs/case-studies/issue-115/README.md new file mode 100644 index 0000000..ac2b4ce --- /dev/null +++ b/docs/case-studies/issue-115/README.md @@ -0,0 +1,184 @@ +# Case Study: Issue #115 - Error was treated as success + +## Summary + +The npm publish step in the CI/CD pipeline falsely reported success despite `changeset publish` failing with an E404 error. This resulted in a GitHub release being created for a version that was never actually published to npm. + +## Timeline of Events + +### January 10, 2026 - False Positive Publish + +| Time (UTC) | Event | +|------------|-------| +| 23:09:37 | Release job starts | +| 23:09:48 | `publish-to-npm.mjs` script starts | +| 23:09:49 | Git pull completes: "Already up to date" | +| 23:09:49 | Version check: "Current version to publish: 0.8.0" | +| 23:09:50 | npm view returns E404 (version not found - expected) | +| 23:09:50 | "Publish attempt 1 of 3..." | +| 23:09:50 | `npm run changeset:publish` starts | +| 23:09:51 | Changeset info: "Publishing @link-assistant/agent at 0.8.0" | +| 23:09:53 | **ERROR**: "E404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent" | +| 23:09:53 | **ERROR**: "npm error 404 Not Found" | +| 23:09:53 | **ERROR**: "packages failed to publish: @link-assistant/agent@0.8.0" | +| 23:09:53 | **FALSE POSITIVE**: "Published @link-assistant/agent@0.8.0 to npm" | +| 23:09:55 | GitHub release js-v0.8.0 created (for unpublished version) | + +### Key Evidence from Logs + +``` +šŸ¦‹ error an error occurred while publishing @link-assistant/agent: E404 Not Found +šŸ¦‹ error npm notice Access token expired or revoked. Please try logging in again. +šŸ¦‹ error packages failed to publish: +šŸ¦‹ @link-assistant/agent@0.8.0 +āœ… Published @link-assistant/agent@0.8.0 to npm +``` + +The checkmark success message appeared **immediately after** the error message, indicating no retry attempts were made. + +## Root Cause Analysis + +### Primary Root Cause: `changeset publish` exits with code 0 on failure + +The `changeset publish` command exits with code 0 even when packages fail to publish. This is a known issue in the changesets ecosystem: +- [Issue #1621](https://github.com/changesets/changesets/issues/1621): Git tag failure isn't handled +- [Issue #1280](https://github.com/changesets/changesets/issues/1280): Action succeeds but package is never published + +### Contributing Factor 1: No output validation + +The `publish-to-npm.mjs` script (lines 125-139) only relies on exception handling to detect failures: + +```javascript +try { + await $`npm run changeset:publish`; + // If no exception, assume success + setOutput('published', 'true'); + console.log(`āœ… Published ${PACKAGE_NAME}@${currentVersion} to npm`); + return; +} catch (_error) { + // Only retries if exception is thrown +} +``` + +### Contributing Factor 2: `command-stream` library behavior + +The `command-stream` library used for shell command execution does **not throw exceptions** when commands exit with non-zero codes. Verification test: + +```javascript +await $`exit 1`; +console.log("This still executes - no error thrown"); +``` + +This means even if `changeset publish` returned exit code 1, the script would still proceed to print success. + +### Contributing Factor 3: No post-publish verification + +The script does not verify that the package was actually published to npm after the publish command completes. + +## Error Chain Diagram + +``` +ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” +│ NPM Publish Failure │ +│ (Token expired/revoked - E404) │ +ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ + │ + ā–¼ +ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” +│ changeset publish outputs errors │ +│ "packages failed to publish: @link-assistant/agent" │ +ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ + │ + ā–¼ +ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” +│ changeset publish exits with code 0 │ +│ (Known changeset behavior issue) │ +ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ + │ + ā–¼ +ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” +│ publish-to-npm.mjs doesn't check output or exit code │ +│ (command-stream doesn't throw on non-zero codes) │ +ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ + │ + ā–¼ +ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” +│ Script prints success message │ +│ "āœ… Published @link-assistant/agent@0.8.0 to npm" │ +ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ + │ + ā–¼ +ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” +│ GitHub release created for unpublished version │ +│ (js-v0.8.0 released) │ +ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ +``` + +## Impact + +1. **User confusion**: A GitHub release exists for version 0.8.0 that cannot be installed from npm +2. **CI/CD trust**: The "green checkmark" on the CI run is misleading +3. **Manual intervention required**: The underlying issue (expired token) was not surfaced properly + +## Solutions + +### Immediate Fix: Verify publish success by checking npm registry + +After running `changeset publish`, verify the package is actually on npm: + +```javascript +// After publish attempt +const verifyResult = await $`npm view "${PACKAGE_NAME}@${currentVersion}" version`.run({ + capture: true, +}); + +if (verifyResult.code !== 0 || !verifyResult.stdout.includes(currentVersion)) { + throw new Error(`Verification failed: ${PACKAGE_NAME}@${currentVersion} not found on npm`); +} +``` + +### Additional Fix: Check for error patterns in output + +Capture the output of `changeset publish` and check for failure indicators: + +```javascript +const result = await $`npm run changeset:publish`.run({ capture: true }); + +if (result.stdout.includes('packages failed to publish') || + result.stderr.includes('error')) { + throw new Error('Changeset publish reported failures'); +} +``` + +### Long-term Fix: Use `.run({ capture: true })` with exit code checking + +The `command-stream` library returns exit codes when using `.run()`: + +```javascript +const result = await $`npm run changeset:publish`.run({ capture: true }); + +if (result.code !== 0) { + throw new Error(`Publish failed with exit code ${result.code}`); +} +``` + +## References + +- [GitHub Actions Run #20885793383](https://github.com/link-assistant/agent/actions/runs/20885793383/job/60008806663) +- [CI Run Log (local copy)](./ci-logs/run-20885793383.log) +- [Issue #115](https://github.com/link-assistant/agent/issues/115) +- [Changesets Issue #1621 - Git tag failure isn't handled](https://github.com/changesets/changesets/issues/1621) +- [Changesets Issue #1280 - Action succeeds but package is never published](https://github.com/changesets/changesets/issues/1280) + +## Files Affected + +| File | Issue | Fix Required | +|------|-------|--------------| +| `scripts/publish-to-npm.mjs` | No output/exit code validation | Add verification and output checking | + +## Lessons Learned + +1. **Don't trust external tool exit codes**: Tools like `changeset` may exit with 0 even on failure +2. **Verify state changes**: After any publish/deploy operation, verify the expected state change occurred +3. **Parse command output**: Check for error patterns in stdout/stderr, not just exceptions +4. **Defense in depth**: Multiple layers of validation prevent false positives diff --git a/docs/case-studies/issue-115/issue-data.json b/docs/case-studies/issue-115/issue-data.json new file mode 100644 index 0000000..b25f480 --- /dev/null +++ b/docs/case-studies/issue-115/issue-data.json @@ -0,0 +1 @@ +{"url":"https://api.github.com/repos/link-assistant/agent/issues/115","repository_url":"https://api.github.com/repos/link-assistant/agent","labels_url":"https://api.github.com/repos/link-assistant/agent/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/link-assistant/agent/issues/115/comments","events_url":"https://api.github.com/repos/link-assistant/agent/issues/115/events","html_url":"https://github.com/link-assistant/agent/issues/115","id":3800504996,"node_id":"I_kwDOQYTy3M7ihxqk","number":115,"title":"Error was treated as success","user":{"login":"konard","id":1431904,"node_id":"MDQ6VXNlcjE0MzE5MDQ=","avatar_url":"https://avatars.githubusercontent.com/u/1431904?v=4","gravatar_id":"","url":"https://api.github.com/users/konard","html_url":"https://github.com/konard","followers_url":"https://api.github.com/users/konard/followers","following_url":"https://api.github.com/users/konard/following{/other_user}","gists_url":"https://api.github.com/users/konard/gists{/gist_id}","starred_url":"https://api.github.com/users/konard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konard/subscriptions","organizations_url":"https://api.github.com/users/konard/orgs","repos_url":"https://api.github.com/users/konard/repos","events_url":"https://api.github.com/users/konard/events{/privacy}","received_events_url":"https://api.github.com/users/konard/received_events","type":"User","user_view_type":"public","site_admin":false},"labels":[{"id":9671680763,"node_id":"LA_kwDOQYTy3M8AAAACQHoi-w","url":"https://api.github.com/repos/link-assistant/agent/labels/bug","name":"bug","color":"d73a4a","default":true,"description":"Something isn't working"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2026-01-10T23:48:36Z","updated_at":"2026-01-10T23:49:05Z","closed_at":null,"author_association":"MEMBER","type":{"id":29937030,"node_id":"IT_kwDODp_H0c4ByM2G","name":"Bug","description":"An unexpected problem or behavior","color":"red","created_at":"2025-11-21T09:35:16Z","updated_at":"2025-11-21T09:35:16Z","is_enabled":true},"active_lock_reason":null,"sub_issues_summary":{"total":0,"completed":0,"percent_completed":0},"issue_dependencies_summary":{"blocked_by":0,"total_blocked_by":0,"blocking":0,"total_blocking":0},"body":"```\nDetected multi-language repository (package.json in js/)\nFrom https://github.com/link-assistant/agent\n * branch main -> FETCH_HEAD\nAlready up to date.\nCurrent version to publish: 0.8.0\nChecking if version 0.8.0 is already published...\nnpm warn Unknown user config \"always-auth\". This will stop working in the next major version of npm.\nnpm error code E404\nnpm error 404 No match found for version 0.8.0\nnpm error 404\nnpm error 404 The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it.\nnpm error 404\nnpm error 404 Note that you can also install from a\nnpm error 404 tarball, folder, http url, or git url.\nnpm error A complete log of this run can be found in: /home/runner/.npm/_logs/2026-01-10T23_09_49_736Z-debug-0.log\nVersion 0.8.0 not found on npm, proceeding with publish...\nPublish attempt 1 of 3...\nnpm warn Unknown user config \"always-auth\". This will stop working in the next major version of npm.\n\n> @link-assistant/agent@0.8.0 changeset:publish\n> changeset publish\n\nšŸ¦‹ info npm info @link-assistant/agent\nšŸ¦‹ info @link-assistant/agent is being published because our local version (0.8.0) has not been published on npm\nšŸ¦‹ info Publishing \"@link-assistant/agent\" at \"0.8.0\"\nšŸ¦‹ error an error occurred while publishing @link-assistant/agent: E404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent - Not found \nšŸ¦‹ error The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it.\nšŸ¦‹ error \nšŸ¦‹ error Note that you can also install from a\nšŸ¦‹ error tarball, folder, http url, or git url.\nšŸ¦‹ error npm warn Unknown user config \"always-auth\". This will stop working in the next major version of npm.\nšŸ¦‹ error \nšŸ¦‹ error > @link-assistant/agent@0.8.0 prepare\nšŸ¦‹ error > cd .. && husky || true\nšŸ¦‹ error \nšŸ¦‹ error npm warn publish npm auto-corrected some errors in your package.json when publishing. Please run \"npm pkg fix\" to address these errors.\nšŸ¦‹ error npm warn publish errors corrected:\nšŸ¦‹ error npm warn publish \"bin[agent]\" script name src/index.js was invalid and removed\nšŸ¦‹ error npm warn publish \"repository.url\" was normalized to \"git+https://github.com/link-assistant/agent.git\"\nšŸ¦‹ error npm notice Security Notice: Classic tokens have been revoked. Granular tokens are now limited to 90 days and require 2FA by default. Update your CI/CD workflows to avoid disruption. Learn more https://gh.io/all-npm-classic-tokens-revoked\nšŸ¦‹ error npm notice Publishing to https://registry.npmjs.org/ with tag latest and public access\nšŸ¦‹ error npm notice Access token expired or revoked. Please try logging in again.\nšŸ¦‹ error npm error code E404\nšŸ¦‹ error npm error 404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent - Not found\nšŸ¦‹ error npm error 404\nšŸ¦‹ error npm error 404 The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it.\nšŸ¦‹ error npm error 404\nšŸ¦‹ error npm error 404 Note that you can also install from a\nšŸ¦‹ error npm error 404 tarball, folder, http url, or git url.\nšŸ¦‹ error npm error A complete log of this run can be found in: /home/runner/.npm/_logs/2026-01-10T23_09_51_233Z-debug-0.log\nšŸ¦‹ error \nšŸ¦‹ error packages failed to publish:\nšŸ¦‹ @link-assistant/agent@0.8.0\nāœ… Published @link-assistant/agent@0.8.0 to npm\n```\n\nhttps://github.com/link-assistant/agent/actions/runs/20885793383/job/60008806663\n\nThat should not happen, if there is an error CI/CD should fail.\n\nI updated npmjs configuration which was the underlying cause, but errors should be treated as fail not success. So there should be no false positives.\n\nPlease download all logs and data related about the issue to this repository, make sure we compile that data to `./docs/case-studies/issue-{id}` folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), in which we will reconstruct timeline/sequence of events, find root causes of the problem, and propose possible solutions.","closed_by":null,"reactions":{"url":"https://api.github.com/repos/link-assistant/agent/issues/115/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0},"timeline_url":"https://api.github.com/repos/link-assistant/agent/issues/115/timeline","performed_via_github_app":null,"state_reason":null} \ No newline at end of file diff --git a/experiments/test-failure-detection.mjs b/experiments/test-failure-detection.mjs new file mode 100644 index 0000000..e044b5a --- /dev/null +++ b/experiments/test-failure-detection.mjs @@ -0,0 +1,120 @@ +#!/usr/bin/env node + +/** + * Test script to verify the failure detection logic works correctly + * + * This script simulates the output that changeset publish produces + * when packages fail to publish, and verifies that our detection + * logic correctly identifies the failure. + */ + +// Copy of the failure patterns from publish-to-npm.mjs +const FAILURE_PATTERNS = [ + 'packages failed to publish', + 'error occurred while publishing', + 'npm error code E', + 'npm error 404', + 'npm error 401', + 'npm error 403', + 'Access token expired', + 'ENEEDAUTH', +]; + +/** + * Check if the output contains any failure patterns + * @param {string} output - Combined stdout and stderr + * @returns {string|null} - The matched failure pattern or null if no failure detected + */ +function detectPublishFailure(output) { + const lowerOutput = output.toLowerCase(); + for (const pattern of FAILURE_PATTERNS) { + if (lowerOutput.includes(pattern.toLowerCase())) { + return pattern; + } + } + return null; +} + +// Test cases based on real CI failure output from issue #115 +const testCases = [ + { + name: 'Real changeset failure output from issue #115', + output: `šŸ¦‹ info npm info @link-assistant/agent +šŸ¦‹ info @link-assistant/agent is being published because our local version (0.8.0) has not been published on npm +šŸ¦‹ info Publishing "@link-assistant/agent" at "0.8.0" +šŸ¦‹ error an error occurred while publishing @link-assistant/agent: E404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent - Not found +šŸ¦‹ error The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it. +šŸ¦‹ error npm error code E404 +šŸ¦‹ error npm error 404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent - Not found +šŸ¦‹ error packages failed to publish: +šŸ¦‹ @link-assistant/agent@0.8.0`, + shouldFail: true, + }, + { + name: 'Access token expired error', + output: `šŸ¦‹ error npm notice Access token expired or revoked. Please try logging in again.`, + shouldFail: true, + }, + { + name: '401 Unauthorized error', + output: `npm error 401 Unauthorized - PUT https://registry.npmjs.org/@link-assistant%2fagent`, + shouldFail: true, + }, + { + name: '403 Forbidden error', + output: `npm error 403 Forbidden - PUT https://registry.npmjs.org/@link-assistant%2fagent`, + shouldFail: true, + }, + { + name: 'ENEEDAUTH error', + output: `npm error code ENEEDAUTH`, + shouldFail: true, + }, + { + name: 'Successful publish output', + output: `šŸ¦‹ info npm info @link-assistant/agent +šŸ¦‹ info @link-assistant/agent is being published because our local version (0.8.0) has not been published on npm +šŸ¦‹ info Publishing "@link-assistant/agent" at "0.8.0" +šŸ¦‹ success packages published successfully: +šŸ¦‹ @link-assistant/agent@0.8.0`, + shouldFail: false, + }, + { + name: 'No output (empty)', + output: '', + shouldFail: false, + }, +]; + +console.log('Testing failure detection logic...\n'); + +let passed = 0; +let failed = 0; + +for (const tc of testCases) { + const result = detectPublishFailure(tc.output); + const detected = result !== null; + const isCorrect = detected === tc.shouldFail; + + if (isCorrect) { + console.log(`āœ… PASS: ${tc.name}`); + if (detected) { + console.log(` Detected pattern: "${result}"`); + } + passed++; + } else { + console.log(`āŒ FAIL: ${tc.name}`); + console.log(` Expected failure: ${tc.shouldFail}, but got: ${detected}`); + if (detected) { + console.log(` Detected pattern: "${result}"`); + } + failed++; + } + console.log(); +} + +console.log(`\nResults: ${passed} passed, ${failed} failed`); + +if (failed > 0) { + process.exit(1); +} diff --git a/js/.changeset/fix-publish-false-positives.md b/js/.changeset/fix-publish-false-positives.md new file mode 100644 index 0000000..f495f86 --- /dev/null +++ b/js/.changeset/fix-publish-false-positives.md @@ -0,0 +1,7 @@ +--- +'@link-assistant/agent': patch +--- + +Add publish verification and failure detection to prevent false positives + +The npm publish script now detects failures even when changeset publish exits with code 0. This prevents the CI from falsely reporting success when packages fail to publish. diff --git a/scripts/publish-to-npm.mjs b/scripts/publish-to-npm.mjs index 9b41bc4..eb3c66e 100644 --- a/scripts/publish-to-npm.mjs +++ b/scripts/publish-to-npm.mjs @@ -58,6 +58,18 @@ const jsRoot = getJsRoot({ jsRoot: jsRootConfig, verbose: true }); const MAX_RETRIES = 3; const RETRY_DELAY = 10000; // 10 seconds +// Patterns that indicate publish failure in changeset output +const FAILURE_PATTERNS = [ + 'packages failed to publish', + 'error occurred while publishing', + 'npm error code E', + 'npm error 404', + 'npm error 401', + 'npm error 403', + 'Access token expired', + 'ENEEDAUTH', +]; + /** * Sleep for specified milliseconds * @param {number} ms @@ -66,6 +78,34 @@ function sleep(ms) { return new Promise((resolve) => globalThis.setTimeout(resolve, ms)); } +/** + * Check if the output contains any failure patterns + * @param {string} output - Combined stdout and stderr + * @returns {string|null} - The matched failure pattern or null if no failure detected + */ +function detectPublishFailure(output) { + const lowerOutput = output.toLowerCase(); + for (const pattern of FAILURE_PATTERNS) { + if (lowerOutput.includes(pattern.toLowerCase())) { + return pattern; + } + } + return null; +} + +/** + * Verify that a package version is published on npm + * @param {string} packageName + * @param {string} version + * @returns {Promise} + */ +async function verifyPublished(packageName, version) { + const result = await $`npm view "${packageName}@${version}" version`.run({ + capture: true, + }); + return result.code === 0 && result.stdout.trim().includes(version); +} + /** * Append to GitHub Actions output file * @param {string} key @@ -122,26 +162,77 @@ async function main() { // Publish to npm using OIDC trusted publishing with retry logic for (let i = 1; i <= MAX_RETRIES; i++) { console.log(`Publish attempt ${i} of ${MAX_RETRIES}...`); + let publishResult; + let lastError = null; + try { // Run changeset:publish from the js directory where package.json with this script exists + // IMPORTANT: Use .run({ capture: true }) to capture output for failure detection // IMPORTANT: cd is a virtual command that calls process.chdir(), so we restore after if (needsCd({ jsRoot })) { - await $`cd ${jsRoot} && npm run changeset:publish`; + publishResult = await $`cd ${jsRoot} && npm run changeset:publish`.run({ capture: true }); process.chdir(originalCwd); } else { - await $`npm run changeset:publish`; + publishResult = await $`npm run changeset:publish`.run({ capture: true }); } - setOutput('published', 'true'); - setOutput('published_version', currentVersion); - console.log( - `\u2705 Published ${PACKAGE_NAME}@${currentVersion} to npm` - ); - return; - } catch (_error) { + } catch (error) { // Restore cwd on error before retry if (needsCd({ jsRoot })) { process.chdir(originalCwd); } + lastError = error; + } + + // Check for failures in multiple ways: + // 1. Check if command threw an exception + // 2. Check exit code (changeset may not return non-zero, but check anyway) + // 3. Check output for failure patterns (most reliable for changeset) + // 4. Verify package is actually on npm (ultimate verification) + + const combinedOutput = publishResult + ? `${publishResult.stdout || ''}\n${publishResult.stderr || ''}` + : ''; + + // Log the output for debugging + if (combinedOutput.trim()) { + console.log('Changeset output:', combinedOutput); + } + + // Check for failure patterns in output + const failurePattern = detectPublishFailure(combinedOutput); + if (failurePattern) { + console.error(`Detected publish failure: "${failurePattern}"`); + lastError = new Error(`Publish failed: detected "${failurePattern}" in output`); + } + + // Check exit code (if available and non-zero) + if (publishResult && publishResult.code !== 0) { + console.error(`Changeset exited with code ${publishResult.code}`); + lastError = lastError || new Error(`Publish failed with exit code ${publishResult.code}`); + } + + // If no errors detected so far, verify the package is actually on npm + if (!lastError) { + console.log('Verifying package was published to npm...'); + // Wait a moment for npm registry to propagate + await sleep(2000); + const isPublished = await verifyPublished(PACKAGE_NAME, currentVersion); + + if (isPublished) { + setOutput('published', 'true'); + setOutput('published_version', currentVersion); + console.log( + `\u2705 Published ${PACKAGE_NAME}@${currentVersion} to npm` + ); + return; + } else { + console.error('Verification failed: package not found on npm after publish'); + lastError = new Error('Package not found on npm after publish attempt'); + } + } + + // If we have an error, either retry or fail + if (lastError) { if (i < MAX_RETRIES) { console.log( `Publish failed, waiting ${RETRY_DELAY / 1000}s before retry...`