Skip to content

fix: Add publish verification and failure detection to prevent false positives #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
konard merged 4 commits into from
Jan 11, 2026
Merged

fix: Add publish verification and failure detection to prevent false positives #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8cd6c75
Initial commit with task details
konard Jan 10, 2026
fafa9c9
fix: Add publish verification and failure detection to prevent false …
konard Jan 10, 2026
4f4e6e7
chore: Add changeset for publish verification fix
konard Jan 11, 2026
0f95085
chore: Remove task config file
konard Jan 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 184 additions & 0 deletions docs/case-studies/issue-115/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Case Study: Issue #115 - Error was treated as success

## Summary

The npm publish step in the CI/CD pipeline falsely reported success despite `changeset publish` failing with an E404 error. This resulted in a GitHub release being created for a version that was never actually published to npm.

## Timeline of Events

### January 10, 2026 - False Positive Publish

| Time (UTC) | Event |
|------------|-------|
| 23:09:37 | Release job starts |
| 23:09:48 | `publish-to-npm.mjs` script starts |
| 23:09:49 | Git pull completes: "Already up to date" |
| 23:09:49 | Version check: "Current version to publish: 0.8.0" |
| 23:09:50 | npm view returns E404 (version not found - expected) |
| 23:09:50 | "Publish attempt 1 of 3..." |
| 23:09:50 | `npm run changeset:publish` starts |
| 23:09:51 | Changeset info: "Publishing @link-assistant/agent at 0.8.0" |
| 23:09:53 | **ERROR**: "E404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent" |
| 23:09:53 | **ERROR**: "npm error 404 Not Found" |
| 23:09:53 | **ERROR**: "packages failed to publish: @link-assistant/agent@0.8.0" |
| 23:09:53 | **FALSE POSITIVE**: "Published @link-assistant/agent@0.8.0 to npm" |
| 23:09:55 | GitHub release js-v0.8.0 created (for unpublished version) |

### Key Evidence from Logs

```
🦋 error an error occurred while publishing @link-assistant/agent: E404 Not Found
🦋 error npm notice Access token expired or revoked. Please try logging in again.
🦋 error packages failed to publish:
🦋 @link-assistant/agent@0.8.0
✅ Published @link-assistant/agent@0.8.0 to npm
```

The checkmark success message appeared **immediately after** the error message, indicating no retry attempts were made.

## Root Cause Analysis

### Primary Root Cause: `changeset publish` exits with code 0 on failure

The `changeset publish` command exits with code 0 even when packages fail to publish. This is a known issue in the changesets ecosystem:
- [Issue #1621](https://github.com/changesets/changesets/issues/1621): Git tag failure isn't handled
- [Issue #1280](https://github.com/changesets/changesets/issues/1280): Action succeeds but package is never published

### Contributing Factor 1: No output validation

The `publish-to-npm.mjs` script (lines 125-139) only relies on exception handling to detect failures:

```javascript
try {
await $`npm run changeset:publish`;
// If no exception, assume success
setOutput('published', 'true');
console.log(`✅ Published ${PACKAGE_NAME}@${currentVersion} to npm`);
return;
} catch (_error) {
// Only retries if exception is thrown
}
```

### Contributing Factor 2: `command-stream` library behavior

The `command-stream` library used for shell command execution does **not throw exceptions** when commands exit with non-zero codes. Verification test:

```javascript
await $`exit 1`;
console.log("This still executes - no error thrown");
```

This means even if `changeset publish` returned exit code 1, the script would still proceed to print success.

### Contributing Factor 3: No post-publish verification

The script does not verify that the package was actually published to npm after the publish command completes.

## Error Chain Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│ NPM Publish Failure │
│ (Token expired/revoked - E404) │
└────────────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ changeset publish outputs errors │
│ "packages failed to publish: @link-assistant/agent" │
└────────────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ changeset publish exits with code 0 │
│ (Known changeset behavior issue) │
└────────────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ publish-to-npm.mjs doesn't check output or exit code │
│ (command-stream doesn't throw on non-zero codes) │
└────────────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Script prints success message │
│ "✅ Published @link-assistant/agent@0.8.0 to npm" │
└────────────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ GitHub release created for unpublished version │
│ (js-v0.8.0 released) │
└─────────────────────────────────────────────────────────────────────┘
```

## Impact

1. **User confusion**: A GitHub release exists for version 0.8.0 that cannot be installed from npm
2. **CI/CD trust**: The "green checkmark" on the CI run is misleading
3. **Manual intervention required**: The underlying issue (expired token) was not surfaced properly

## Solutions

### Immediate Fix: Verify publish success by checking npm registry

After running `changeset publish`, verify the package is actually on npm:

```javascript
// After publish attempt
const verifyResult = await $`npm view "${PACKAGE_NAME}@${currentVersion}" version`.run({
capture: true,
});

if (verifyResult.code !== 0 || !verifyResult.stdout.includes(currentVersion)) {
throw new Error(`Verification failed: ${PACKAGE_NAME}@${currentVersion} not found on npm`);
}
```

### Additional Fix: Check for error patterns in output

Capture the output of `changeset publish` and check for failure indicators:

```javascript
const result = await $`npm run changeset:publish`.run({ capture: true });

if (result.stdout.includes('packages failed to publish') ||
result.stderr.includes('error')) {
throw new Error('Changeset publish reported failures');
}
```

### Long-term Fix: Use `.run({ capture: true })` with exit code checking

The `command-stream` library returns exit codes when using `.run()`:

```javascript
const result = await $`npm run changeset:publish`.run({ capture: true });

if (result.code !== 0) {
throw new Error(`Publish failed with exit code ${result.code}`);
}
```

## References

- [GitHub Actions Run #20885793383](https://github.com/link-assistant/agent/actions/runs/20885793383/job/60008806663)
- [CI Run Log (local copy)](./ci-logs/run-20885793383.log)
- [Issue #115](https://github.com/link-assistant/agent/issues/115)
- [Changesets Issue #1621 - Git tag failure isn't handled](https://github.com/changesets/changesets/issues/1621)
- [Changesets Issue #1280 - Action succeeds but package is never published](https://github.com/changesets/changesets/issues/1280)

## Files Affected

| File | Issue | Fix Required |
|------|-------|--------------|
| `scripts/publish-to-npm.mjs` | No output/exit code validation | Add verification and output checking |

## Lessons Learned

1. **Don't trust external tool exit codes**: Tools like `changeset` may exit with 0 even on failure
2. **Verify state changes**: After any publish/deploy operation, verify the expected state change occurred
3. **Parse command output**: Check for error patterns in stdout/stderr, not just exceptions
4. **Defense in depth**: Multiple layers of validation prevent false positives
1 change: 1 addition & 0 deletions docs/case-studies/issue-115/issue-data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"url":"https://api.github.com/repos/link-assistant/agent/issues/115","repository_url":"https://api.github.com/repos/link-assistant/agent","labels_url":"https://api.github.com/repos/link-assistant/agent/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/link-assistant/agent/issues/115/comments","events_url":"https://api.github.com/repos/link-assistant/agent/issues/115/events","html_url":"https://github.com/link-assistant/agent/issues/115","id":3800504996,"node_id":"I_kwDOQYTy3M7ihxqk","number":115,"title":"Error was treated as success","user":{"login":"konard","id":1431904,"node_id":"MDQ6VXNlcjE0MzE5MDQ=","avatar_url":"https://avatars.githubusercontent.com/u/1431904?v=4","gravatar_id":"","url":"https://api.github.com/users/konard","html_url":"https://github.com/konard","followers_url":"https://api.github.com/users/konard/followers","following_url":"https://api.github.com/users/konard/following{/other_user}","gists_url":"https://api.github.com/users/konard/gists{/gist_id}","starred_url":"https://api.github.com/users/konard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konard/subscriptions","organizations_url":"https://api.github.com/users/konard/orgs","repos_url":"https://api.github.com/users/konard/repos","events_url":"https://api.github.com/users/konard/events{/privacy}","received_events_url":"https://api.github.com/users/konard/received_events","type":"User","user_view_type":"public","site_admin":false},"labels":[{"id":9671680763,"node_id":"LA_kwDOQYTy3M8AAAACQHoi-w","url":"https://api.github.com/repos/link-assistant/agent/labels/bug","name":"bug","color":"d73a4a","default":true,"description":"Something isn't working"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2026-01-10T23:48:36Z","updated_at":"2026-01-10T23:49:05Z","closed_at":null,"author_association":"MEMBER","type":{"id":29937030,"node_id":"IT_kwDODp_H0c4ByM2G","name":"Bug","description":"An unexpected problem or behavior","color":"red","created_at":"2025-11-21T09:35:16Z","updated_at":"2025-11-21T09:35:16Z","is_enabled":true},"active_lock_reason":null,"sub_issues_summary":{"total":0,"completed":0,"percent_completed":0},"issue_dependencies_summary":{"blocked_by":0,"total_blocked_by":0,"blocking":0,"total_blocking":0},"body":"```\nDetected multi-language repository (package.json in js/)\nFrom https://github.com/link-assistant/agent\n * branch main -> FETCH_HEAD\nAlready up to date.\nCurrent version to publish: 0.8.0\nChecking if version 0.8.0 is already published...\nnpm warn Unknown user config \"always-auth\". This will stop working in the next major version of npm.\nnpm error code E404\nnpm error 404 No match found for version 0.8.0\nnpm error 404\nnpm error 404 The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it.\nnpm error 404\nnpm error 404 Note that you can also install from a\nnpm error 404 tarball, folder, http url, or git url.\nnpm error A complete log of this run can be found in: /home/runner/.npm/_logs/2026-01-10T23_09_49_736Z-debug-0.log\nVersion 0.8.0 not found on npm, proceeding with publish...\nPublish attempt 1 of 3...\nnpm warn Unknown user config \"always-auth\". This will stop working in the next major version of npm.\n\n> @link-assistant/agent@0.8.0 changeset:publish\n> changeset publish\n\n🦋 info npm info @link-assistant/agent\n🦋 info @link-assistant/agent is being published because our local version (0.8.0) has not been published on npm\n🦋 info Publishing \"@link-assistant/agent\" at \"0.8.0\"\n🦋 error an error occurred while publishing @link-assistant/agent: E404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent - Not found \n🦋 error The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it.\n🦋 error \n🦋 error Note that you can also install from a\n🦋 error tarball, folder, http url, or git url.\n🦋 error npm warn Unknown user config \"always-auth\". This will stop working in the next major version of npm.\n🦋 error \n🦋 error > @link-assistant/agent@0.8.0 prepare\n🦋 error > cd .. && husky || true\n🦋 error \n🦋 error npm warn publish npm auto-corrected some errors in your package.json when publishing. Please run \"npm pkg fix\" to address these errors.\n🦋 error npm warn publish errors corrected:\n🦋 error npm warn publish \"bin[agent]\" script name src/index.js was invalid and removed\n🦋 error npm warn publish \"repository.url\" was normalized to \"git+https://github.com/link-assistant/agent.git\"\n🦋 error npm notice Security Notice: Classic tokens have been revoked. Granular tokens are now limited to 90 days and require 2FA by default. Update your CI/CD workflows to avoid disruption. Learn more https://gh.io/all-npm-classic-tokens-revoked\n🦋 error npm notice Publishing to https://registry.npmjs.org/ with tag latest and public access\n🦋 error npm notice Access token expired or revoked. Please try logging in again.\n🦋 error npm error code E404\n🦋 error npm error 404 Not Found - PUT https://registry.npmjs.org/@link-assistant%2fagent - Not found\n🦋 error npm error 404\n🦋 error npm error 404 The requested resource '@link-assistant/agent@0.8.0' could not be found or you do not have permission to access it.\n🦋 error npm error 404\n🦋 error npm error 404 Note that you can also install from a\n🦋 error npm error 404 tarball, folder, http url, or git url.\n🦋 error npm error A complete log of this run can be found in: /home/runner/.npm/_logs/2026-01-10T23_09_51_233Z-debug-0.log\n🦋 error \n🦋 error packages failed to publish:\n🦋 @link-assistant/agent@0.8.0\n✅ Published @link-assistant/agent@0.8.0 to npm\n```\n\nhttps://github.com/link-assistant/agent/actions/runs/20885793383/job/60008806663\n\nThat should not happen, if there is an error CI/CD should fail.\n\nI updated npmjs configuration which was the underlying cause, but errors should be treated as fail not success. So there should be no false positives.\n\nPlease download all logs and data related about the issue to this repository, make sure we compile that data to `./docs/case-studies/issue-{id}` folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), in which we will reconstruct timeline/sequence of events, find root causes of the problem, and propose possible solutions.","closed_by":null,"reactions":{"url":"https://api.github.com/repos/link-assistant/agent/issues/115/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0},"timeline_url":"https://api.github.com/repos/link-assistant/agent/issues/115/timeline","performed_via_github_app":null,"state_reason":null}
Loading