Move recalculate analyze/upsert off the main thread

[coji]

Summary

• run recalculate analyze and upsert phases in worker threads so the web process is not blocked by synchronous better-sqlite3 work
• add a worker import resolver for tsx/Node worker execution so existing project aliases and extensionless local imports still load
• batch reviewer replacement writes to reduce per-PR transaction overhead during upsert

Validation

• ran node --import tsx batch/cli.ts recalculate aScaf_RvvFiOJWD3m8X32
• confirmed TechTalk recalculate completed successfully with 338 PRs updated

Summary by CodeRabbit

リリースノート

• New Features

  • 分析処理とアップサート処理を独立したワーカーで実行できるようになりました。
  • モジュール解決のエイリアス対応を追加しました。
  • SQLiteのロック発生時に再試行と通知を行う仕組みを導入しました。

• Refactor

  • 分析〜アップサートの処理フローをワーカー中心に切り替え、処理のタイミング計測を追加しました。
  • レビュー担当者の更新をバッチ置換に変更しました。

• Chores

  • ワーカーのビルドをビルドプロセスに統合しました。

• Tests

  • ワーカー実行設定に関する単体テストを追加しました。

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 25dbeccf-d6c9-4aa4-b250-747d3786e286

📥 Commits

Reviewing files that changed from the base of the PR and between a552c93 and 18b5a93.

📒 Files selected for processing (11)

• app/services/db.server.ts
• app/services/jobs/analyze-worker.ts
• app/services/jobs/crawl.server.ts
• app/services/jobs/path-alias-hooks.mjs
• app/services/jobs/run-in-worker.test.ts
• app/services/jobs/run-in-worker.ts
• app/services/jobs/shared-steps.server.ts
• app/services/jobs/upsert-worker.ts
• app/services/tenant-db.server.ts
• batch/db/mutations.ts
• package.json

---

📝 Walkthrough

Walkthrough

このPRはPR分析とDBアップサートをワーカースレッドへ分離する変更を導入し、パスエイリアス解決フック、ワーカー実行ユーティリティ、ワーカー本体（analyze/upsert）、ビルドスクリプト、及びレビュアーのバッチ置換を追加します。

Changes

Cohort / File(s)	Summary
Workers (implementations) app/services/jobs/analyze-worker.ts, app/services/jobs/upsert-worker.ts	新規ワーカースクリプト。analyze-workerはPR分析をワーカースレッドで実行して結果をpostMessageで返却、upsert-workerはDB向けのupsert処理をワーカースレッドで実行。
Worker runtime & hooks app/services/jobs/run-in-worker.ts, app/services/jobs/path-alias-hooks.mjs, app/services/jobs/run-in-worker.test.ts	ワーカー起動/再試行/SQLite-BUSYハンドリングを実装するランタイムAPI（runAnalyzeInWorker/runUpsertInWorker含む）と、~/等のモジュールパスエイリアス解決フックを追加。getWorkerRuntimeのユニットテストを追加。
Workflow integration app/services/jobs/shared-steps.server.ts, app/services/jobs/crawl.server.ts	既存のshared-stepsをワーカーベースに置換（分析→アップサートをワーカー経由に）。最終化でのWAL checkpointがTRUNCATE→PASSIVEに変更。処理時間計測ラッパーを導入。
DB mutations & tenant DB configs batch/db/mutations.ts, app/services/db.server.ts, app/services/tenant-db.server.ts	batchReplacePullRequestReviewersを追加してレビュアーをチャンク単位で置換するように変更。SQLite接続にbusy_timeout（5000ms）を設定。
Build & tests package.json, app/services/jobs/run-in-worker.test.ts	build:workersスクリプトを追加しメインbuildに組み込み。ワーカーRuntimeテストを追加。

Sequence Diagram

sequenceDiagram
    participant Main as Main Thread
    participant AnalyzeW as Analyze Worker
    participant PRBuilder as Store/PR Builder
    participant UpsertW as Upsert Worker
    participant DB as Database

    Main->>AnalyzeW: runAnalyzeInWorker({ organizationId, repositoryId, ... })
    activate AnalyzeW
    AnalyzeW->>PRBuilder: initialize store (org/repo)
    PRBuilder-->>AnalyzeW: store ready
    AnalyzeW->>PRBuilder: buildPullRequests(input, filterPrNumbers?)
    PRBuilder-->>AnalyzeW: { pulls, reviews, reviewers, reviewResponses }
    AnalyzeW-->>Main: postMessage(AnalyzeResult)
    deactivate AnalyzeW

    Main->>UpsertW: runUpsertInWorker({ organizationId, pulls, reviews, reviewers })
    activate UpsertW
    UpsertW->>DB: upsertAnalyzedData(orgId, data)
    DB->>DB: batchReplacePullRequestReviewers(rows)
    DB-->>UpsertW: { ok: true }
    UpsertW-->>Main: postMessage({ ok: true })
    deactivate UpsertW

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• perf: make batch analyze incremental to eliminate CPU spikes #159: filterPrNumbers の導入とそれを buildPullRequests に渡す変更 — ワーカーフローで同フィルタが使われています。
• fix: unify timestamp format to ISO 8601 for correct timezone display #194: batch/db/mutations.ts のレビュアー/アップサート周りの変更 — 本PRの batchReplacePullRequestReviewers と関連。
• feat: integrate durably for durable recalculate jobs #196: upsertAnalyzedData とレビュアー処理の改修に関する変更 — DBアップサートロジックの近接改修。

Poem

🐰 ワーカーに飛び乗り、コードの野へ、
タイルダの道をすいすい渡り、
分析は踊り、データは整う、
バッチで整えて、静かに返す、
さあビルドで出発、春風のよう！

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: moving analyze/upsert operations to worker threads instead of running them on the main thread.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch improve/worker-thread-analyze

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@app/services/jobs/run-in-worker.ts`:
- Around line 61-65: The exit handler currently ignores exit code 1
(worker.on('exit', (code) => { if (code !== 0 && code !== 1) ... })), so a
worker that exits with code 1 won't cause rejection; update the check in the
worker.on('exit') callback used in run-in-worker.ts to treat any non-zero code
as an error (e.g., use code !== 0), and ensure the reject call includes the exit
code in the Error message to preserve context.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: bdb4ffd8-dcc7-4eed-81d4-96c0d18c5ed0

📥 Commits

Reviewing files that changed from the base of the PR and between e7dcc01 and a552c93.

📒 Files selected for processing (7)

• app/services/jobs/analyze-worker.ts
• app/services/jobs/path-alias-hooks.mjs
• app/services/jobs/run-in-worker.ts
• app/services/jobs/shared-steps.server.ts
• app/services/jobs/upsert-worker.ts
• batch/db/mutations.ts
• package.json