Skip to content

feat(branch): async API surface + AsyncWorker promotion#18

Merged
lloyal-research merged 2 commits intomainfrom
feat/api-surface-shift
Feb 18, 2026
Merged

feat(branch): async API surface + AsyncWorker promotion#18
lloyal-research merged 2 commits intomainfrom
feat/api-surface-shift

Conversation

@lloyal-research
Copy link
Copy Markdown
Contributor

Breaking change: Branch and BranchStore methods that touch GPU decode
or KV mutation are now async.

Three-tier strategy:

  • Sync: pure CPU ops (produce, sample, accept, accessors, steer)
  • Async wrapper: JS async over sync native (fork, prune, pruneSubtree, retainOnly) — KV metadata ops where worker overhead > operation cost
  • True AsyncWorker: GPU decode moved to libuv thread pool (branchDecodeAndCaptureOne, branchDecodeAndCaptureBatch, storeCommit, storePrefill, decodeAndCapture, jsonSchemaToGrammar)

Branch is now an async iterable — for await (const { token, text } of branch)
generates until EOG with commit-before-yield semantics. produce/commit
remains available for multi-branch coordination.

No liblloyal changes. Callers serialize with await (same contract as
existing DecodeWorker). Removed _decodeMutex.

   Breaking change: Branch and BranchStore methods that touch GPU decode
   or KV mutation are now async.

   Three-tier strategy:
   - Sync: pure CPU ops (produce, sample, accept, accessors, steer)
   - Async wrapper: JS async over sync native (fork, prune, pruneSubtree,
     retainOnly) — KV metadata ops where worker overhead > operation cost
   - True AsyncWorker: GPU decode moved to libuv thread pool
     (branchDecodeAndCaptureOne, branchDecodeAndCaptureBatch, storeCommit,
     storePrefill, decodeAndCapture, jsonSchemaToGrammar)

   Branch is now an async iterable — for await (const { token, text } of branch)
   generates until EOG with commit-before-yield semantics. produce/commit
   remains available for multi-branch coordination.

   No liblloyal changes. Callers serialize with await (same contract as
   existing DecodeWorker). Removed _decodeMutex.
Copilot AI review requested due to automatic review settings February 18, 2026 07:21
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR converts the Branch and BranchStore APIs from synchronous to asynchronous, introducing a three-tier architecture:

  • Sync tier: Pure CPU operations (produce, sample, accept, accessors, steer)
  • Async wrapper tier: JavaScript async methods over synchronous native calls (fork, prune, pruneSubtree, retainOnly)
  • True AsyncWorker tier: GPU decode operations moved to libuv thread pool (commit, prefill, decodeAndCapture, jsonSchemaToGrammar)

The PR also introduces Branch as an async iterable, enabling simple for await loops for token generation with commit-before-yield semantics. The mutex was removed as operations are now serialized through JavaScript's await contract rather than native locking.

Changes:

  • Converted GPU decode and KV mutation methods to return Promises, offloading work to libuv thread pool via AsyncWorker classes
  • Made Branch an async iterable with commit-before-yield semantics
  • Removed _decodeMutex from SessionContext, relying on JavaScript-level serialization
  • Added comprehensive test coverage for async rejection, empty inputs, JSON schema conversion, disposal during async operations, and async iteration

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/SessionContext.cpp Added 6 AsyncWorker classes for GPU decode operations; converted decodeAndCapture, branch operations, and store operations to return Promises; removed mutex
src/SessionContext.hpp Removed _decodeMutex member and include; updated comment for decodeAndCapture
lib/Branch.js Converted fork, prune, pruneSubtree, commit, prefill, decodeAndCaptureOne to async; implemented Symbol.asyncIterator; updated documentation
lib/BranchStore.js Converted commit, prefill, retainOnly to async methods
lib/index.d.ts Updated type signatures for all async methods; added Symbol.asyncIterator type; updated JSDoc examples
test/integration.js Added await keywords throughout; added 5 new test functions for async features (rejection, empty inputs, JSON schema, disposal, iteration)
examples/*.mjs Updated all examples to use await with Branch/BranchStore methods; simplified chat example using async iterator
README.md Updated examples to show async API usage; added async iterator example

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@lloyal-research lloyal-research merged commit efd6a27 into main Feb 18, 2026
7 checks passed
@lloyal-research lloyal-research deleted the feat/api-surface-shift branch February 20, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants