Skip to content

Implement guided generation for CoreML#116

Draft
mattt wants to merge 10 commits intomainfrom
mattt/coreml-guided-generation
Draft

Implement guided generation for CoreML#116
mattt wants to merge 10 commits intomainfrom
mattt/coreml-guided-generation

Conversation

@mattt
Copy link
Owner

@mattt mattt commented Feb 7, 2026

Related to #27

@mattt
Copy link
Owner Author

mattt commented Feb 7, 2026

Currently one failing test:

❌ Test structuredGenerationSimpleInt() recorded an issue at CoreMLLanguageModelTests.swift:194:10: Caught error: numberOutOfRange("Invalid integer: <unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>")

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds guided/structured JSON generation support for the CoreML backend by wiring CoreML into the existing ConstrainedJSONGenerator infrastructure (token-level constrained sampling), and updates structured generation to be async-compatible across backends.

Changes:

  • Made TokenBackend.decode/sample and ConstrainedJSONGenerator.generate() async, updating all call sites and tests accordingly.
  • Implemented CoreML structured generation via a new CoreML token backend that applies logits processors and masks logits to the schema-allowed token set.
  • Added CoreML structured generation tests and adjusted CoreML test suite execution to run serially.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
Tests/AnyLanguageModelTests/StructuredGenerationTests.swift Updates structured generation tests to use async generate().
Tests/AnyLanguageModelTests/Shared/MockTokenBackend.swift Updates mock backend to conform to async TokenBackend API.
Tests/AnyLanguageModelTests/CoreMLLanguageModelTests.swift Serializes CoreML suite, caches model download/compile in a task, and adds structured generation tests.
Sources/AnyLanguageModel/StructuredGeneration.swift Converts constrained generation to async and skips special tokens for numeric token sets.
Sources/AnyLanguageModel/Models/MLXLanguageModel.swift Adapts structured generation path to async generate(), updates backend methods.
Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift Adapts structured generation to async generate() and updates backend methods (currently broken due to pointer lifetime issue).
Sources/AnyLanguageModel/Models/CoreMLLanguageModel.swift Adds structured generation for CoreML (schema prompting + constrained sampling backend) and adjusts string generation output handling.
Package.resolved Adds/updates Swift package dependencies used by the CoreML implementation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant