Skip to content

feat: add reusable dataset generation library API while preserving CLI#16

Draft
owenqwenstarsky wants to merge 1 commit intomainfrom
codex/add-library-import-functionality
Draft

feat: add reusable dataset generation library API while preserving CLI#16
owenqwenstarsky wants to merge 1 commit intomainfrom
codex/add-library-import-functionality

Conversation

@owenqwenstarsky
Copy link
Copy Markdown
Member

Motivation

  • Provide a programmatic API so the tool can be imported as a library and used to generate datasets without the CLI while keeping the existing CLI behavior intact.

Description

  • Added exported types GenerateDatasetOptions and GenerateDatasetResult and a new generateDataset(options) function that encapsulates prompt reading, concurrency, OpenRouter calls, JSONL writing, spend tracking, and key cleanup.
  • Refactored the existing generation flow out of main() into generateDataset() and updated main() to parse CLI args/env then call the new function so CLI behavior is preserved.
  • Made stderr injectable via the options to allow library consumers to control output and progress rendering, and added a small refactor to use ensureReadableFile() inside the new function.
  • Added an end-to-end test that mocks fetch and validates JSONL output and counts, and bumped lockfile root version to match package.json (0.1.12).

Testing

  • Attempted npm test with the default test runner invocation (failed in this environment due to how the test glob is invoked), and then ran a targeted run: npm run build:test && node --test dist-test/test/datagen.test.js which executed the test suite and all tests passed (18 tests, 0 failures).
  • Ran npm run build which succeeded and produced compiled outputs.
  • Added and ran a new test generateDataset writes JSONL output for prompts that validates output structure and counts and it passed in the test run.

Codex Task

@owenqwenstarsky owenqwenstarsky marked this pull request as draft February 26, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant