Design post-processing recipe system for stub generator

[seanthimons]

Context

The stub generator (dev/generate_stubs.R) produces clean R wrapper functions from OpenAPI schemas. Some functions need post-processing logic beyond what the generated stub provides — dispatching across multiple stubs, annotation joins, string coercion, DTXSID extraction. Currently these are hand-written files that get overwritten when stubs are regenerated after schema changes.

We need a system that:

1. Stores post-processing logic separately from generated output
2. Automatically applies it during stub generation
3. Survives schema-driven regeneration (the whole point)
4. Is easy to add new recipes and edit existing ones
5. Works with the existing lifecycle badge protection in 05_file_scaffold.R

Current functions that need post-processing

• ct_bioactivity — dispatches to 4 generated stubs by search_type, optional annotation join via secondary API call
• ct_lists_all — projection selection logic, DTXSID comma-separated string coercion
• ct_list — uppercase coercion, DTXSID extraction + string split + dedup

More will be added as untested endpoints are validated.

Design Options

Option A: R list registry in a single file

All recipes live in one file (dev/endpoint_eval/09_recipes.R) as a named R list. Each entry contains metadata (title, lifecycle, params) and the function body as a character string. The generator reads this list and produces complete R files.

Advantages:

• Single source of truth — one file to manage all recipes
• Follows existing pipeline conventions (all dev/endpoint_eval/ modules are single R files)
• Easy to iterate the registry programmatically (validation, reporting, drift detection)
• No file discovery logic needed — just read the list

Disadvantages:

• Writing R code inside character strings — no syntax highlighting, autocomplete, or linting in IDE
• Harder to review diffs (string changes vs. real code changes)
• File grows linearly with number of recipes; complex recipes (like chemi_safety with ~100 lines) make the file unwieldy
• Syntax errors in recipe bodies are only caught at generation time, not at edit time

Option B: Separate R files per recipe

Each recipe gets its own file in dev/recipes/ (e.g., dev/recipes/ct_bioactivity.R). Each file contains a standard R function definition that the generator reads, wraps with roxygen docs, and writes to R/. Metadata (title, lifecycle, params) could be in roxygen-style comments or a companion list at the top of the file.

Advantages:

• Full IDE support — syntax highlighting, autocomplete, linting, debugging all work
• Each recipe is independently readable and reviewable
• Complex recipes stay manageable (own file, own git history)
• Easy to test recipes in isolation (source the file, call the function)
• Git blame works per-recipe

Disadvantages:

• File discovery logic needed (glob dev/recipes/*.R, parse metadata)
• Metadata format needs design (roxygen comments? A header list? A companion YAML?)
• More files to manage
• Need convention for how the generator extracts the function body vs. metadata

Option C: Marker-protected regions in R/ files

Post-processing is written directly in the generated R/ files as normal R code. Special marker comments (e.g., # <<< RECIPE START >>> / # <<< RECIPE END >>>) delineate hand-written sections. The generator preserves everything between markers during regeneration and only rewrites the generated portions.

Advantages:

• Most natural workflow — edit the actual R file you're working with
• Full IDE support with complete file context
• No separate recipe files or registries to maintain
• What you see is what you get — the file in R/ IS the source of truth

Disadvantages:

• Fragile — marker comments can be accidentally deleted, moved, or malformed
• Merges and rebases can corrupt marker boundaries
• Mixes generated and hand-written code in the same file (unclear ownership)
• Generator needs complex parsing logic to extract and re-inject protected regions
• Harder to validate — is the marker region valid? Did the generated portion change in a way that breaks the protected region?
• No clean separation between "what the schema gives us" and "what we added"

[seanthimons]

Recipe System Design — Strategy Analysis

Five strategies were researched by dedicated agents analyzing the full pipeline (dev/generate_stubs.R, 05_file_scaffold.R, 07_stub_generation.R, existing hand-written functions).

---

Strategy Comparison Matrix

Criterion	S1: Separate R Files	S2: YAML+R Hybrid	S3: Registry+Bodies	S4: Transform Chain	S5: Override Files
IDE support	Full	R: full, YAML: none	Bodies: full, registry: limited	Transforms: full, YAML: none	Full
Git friendliness	Excellent	Good	Good	Good	Excellent
Reusable patterns	Low	Low	Low	High (transform library)	None
Validation depth	Strong	Strong	Strong	Very strong (3 layers)	Basic
Testability	Source directly	Source R file	Needs wrapper helper	Transforms: unit-testable	Source directly
Pipeline changes	Moderate	Moderate	Moderate	Substantial (3 new modules)	Minimal
Complexity	Moderate	Moderate	Moderate	High	Low
Files per recipe	1	2 (YAML + R)	2 (registry entry + body)	1 YAML + N transforms	1 + MANIFEST entry
Learning curve	Low	Medium	Medium	Steep	Very low
Fragility	Low	Low (YAML gotchas)	Low	Medium (YAML + transform coupling)	Low
Scalability	Good (1 file each)	Good	Good	Excellent (shared transforms)	Fair (no composition)

---

S1: Separate R Files Per Recipe

Each recipe is a single .R file in dev/recipes/ containing both metadata (as an R list) and the function body (as a normal R function definition). The generator reads these files, extracts metadata and function body, wraps them with roxygen docs, and writes to R/.

Recipe Anatomy

recipe_metadata <- list(
  fn = "ct_bioactivity",
  lifecycle = "stable",
  title = "Bioactivity assay data",
  depends_on = c(
    "ct_bioactivity_data_search_bulk",
    "ct_bioactivity_data_search_by_aeid_bulk",
    "ct_bioactivity_data_search_by_spid_bulk",
    "ct_bioactivity_data_search_by_m4id_bulk"
  ),
  depends_on_optional = c("ct_bioactivity_assay"),
  params = list(
    list(name = "query", type = "character",
         doc = "Character vector of identifiers to query."),
    list(name = "search_type", type = "character",
         default = 'c("dtxsid", "aeid", "spid", "m4id")',
         doc = 'One of "dtxsid" (default), "aeid", "spid", or "m4id".'),
    list(name = "annotate", type = "logical", default = "FALSE",
         doc = "Logical; if TRUE, join assay annotations. Default FALSE.")
  ),
  return_doc = "A tibble of bioactivity results.",
  examples = c('ct_bioactivity(query = "DTXSID7020182")')
)

ct_bioactivity <- function(query, search_type = c("dtxsid", "aeid", "spid", "m4id"), annotate = FALSE) {
  search_type <- match.arg(search_type)
  df <- switch(search_type,
    "dtxsid" = ct_bioactivity_data_search_bulk(query = query),
    "aeid"   = ct_bioactivity_data_search_by_aeid_bulk(query = query),
    "spid"   = ct_bioactivity_data_search_by_spid_bulk(query = query),
    "m4id"   = ct_bioactivity_data_search_by_m4id_bulk(query = query)
  )
  if (annotate) {
    bioassay_all <- ct_bioactivity_assay()
    df <- dplyr::left_join(df, bioassay_all, by = "aeid")
  }
  return(df)
}

Generator Integration

New module dev/endpoint_eval/09_recipe_system.R:

• discover_recipes() — glob dev/recipes/*.R
• parse_recipe() — source file in isolated env, extract metadata + function
• validate_recipe() — check metadata completeness, dependency existence
• apply_recipes() — called after render_endpoint_stubs(), before scaffold_files()

Strengths

• Single file per recipe — metadata and code colocated
• Full IDE support — real R function with syntax highlighting, autocomplete, linting
• Directly sourceable — source("dev/recipes/ct_bioactivity.R") loads both metadata and function
• Atomic git history — one commit per recipe change
• Strong validation — dependency checking, metadata completeness, param/formal mismatch detection

Weaknesses

• Metadata is R code — slightly more verbose than YAML for declaring params
• No shared patterns — common operations (DTXSID splitting) must be reimplemented per recipe

---

S2: YAML Metadata + R Code Hybrid

Each recipe is a pair of files: a YAML file for declarative metadata and an R file containing only the executable function body.

YAML Schema

recipe_version: "1.0"
function: "ct_bioactivity"
lifecycle: "stable"
title: "Bioactivity assay data"
depends_on:
  - ct_bioactivity_data_search_bulk
  - ct_bioactivity_data_search_by_aeid_bulk
params:
  query:
    type: "character"
    description: "Character vector of identifiers to query."
  search_type:
    type: "character"
    description: 'One of "dtxsid" (default), "aeid", "spid", or "m4id".'
    default: 'c("dtxsid", "aeid", "spid", "m4id")'
returns: "A tibble of bioactivity results."

Strengths

• Clean YAML-vs-R separation
• Versioned schema (recipe_version) allows future evolution
• YAML is easy to validate programmatically

Weaknesses

• Two files per recipe that must stay in sync
• YAML gotchas (true/false as booleans, colon quoting)
• Cannot see roxygen docs while editing R code
• Requires yaml package dependency for dev tooling

---

S3: R List Registry with External Code Files

Single registry file (dev/endpoint_eval/09_recipes.R) contains all recipe metadata. Each recipe function body lives in a separate file (dev/recipes/bodies/<fn>.R).

Body files contain ONLY the function body — no function(...) { wrapper, no roxygen:

# ct_bioactivity recipe body
search_type <- match.arg(search_type)
df <- switch(search_type,
  "dtxsid" = ct_bioactivity_data_search_bulk(query = query),
  ...
)
return(df)

Strengths

• Single source of truth registry — iterate all recipes for validation/reporting
• Follows pipeline conventions (another dev/endpoint_eval/ module)

Weaknesses

• Body files cannot be sourced directly — need wrapper helper to test
• Two-location editing (metadata in registry, code in body file)
• Registry grows unwieldy with many recipes

---

S4: Composable Transform/Decorator Chain

Define composable transforms that wrap around generated stub calls. Each recipe specifies pre-transforms (input normalization), dispatch (which stubs to call), and post-transforms (result cleaning). Transforms are named, reusable R functions.

Recipe Specification (YAML)

name: ct_bioactivity
title: "Bioactivity assay data"
lifecycle: stable
transforms:
  pre:
    - transform: match_arg
      args: { x: search_type, choices: ["dtxsid", "aeid", "spid", "m4id"] }
  dispatch:
    type: switch
    on: search_type
    cases:
      dtxsid: ct_bioactivity_data_search_bulk
      aeid: ct_bioactivity_data_search_by_aeid_bulk
  post:
    - transform: conditional_transform
      condition: annotate
      if_true:
        - transform: transform_join_bioassay_annotations
          args: { join_key: "aeid" }

Dispatch Patterns

Type	Description	Example
direct_call	Single stub call	ct_lists_all calling ct_chemical_list_all
switch	Route by parameter	ct_bioactivity on search_type
sequential	Chain stubs	Resolver then query pattern
parallel	Call multiple, combine	Aggregate hazard data

Strengths

• Transform reusability — fix DTXSID splitting once, all recipes benefit
• Extensible dispatch types
• Transforms are unit-testable
• Scales best with shared patterns

Weaknesses

• High initial complexity (3 new infrastructure modules)
• YAML for R logic loses IDE support
• Understanding a function requires reading YAML + transforms + stubs
• Not all logic decomposes cleanly into pre/dispatch/post
• Transform API stability — changing a signature breaks referencing recipes

---

S5: Convention-Based Override Files

Simplest approach. Recipe files ARE plain, complete R files in dev/recipes/. The generator checks: "Does dev/recipes/ct_bioactivity.R exist? If so, copy that instead of generating a stub." A MANIFEST declares dependencies.

Override files are identical to what currently lives in R/ — full roxygen, full function definition, lifecycle badge. Migration is literally cp R/ct_bioactivity.R dev/recipes/.

Strengths

• Extreme simplicity — no DSL, no parser, no metadata extraction
• Full IDE support
• Near-zero learning curve
• Minimal pipeline changes
• Easiest migration path

Weaknesses

• No composition — each override is a complete file
• Duplication risk (files in dev/recipes/ and R/ can diverge)
• MANIFEST is manual
• Scales poorly for shared patterns

---

Implementation Effort

Strategy	New Files	Modified Files	Effort
S1: Separate R Files	1 module + N recipes	generate_stubs.R, 05_file_scaffold.R	Medium
S2: YAML+R Hybrid	1 module + 2N recipes	generate_stubs.R, 05_file_scaffold.R	Medium
S3: Registry+Bodies	1 registry + N bodies	generate_stubs.R	Medium
S4: Transform Chain	3 modules + M transforms + N recipes	generate_stubs.R, 05_file_scaffold.R	High
S5: Override Files	1 helper + MANIFEST + N overrides	generate_stubs.R	Low

---

Scaling Analysis (400+ functions)

With 400+ generated functions, even small percentages requiring recipes means real volume:

% needing recipes	Count	Impact
3% (current)	~12 recipes	Any strategy works fine
5%	~20 recipes	Pattern duplication starts to sting
10%	~40 recipes	Shared transforms become valuable
20%	~80 recipes	Need reusable composition or face massive duplication

Phased Recommendation: S1 Now, S4 Later

Phase 1 (now): Start with S1 (Separate R Files). One file per recipe, full IDE support, strong validation, low complexity. At 3-12 recipes this is the right tool.

Phase 2 (organic): As patterns repeat across recipes (DTXSID splitting appears in 5+ recipes, uppercase coercion in 8+), extract shared helpers into dev/recipes/helpers/. This is natural R refactoring — no new infrastructure. S1 accommodates this without changes.

Phase 3 (if needed, 20-30+ recipes): When helper extraction becomes unwieldy and you are maintaining 6+ shared helpers with complex interaction patterns, the formal transform library (S4) pays for itself. At that point, migrate: each S1 recipe becomes a YAML spec referencing transforms that already exist as the helpers you extracted.

The Pivot Signal

You know it is time to move from S1 to S4 when:

1. More than 5 helpers exist in dev/recipes/helpers/
2. Recipe files are mostly boilerplate calling helpers (the function body is just wiring)
3. New recipes are copy-paste of existing ones with 1-2 lines changed
4. You are maintaining the same transform logic in multiple places despite helpers

Below that threshold, S4 YAML indirection and transform API coupling cost more than they save.

---

Gap & Failure Analysis

Gaps Common to All Strategies

1. Schema description capture: The current stub generator parses schemas for endpoint structure but may not capture description text, parameter docs, or enum constraints from OpenAPI description fields. Recipes that auto-generate roxygen @param docs would benefit from schema descriptions flowing into the metadata.

2. Testing recipe outputs: No strategy addresses how to test that recipe-generated functions actually work. VCR cassettes exist for current hand-written functions, but there is no framework for validating a newly applied recipe produces a working function.

3. Recipe-stub signature drift: If a stub signature changes (new required parameter, renamed argument), recipes that depend on that stub will break silently unless validation catches the mismatch. Only S1 and S4 have strong validation for this.

4. Cheminformatics recipes: All examples use ct_* functions. The chemi_* functions use generic_chemi_request() with a different payload structure (chemicals/options wrapper). Recipes for chemi functions need to handle this.

5. Batch limit interaction: Some recipes (like ct_bioactivity) dispatch to bulk endpoints. If the recipe does not account for batch_limit settings, large queries may fail or behave differently than the underlying stubs.

Failure Modes by Strategy

S1 (Recommended):

• Metadata format drift: If recipe_metadata list structure evolves, older recipes break. Mitigation: version field in metadata, migration script.
• Orphaned recipes: Recipe file exists but the underlying stubs were removed from schema. Mitigation: validate_recipe() checks dependency existence on every run.
• Parse failure: source() in isolated env could fail if recipe references package internals. Mitigation: recipe files should only reference exported functions and standard R.

S4 (Future):

• Transform coupling: Renaming or refactoring a transform breaks every YAML referencing it. Mitigation: transform names are stable API; deprecation cycle for changes.
• YAML complexity explosion: Conditional transforms, nested dispatch, error handling — the YAML DSL grows toward a programming language. Mitigation: escape hatch to inline R code blocks.
• Debugging opacity: When a generated function fails, tracing back through YAML to transform to stub is painful. Mitigation: generated code includes comment annotations tracing each section to its recipe source.

What Could Kill Each Strategy

Strategy	Kill scenario
S1	50+ recipes with identical DTXSID-splitting boilerplate — duplication becomes unmaintainable
S2	YAML/R sync drift causes silent generation failures that pass validation but produce wrong code
S3	Body files without function wrappers confuse every new contributor — convention is too unusual
S4	Transform library becomes its own project with its own bugs, versioning, and learning curve
S5	Override files diverge from generated patterns; no validation catches the gap until runtime

---

Decision Matrix by Priority

If your top priority is...	Choose
Fastest to implement	S5 (Override Files)
Best IDE experience	S1 or S5 (tie)
Strongest validation	S4 (Transform Chain)
Simplest mental model	S5 (Override Files)
Best git workflow	S1 (Separate R Files)
Most future-proof	S1 now, S4 when pivot signals appear
Reusable across recipes	S4 (Transform Chain)
Least learning curve	S5 (Override Files)