[LLM] Add verify-guided repair loop to raise pass rates

[Simonsbs]

Goal

Raise LLM generation reliability from current 0% verify/semantic success to usable levels.

Scope

• Introduce a two-stage generation flow: initial generation + automatic verify-guided repair loop.
• Feed structured verifier errors back into subsequent attempts.
• Cap attempts and track turns-to-pass.

Target

• Verify success >=70% and semantic success >=50% on current 6-task corpus for baseline local model.

Acceptance

• New repair loop integrated into tests/llm_usability_bench.sh (or adapter layer).
• Report includes pass-at-k metrics.
• Updated docs/wiki with before/after comparison.

[Simonsbs]

Implemented core #6 work in this round:\n- added verify-guided multi-attempt repair loop in ok\n- added configurable attempts and loop controls (, )\n- added pass@k metrics (, ) and attempt metrics ()\n- expanded report outputs with per-task attempts and error class taxonomy\n\nNext: tune prompts/models to drive pass-rate improvements against the new instrumentation.

[Simonsbs]

Correction/update for this issue:

Implemented core #6 work in this round:

• Added verify-guided multi-attempt repair loop in tests/llm_usability_bench.sh.
• Added configurable loop controls: L0_LLM_MAX_ATTEMPTS and L0_LLM_ENABLE_REPAIR_LOOP.
• Added pass@k metrics: verify_pass_at_k_pct and semantic_pass_at_k_pct.
• Added attempt metrics: avg_attempts_used and per-task attempt counters.
• Expanded report outputs with per-task error class taxonomy.

Next:

• Tune prompts/models against this instrumentation to increase real cmd-mode verify/semantic rates.

[Simonsbs]

Update: I added an OpenAI benchmark adapter (tests/llm_bench/adapters/openai_l0.sh), hardened the adapter fallback path, and improved cmd-mode canonicalization repair. Latest local cmd benchmark with qwen2.5:3b improved from 0% -> 16.7% verify/semantic. Wiki/docs now include direct OpenAI benchmark commands.

[Simonsbs]

M71 reliability uplift complete: tightened adapter system prompts, added error-class + task-specific repair hints in llm_usability_bench, added keep-work-dir debug mode, and refreshed docs/wiki. New OpenAI benchmark (gpt-4.1-mini, max_attempts=3): verify 100%, semantic 100%, pass@1 50%, pass@2 100%.