Skip to content

[LLM] Add tokenizer-native token accounting and cost projection #9

@Simonsbs

Description

@Simonsbs

Goal

Strengthen token-efficiency measurement so it predicts real LLM cost/latency decisions.

Scope

  • Add provider-tokenizer-based counts (where available) alongside whitespace proxy.
  • Report prompt, completion, and total tokens per task and per pass.
  • Add cost projection mode (optional) using configurable token price table.

Acceptance

  • Reports include both proxy and tokenizer-native counts.
  • Token efficiency trends are charted in wiki.
  • Methodology doc updated with caveats.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/benchmarkBenchmark methodology and automationarea/llm-usabilityLLM generation reliability and token efficiencytype/taskImplementation task

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions