TokenKit v.1.2.0

• Moved Encoders into core project for external compatibility
• Added Get engines endpoint

TokenKit v1.1.0

🏷️ TokenKit v1.1.0 Release Notes

TokenKit v1.1.0 introduces a major internal refactor and modularization of the TokenKit ecosystem.
This release splits reusable functionality into a new TokenKit.Core library — enabling clean integration with backend APIs, web services, and other .NET applications beyond the CLI.

✨ Highlights

🧩 New Core Library (TokenKit.Core)

• Provides a reusable tokenization and cost-estimation engine.
• Introduces clean abstractions via ITokenKitCore, ITokenizerEngine, ICostEstimator, and IModelRegistry.
• Includes JSON-based model registry (JsonModelRegistry) for local or embedded model data.
• Offers full test coverage and DI-friendly design.

⚙️ CLI Powered by Core

• All CLI commands (analyze, validate, models list, etc.) now execute through TokenKit.Core.
• Consistent behaviour across CLI, tests, and API integrations.
• Improved error handling and JSON output modes.

🧠 Encoders Refactored

• SimpleTextEncoder, SharpTokenEncoder, and MLTokenizersEncoder now implement Core’s ITokenizerEngine interface.
• Added internal fallback logic for unsupported models and encodings.

🧾 Expanded Test Coverage

• Full coverage for Core logic, CLI commands, encoders, and fallback branches.
• Integration tests for model registry, analyzers, and validation services.

🧩 New in TokenKit.Core

TokenKit.Core

A standalone .NET library providing the foundational components of TokenKit:

• ✅ ITokenKitCore — unified interface for analyzing and validating text.
• ✅ ITokenizerEngine — pluggable token counting engines (Simple, SharpToken, MLTokenizers).
• ✅ IModelRegistry — JSON-based model registry supporting multiple providers.
• ✅ ICostEstimator — cost calculation based on provider model pricing.
• ✅ JsonModelRegistry — reads models.data.json and supports provider filtering.

Designed for use in APIs, background services, or other projects that need
tokenization, cost estimation, and model metadata — without depending on the CLI.

📦 NuGet Packages

• TokenKit — CLI + SDK tools.
• TokenKit.Core — Core tokenization and registry engine.

# 🚀 TokenKit v1.0.0 — Official Release

🎉 Highlights

TokenKit 1.0.0 is the first full release of the .NET tokenization and AI cost estimation toolkit.

This version introduces a robust, production-ready CLI and comprehensive API coverage for tokenization, validation, cost tracking, and model registry management — all with 100% unit test coverage.

---

✨ New Features

• CLI Interface: Analyze, validate, and update model registries directly from the command line.
• Model Registry: JSON-based registry with GPT‑4o, GPT‑5, and Claude model entries.
• Tokenization Engines: Dual support for Microsoft.ML.Tokenizers and SharpToken.
• Cost Estimation: Built-in cost calculator for both input and output tokens.
• Validation Engine: Lightweight service to check token limits per model.
• Logger & ConsoleStyler: Structured logging and styled CLI output.
• Cross‑Platform: Fully compatible with Windows, macOS, and Linux.

---

🧪 Quality & Testing

• ✅ 100% code coverage across all core services.
• 🧩 Extensive unit tests for all tokenization, validation, and registry operations.
• ⚙️ CI/CD with GitHub Actions + Codecov integration.

---

🧰 Package Metadata

• NuGet: TokenKit
• GitHub: github.com/AndrewClements84/TokenKit
• License: MIT
• Authors: Andrew Clements (@AndrewClements84)

---

💡 Example Usage

# Analyze text tokenization cost
tokenkit analyze "Hello, world!" --model gpt-4o

# Validate text length against model limits
tokenkit validate "A very long input..." --model gpt-5

# List available models
tokenkit models list

---

🏁 Summary

This release marks TokenKit’s graduation from experimental stage to a fully‑featured AI developer toolkit.
It’s lightweight, dependency‑minimal, and built for both CLI users and .NET integrators.

---

Thank you to everyone who tested, refined, and contributed ideas during development.
TokenKit is now officially production‑ready! 🧠