Feature Discussion: Turning Token Counting into Context Management

[ashu17706]

Context

This is a fantastic product and solves a real, recurring problem that every LLM practitioner faces: knowing whether a corpus fits in a context window and how much it will cost to process. I've seen many users reach for this exact tool.

I've been working in the tooling space building Smriti — a unified memory layer for AI agents — and that experience made it easy for me to identify some gaps in the token-counting workflow. This tool is clearly a strong foundation; here are some thoughts on what could make it even more powerful.

The Real Gap: Counting vs. Acting

Currently, folder-tokenizer excellently answers: "How many tokens?"

But users almost immediately follow with: "Okay, now what?"

• What files actually fit in my budget?
• Which ones are most relevant to my task?
• Can I compress the large ones?
• How do I actually prepare the context?

Potential Enhancements (not necessarily all needed at once)

1. Token Budget Mode (high-impact, low-effort)

folder-tokenizer /path --model gpt4 --budget 200000

Output: which files fit, which overflow, and a "fits in budget" subset.

2. Smart File Filtering

• Exclude patterns: --exclude "*.lock,node_modules,test*"
• Only include patterns: --only "*.py,*.ts"
• This alone would handle 80% of real-world use cases.

3. Relevance Ranking (medium-effort)

Given a query string, rank files by semantic similarity:

folder-tokenizer /path --model gpt4 --rank-by-query "authentication and OAuth"

Output: files sorted by relevance so users can cherry-pick.

4. Accurate Tokenizers for Closed Models (medium-effort)

• Use tiktoken directly for OpenAI models (vs. community HF ports)
• Use Anthropic's official tokenizer SDK for Claude (vs. reverse-engineered approximations)
• This would improve accuracy for the most popular models.

5. Compression/Summarization Integration (higher-effort)

For files over N tokens, optionally auto-summarize them before including:

folder-tokenizer /path --compress-above 5000 --llm-model gpt4

Why This Matters

Each of these features would move folder-tokenizer from a measurement tool to a context management tool — solving the full workflow users actually need.

---

About Smriti

Since you're thinking about agent tooling, you might find Smriti interesting. It's a unified memory layer for AI agents that enables context awareness across sessions:

• Ingest conversations: smriti ingest claude - Import Claude Code sessions
• Search memory: Find relevant past interactions across sessions
• Recall with synthesis: Get AI-synthesized insights from your memory
• Generate context: Auto-generate project context for future sessions

If you're interested in how agents interact with context and memory, it might be worth checking out. Happy to discuss ideas in this space.

---

This is clearly early days for the project, and these are just observations from having worked in adjacent tooling. The foundation you've built is solid!