feat: add rtk rgai command for semantic code search

[heAdz0r]

Summary

Extracted from #118 per reviewer feedback. This is the actual feature implementation — the command that #118's docs/hooks referenced but didn't include.

rtk rgai is a Rust-native semantic search that scores files by term relevance without requiring external embedding services.

Usage

rtk rgai "auth token refresh"              # multi-word semantic query
rtk rgai auth token refresh --compact      # unquoted, compact output
rtk rgai "error handler" --json            # machine-readable JSON
rtk rgai "database migration" -t rust      # filter by file type
rtk rgai "auth flow" -p ./src              # explicit path

How it works

1. Parse query → remove stop words, stem tokens
2. Walk project (gitignore-aware, skip binary/large files)
3. Score each line: term match (+1.4-1.7), phrase match (+6.0), multi-term bonus (+1.2), symbol definition boost (+2.5), comment penalty (×0.7)
4. Score each file: path relevance + top snippet scores + match density
5. Rank files, emit top N with context snippets

Token savings

• 75-90% reduction vs raw grep output (fewer files, ranked by relevance, truncated lines)
• Compact mode: 1 snippet per file, 0 context lines
• JSON mode: structured output for programmatic consumption

Changes

• src/rgai_cmd.rs: 789 lines — full search implementation with scoring, ranking, output formatting
• src/main.rs: Commands::Rgai variant, match arm, normalize_rgai_args() for backward-compat path detection

Test plan

• cargo test rgai — 8 tests pass (5 unit + 3 arg normalization)
• cargo test — 321 total tests pass
• cargo fmt --all --check — clean
• cargo clippy --all-targets — no new warnings
• Manual: rtk rgai "token tracking" -p . returns ranked results from this repo

[pszymkowiak]

stem_token("caches") → "cach", stem_token("services") → "servic", stem_token("changes") → "chang". Any word ending in -ce, -ge, -se + s loses its final e. These broken stems won't match actual occurrences in code.

[pszymkowiak]

token.contains('/') is too greedy. A query like rtk rgai "client/server architecture" will treat "client/server" as a path and silently drop it from the query.

[pszymkowiak]

starts_with('#') penalizes Markdown headers and YAML keys on top of Python comments. Can skew scoring on non-code files.

[heAdz0r]

@pszymkowiak All three issues have been addressed in commit db322a6:

1. stem_token broken stems — Removed "es" suffix rule that broke -ce/-ge/-ve words. Now:

• caches → cache (was cach)
• services → service (was servic)
• changes → change (was chang)
• Tests added: stem_token_preserves_trailing_e

2. token.contains('/') too greedy — Replaced bare contains('/') with explicit path prefixes (./, ../, /, ~/). "client/server architecture" is now correctly treated as a query, not a path.

• Test added: normalize_rgai_does_not_treat_slash_word_as_path

3. starts_with('#') penalizing Markdown/YAML — is_comment_line now accepts file extension parameter. # is only treated as a comment for scripting languages (py, sh, rb, pl, r, yaml, yml, toml, conf, ini). Markdown headers and YAML keys in other files are not penalized.

• Test added: is_comment_line_extension_aware

Ready for re-review. All tests pass, CI checks are green.

[pszymkowiak]

I'm having trouble understanding the rationale here. Why would rtk auto-install a third-party binary (grepai) via rtk init? This raises supply chain concerns — rtk shouldn't be
downloading and installing external tools on behalf of the user.

As a reminder, rtk's scope is intentionally narrow: it's a lightweight CLI proxy that compresses command output to save LLM tokens. It wraps existing commands — it doesn't implement new
search engines or manage third-party dependencies.

Looking at the bigger picture, PRs #124, #125, #127, and #136 form a chain that progressively introduces grepai into rtk. Can you explain the relationship between you and the grepai
project, and why this integration belongs in rtk rather than being a standalone tool?

[heAdz0r]

Closing — agreed with maintainers to keep grepai/rgai activity in my fork (heAdz0r/rtk) and not mix it into upstream for now.