[EXAMPLE] How it all works? Reference architecture?

[buger]

I'm keeping this ticket, just to show probe integration with Github actions, and how cool answers it generate :)

You can add it to own repository by adding following github action: https://github.com/buger/probe/blob/main/.github/workflows/probe-issues.yml

[github-actions]

Probe Reference Architecture

Probe is a code search and analysis tool that combines efficient text search with code intelligence to find relevant code in your codebase. Here's a comprehensive overview of how it works and its architecture.

System Architecture Overview

Probe operates through six main stages:

1. Scan: File identification using ripgrep
2. Parse: Code structure analysis via Abstract Syntax Trees (AST)
3. Process: Query enhancement with NLP techniques
4. Rank: Result prioritization
5. Extract: Code block isolation
6. Format: Output generation

Core Components

The system consists of three primary functional components:

1. Search Component

The search component is responsible for finding code based on text patterns. It uses a combination of regex-based search and AST-based filtering to provide accurate results.

Key modules:

• search_runner.rs: Orchestrates the search process
• file_processing.rs: Processes files and extracts code blocks
• elastic_query.rs: Parses and evaluates complex search queries
• result_ranking.rs: Ranks search results based on relevance
• block_merging.rs: Merges related code blocks
• cache.rs: Caches search results for better performance
• tokenization.rs: Handles query tokenization and processing

2. Extract Component

The extract component is responsible for extracting code blocks from files based on file paths and optional line numbers or symbol names.

Key modules:

• file_paths.rs: Parses file paths with line numbers or symbols
• processor.rs: Processes files and extracts code blocks
• formatter.rs: Formats extraction results
• symbol_finder.rs: Finds symbols (functions, classes, etc.) in code

3. Query Component

The query component uses ast-grep for structural pattern matching, allowing users to find specific code structures like functions, classes, or methods.

Key modules:

• query.rs: Handles AST-based pattern matching
• Integration with ast-grep-core for AST pattern matching

Language Support System

Probe's language awareness is a key feature, enabling it to understand the structure of code in different programming languages.

Key components:

• language/mod.rs: Core language module
• language_impl trait: Common interface for all language implementations
• Tree-sitter integration for AST parsing
• Language-specific implementations for each supported language

Integration Architecture

Probe connects with other tools through several interfaces:

MCP Server

• STDIO Transport for communication
• JSON Protocol for structured messages
• Tool definitions for search, query, and extract capabilities

Node.js SDK

• JavaScript bindings to core functionality
• Vercel AI SDK integration
• Promise-based API
• TypeScript support

Web Interface

• Express backend
• Vanilla JS frontend
• Streaming responses
• Markdown rendering

Data Flow

Here's how data flows through the system:

graph TD
    A[User Input] --> B[CLI Parser]
    B --> C{Command Type}
    C -->|Search| D[Search Module]
    C -->|Extract| E[Extract Module]
    C -->|Query| F[Query Module]
    
    D --> G[File Scanner]
    G --> H[AST Parser]
    H --> I[Result Processor]
    I --> J[Result Ranker]
    J --> K[Output Formatter]
    
    E --> L[File Path Parser]
    L --> M[Symbol Finder]
    M --> N[Code Extractor]
    N --> O[Extract Formatter]
    
    F --> P[AST-grep Pattern Matcher]
    P --> Q[Query Result Processor]
    Q --> R[Query Formatter]
    
    K --> S[User Output]
    O --> S
    R --> S

Loading

Search Workflow

When you search for a pattern like "error handling":

1. Query Processing:

   • Tokenize → [error, handling]
   • Stem → [error, handl]
   • Generate patterns → \berror\b, \bhandl\w*\b
   • Parse query syntax → AND(error, handling)

2. File Scanning:

   • Identify potential matches using ripgrep
   • Filter based on .gitignore and custom patterns
   • Exclude test files (unless specified otherwise)

3. Code Analysis:

   • Parse matching files into ASTs
   • Identify complete code blocks containing matches
   • Extract metadata (function names, parameters, etc.)

4. Result Ranking:

   • Calculate relevance scores
   • Apply position weights for terms in identifiers
   • Sort by combined relevance metrics
   • Filter based on session cache (if enabled)

5. Block Extraction:

   • Extract complete functions/methods with matches
   • Merge related code blocks when appropriate
   • Apply context lines if requested

6. Result Delivery:

   • Format with syntax highlighting
   • Apply token/size constraints
   • Generate structured output

Performance Optimizations

Probe achieves high performance through several optimizations:

• Ripgrep Engine: Fast line scanning at core
• Parallel Processing: Utilizes all CPU cores
• Smart Filtering: Respects .gitignore patterns
• Stream Processing: Minimal memory footprint
• Incremental Matching: Stops scanning when limits are reached
• Session-based Caching: Avoids duplicate results across multiple searches

Output Strategies

Probe delivers results in various formats:

• Markdown/Syntax: Rich, readable code presentation
• JSON: Structured for programmatic use
• XML: Structured for integration with other tools
• Token Limiting: Fits within AI context windows
• Priority Handling: Most relevant results survive limits
• Streaming: Real-time output for interactive use

This architecture enables Probe to provide fast, accurate code search and extraction with language awareness, making it an effective tool for developers to navigate and understand codebases.

---

Tip: Start comment with /probe <request>.
Built by probe