Sol-Indexer

A high-performance Solana blockchain indexer built in Rust. Connects to Yellowstone gRPC (Geyser plugin) for real-time data streaming and stores indexed data in Redis (caching) and ClickHouse (analytics).

Features

• Real-time streaming via Yellowstone gRPC (Geyser plugin)
• Dual storage backend: Redis/DragonflyDB for caching, ClickHouse for analytics
• DragonflyDB support with multi-threaded optimizations for high throughput
• LMAX Disruptor pattern for low-latency inter-thread messaging
• WebSocket server for streaming data to clients
• Instruction decoding for System, Token, and Compute Budget programs
• In-memory state management with write-version conflict resolution
• Docker support with docker-compose for easy deployment

Architecture

┌─────────────────┐     ┌────────────────┐     ┌─────────────────┐
│  Solana Node    │────▶│   Ingestor     │────▶│   Dispatcher    │
│  (Yellowstone)  │     │  (gRPC Client) │     │  (Message Bus)  │
└─────────────────┘     └────────────────┘     └────────┬────────┘
                                                        │
                        ┌───────────────────────────────┼───────────────────────────────┐
                        │                               │                               │
                        ▼                               ▼                               ▼
               ┌────────────────┐             ┌────────────────┐             ┌────────────────┐
               │     State      │             │      Sink      │             │   WS Server    │
               │   (In-Memory)  │             │ (Redis + CH)   │             │   (Clients)    │
               └────────────────┘             └────────────────┘             └────────────────┘

Quick Start

Prerequisites

• Rust 1.75+ (or Docker)
• Redis
• ClickHouse
• Access to a Yellowstone gRPC endpoint

Using Docker Compose (Recommended)

# Clone the repository
git clone https://github.com/your-org/sol-indexer.git
cd sol-indexer

# Copy environment template
cp .env.example .env

# Edit .env with your gRPC endpoint
vim .env

# Start all services (DragonflyDB is the default)
docker-compose up -d

# Run migrations (first time only)
docker-compose --profile migrate up migrate

# View logs
docker-compose logs -f indexer

Why DragonflyDB is Default

DragonflyDB is a Redis-compatible, multi-threaded in-memory datastore that offers significantly better performance than Redis for high-throughput workloads.

Performance Benefits:

• 4-8x faster write throughput compared to Redis
• Multi-threaded architecture (vs Redis single-threaded)
• Better memory efficiency (up to 25% less memory usage)
• Perfect for DEX indexing with high transaction volumes
• Drop-in Redis replacement (same protocol)

Using Legacy Redis (Optional)

If you need to use standard Redis instead:

# Start with Redis instead of DragonflyDB
docker-compose --profile redis up -d

# Run migrations
docker-compose --profile migrate up migrate

# View logs
docker-compose logs -f indexer-redis

DragonflyDB Configuration:

# Set number of threads (default: 8 for optimal performance)
export DRAGONFLY_THREADS=8
docker-compose up -d

Note: DragonflyDB is now the default. All optimizations are pre-configured in config.toml.

Building from Source

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Build
cargo build --release

# Run migrations
./target/release/sol-indexer migrate \
  --clickhouse-url http://localhost:8123 \
  --clickhouse-database solana

# Run the indexer
./target/release/sol-indexer run \
  --grpc-endpoint https://your-grpc-endpoint.com \
  --grpc-token YOUR_TOKEN \
  --redis-url redis://localhost:6379 \
  --clickhouse-url http://localhost:8123 \
  --ws-bind 0.0.0.0:9000

Configuration

Environment Variables

Variable	Description	Default
GRPC_ENDPOINT	Yellowstone gRPC endpoint URL	Required
GRPC_TOKEN	Authentication token for gRPC	Optional
REDIS_URL	Redis/DragonflyDB connection URL	redis://127.0.0.1:6379
CLICKHOUSE_URL	ClickHouse HTTP URL	http://127.0.0.1:8123
CLICKHOUSE_DATABASE	ClickHouse database name	solana
WS_BIND	WebSocket server bind address	0.0.0.0:9000
METRICS_BIND	Prometheus metrics bind address	0.0.0.0:9090
HEALTH_BIND	Health check endpoint bind address	0.0.0.0:8081
RUST_LOG	Log level (debug, info, warn, error)	info
DRAGONFLY_THREADS	DragonflyDB proactor threads	4

CLI Options

sol-indexer run --help

Options:
      --grpc-endpoint <GRPC_ENDPOINT>  gRPC endpoint URL [env: GRPC_ENDPOINT]
      --grpc-token <GRPC_TOKEN>        gRPC authentication token [env: GRPC_TOKEN]
      --redis-url <REDIS_URL>          Redis URL [env: REDIS_URL] [default: redis://127.0.0.1:6379]
      --clickhouse-url <URL>           ClickHouse URL [env: CLICKHOUSE_URL] [default: http://127.0.0.1:8123]
      --clickhouse-database <DB>       ClickHouse database [env: CLICKHOUSE_DATABASE] [default: solana]
      --ws-bind <WS_BIND>              WebSocket bind address [env: WS_BIND] [default: 0.0.0.0:9000]
  -v, --verbose                        Enable verbose logging
  -h, --help                           Print help

WebSocket API

Connect to the WebSocket server to receive real-time updates.

Subscribe to Transactions

{
  "method": "subscribe",
  "params": {
    "channel": "transactions",
    "filters": {
      "accounts": ["TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"],
      "programs": ["JUP6LkbZbjS1jKKwapdHNy74zcZ3tLUZoi5QNyVTaV4"]
    }
  }
}

Subscribe to Account Updates

{
  "method": "subscribe",
  "params": {
    "channel": "accounts",
    "filters": {
      "accounts": ["your-account-pubkey"]
    }
  }
}

Subscribe to Slots

{
  "method": "subscribe",
  "params": {
    "channel": "slots"
  }
}

Notifications

{
  "jsonrpc": "2.0",
  "method": "transactionNotification",
  "params": {
    "signature": "5VERv8NMvzbJMEkV8xnrLkEaWRtSz9CosKDYjCJjBRnbJLgp8uirBgmQpjKhoR4tjF3ZpRzrFmBV6UjKdiSZkQUW",
    "slot": 123456789,
    "success": true,
    "fee": 5000,
    "accounts": ["..."],
    "program_ids": ["..."]
  }
}

Database Schema

ClickHouse Tables

transactions

• signature - Transaction signature (String)
• slot - Slot number (UInt64)
• block_time - Unix timestamp (Int64)
• success - Success flag (UInt8)
• fee - Transaction fee in lamports (UInt64)
• accounts - Account addresses (Array(String))
• program_ids - Program IDs (Array(String))
• log_messages - Log messages (Array(String))

accounts

• pubkey - Account public key (String)
• slot - Last updated slot (UInt64)
• lamports - Balance in lamports (UInt64)
• owner - Owner program ID (String)
• executable - Executable flag (UInt8)
• data_len - Data length (UInt64)
• write_version - Write version for deduplication (UInt64)

slots

• slot - Slot number (UInt64)
• parent - Parent slot (UInt64)
• status - Slot status (String: Processed/Confirmed/Finalized)
• block_time - Unix timestamp (Int64)

Project Structure

sol-indexer/
├── common/          # Shared types, errors, configuration
├── ingestor/        # Yellowstone gRPC client
├── dispatcher/      # Message routing and broadcast
├── decoders/        # Instruction and account decoding
├── state/           # In-memory state management
├── sink/            # Redis and ClickHouse persistence
├── migration/       # Database migrations
├── ws-server/       # WebSocket server for clients
├── indexer/         # Main binary and CLI
├── Dockerfile       # Multi-stage Docker build
├── docker-compose.yml
└── README.md

Development

Running Tests

cargo test

Building Documentation

cargo doc --open

Yellowstone gRPC Providers

• Helius - Managed Solana infrastructure
• Triton - Solana RPC and streaming
• QuickNode - Multi-chain node provider

License

MIT License - see LICENSE for details.