BugChainIndexer

Multi-blockchain contract analysis and indexing system

🌐 Live Platform: https://bugchain.xyz/

BugChainIndexer is a comprehensive blockchain analysis platform that monitors, analyzes, and indexes contract data across 14 blockchain networks (12 active). The system uses Alchemy API for reliable data access and features an optimized backend delivering sub-second response times.

✨ Key Features

🔍 Multi-Chain Analysis

14 Blockchain Networks (12 Active): Ethereum, BSC, Polygon, Arbitrum, Optimism, Base, Avalanche, Gnosis, Linea, Scroll, Mantle, opBNB, Unichain, Berachain
Unified Processing: Single codebase handles all networks with consistent data structures
Parallel Execution: Process multiple networks simultaneously for maximum efficiency
Network-Specific Token Decimals: Accurate decimals for tokens across all networks

🚀 High-Performance Scanning

50,000+ addresses/hour: Revolutionary processing speed per network
5-in-1 Pipeline: Transfer events → Address filtering → EOA detection → Contract verification → Database storage
Smart Batching: Dynamic batch sizing with 300-1000 addresses per contract call
80%+ Efficiency Gain: Massive improvement over traditional individual scanners
Unified Alchemy RPC: All RPC calls (getLogs, eth_call, eth_getTransactionByHash, etc.) use Alchemy for reliability
Optimized Deployment Time Fetching: Uses Alchemy API for eth_getTransactionByHash and eth_getBlockByNumber

💰 Asset & Fund Tracking

BalanceHelper Contracts: Batch balance queries using on-chain contracts (550M gas limit optimized)
Alchemy Prices API: Real-time token prices with 7-day update cycle
Accurate Token Decimals: Network-specific decimals from tokens files (not DB)
Multi-Network Support: Portfolio tracking across 14 blockchain networks
PostgreSQL Advisory Locks: Concurrent-safe fund updates across multiple networks
ERC20 Balance Checking: UnifiedScanner includes contracts with token holdings
Dynamic Batch Sizing: Adaptive chunk sizes (50-1000 addresses) based on performance
Multi-level Fallback: Full chunk → half chunk → individual calls for reliability

🗄️ Advanced Database Management

PostgreSQL Optimization: 455x performance improvement with partial indexes
Smart Caching: 4-hour network counts cache eliminates expensive queries
Query Optimization: Intelligent count queries with fast path for network filters
Automated Maintenance: Daily/weekly/monthly optimization schedules
Real-time Monitoring: Query performance analysis and recommendations

🌐 Fast Backend API

Sub-second Response: Page loads in <1s (96% improvement from 25s)
Optimized Queries: Composite indexes for fund/deployed/address sorting
Smart Count Strategy:
- Fast path: Use cached network counts (0.09s)
- Skip count: Avoid full scans for 10+ networks
- Exact count: Run COUNT(*) only when needed
REST API: Clean endpoints for filtering, statistics, and network counts

🤖 Intelligent Automation

Cron Integration: Fully automated scanning with customizable schedules
Lock Management: Prevents duplicate processes with file-based locking
Error Recovery: Robust error handling with automatic retries
Log Management: Comprehensive logging with automatic cleanup

📊 Data Validation & Quality

EIP-55 Checksum: Uses ethers.getAddress() for proper address validation
Address Normalization: Consistent lowercase formatting with checksum verification
Contract Classification: Automatic EOA vs Contract detection
Source Code Verification: Etherscan API integration for contract metadata
Duplicate Prevention: Advanced deduplication and data integrity checks

🔧 Developer-Friendly Tools

Modular Architecture: Clean separation of concerns with reusable components
Comprehensive Testing: Built-in test suite for Alchemy API integration
Flexible Configuration: Environment-based settings with intelligent defaults
API Proxy Support: Centralized API management with parallel processing

🏗️ Architecture

BugChainIndexer/
├── scanners/                      # Core blockchain analysis engine
│   ├── common/                    # Shared utilities and base classes
│   │   ├── core.js                # Core blockchain functions with Alchemy integration
│   │   ├── database.js            # PostgreSQL operations
│   │   ├── Scanner.js             # Base scanner with dual RPC clients
│   │   ├── alchemyRpc.js          # Alchemy RPC client with Prices API support
│   │   ├── TokenPriceCache.js     # Token price fetching (price only)
│   │   └── addressUtils.js        # EIP-55 address validation
│   ├── core/                      # Scanner implementations
│   │   ├── UnifiedScanner.js      # Main pipeline with ERC20 balance checking
│   │   ├── FundUpdater.js         # Portfolio tracker with advisory locks
│   │   └── DataRevalidator.js     # Data validation and tagging
│   ├── tokens/                    # Token configurations (18 networks)
│   │   ├── ethereum.json          # 99 tokens with decimals
│   │   ├── binance.json           # 100 tokens with decimals
│   │   └── ...                    # 16 more networks (1,254 total)
│   └── config/
│       └── networks.js            # Network configurations (18 active)
├── server/
│   ├── backend/                   # REST API backend
│   │   └── services/
│   │       └── address.service.js # Optimized query service
│   ├── frontend/                  # Web interface
│   │   └── index.html             # Optimized UI (no blocking calls)
│   └── etherscan-proxy-server/    # Centralized Etherscan API proxy
└── docs/                          # Documentation

🚀 Quick Start

Prerequisites

Node.js (v16+)
PostgreSQL (v12+)
Alchemy API key
Etherscan API keys for blockchain explorers

1. Clone and Install

git clone <repository-url>
cd BugChainIndexer/scanners
npm install

2. Configure Environment

cp .env.example .env
# Edit .env with your API keys and database settings

Required Environment Variables:

# Alchemy API (required for FundUpdater)
ALCHEMY_API_KEY=your_alchemy_key

# Etherscan API keys
DEFAULT_ETHERSCAN_KEYS=key1,key2,key3

# Database
PGDATABASE=bugchain_indexer
PGUSER=postgres
PGPASSWORD=your_password

3. Set up Database

# BugChainIndexer automatically creates the database!
# Just ensure PostgreSQL is running and configure .env

# The system will automatically create:
# - addresses table with optimized indexes
# - token_metadata_cache table (30-day TTL)
# - token_price_cache table (4-hour TTL)

4. Run Scanner

# Analyze single network (RECOMMENDED METHOD)
NETWORK=ethereum ./run.sh unified

# Alternative method with correct parameter order
./run.sh unified auto ethereum

# Analyze all networks in parallel
./run.sh unified parallel

# Update asset prices and balances (uses Alchemy API)
NETWORK=ethereum ./run.sh funds

# Validate existing data
./run.sh revalidate

5. Start Backend API

cd server/backend
npm install
npm start  # Starts HTTPS server on port 443

🌐 Supported Networks

14 Configured Networks (12 Active in run.sh):

✅ Fully Operational (Active in run.sh):

Tier 1: Ethereum, Binance Smart Chain, Polygon
Tier 2: Arbitrum, Optimism, Base
Tier 3: Avalanche, Gnosis, Linea, Scroll, Mantle, opBNB

✅ Configured (Available but not in default run.sh):

Unichain, Berachain

Network Support:

All networks have Alchemy API integration
BalanceHelper contracts deployed on all active networks
Network-specific RPC fallbacks for reliability

📊 Performance Metrics

Frontend Performance

Page Load: 25s → <1s (96% improvement)
Multi-network Query: 19s → 0.09s (99.5% improvement)
Network Counts API: 23s → <0.1s with 4-hour cache
Single Network Query: <0.02s with cache hit

Scanner Performance

Processing Speed: ~50,000 addresses/hour per network
Efficiency Gain: 80%+ improvement over individual scanners
Database Optimization: 455x performance boost in data validation
Architecture: Unified 5-in-1 analysis pipeline

Backend Optimizations

Composite Index: (fund DESC, deployed DESC, address ASC) for fast sorting
Cached Counts: 4-hour TTL eliminates 23s GROUP BY queries
Smart Query Strategy: Fast path uses cache, skip count for 10+ networks
No Blocking Calls: Removed highlightNetworksWithData from frontend

🏭 Components

Scanners

High-performance blockchain analysis engine with Alchemy API integration.

UnifiedScanner: Main analysis pipeline with ERC20 balance checking
FundUpdater: Portfolio tracking with PostgreSQL advisory locks and network-specific decimals
DataRevalidator: Validates and updates addresses with incomplete data

Key Features:

Network-specific token decimals (1,254 tokens across 18 networks)
PostgreSQL advisory locks prevent concurrent update conflicts
ERC20 balance checking includes contracts with token holdings
Accurate fund calculations using tokens file decimals (not DB)
Simplified DataRevalidator with unified reclassification logic

Backend Server

Express.js REST API with PostgreSQL database.

Optimized queries with composite indexes
4-hour network counts cache
Advanced filtering and pagination
Real-time contract and address data

Frontend

Fast web interface with no blocking API calls.

Instant page load (removed 23s highlightNetworksWithData)
Advanced filtering (address, name, time, fund, networks)
Pagination with cursor-based navigation

API Proxy Server

High-performance centralized Etherscan API management.

Parallel Processing: 4 concurrent requests with multiple API keys
Address Normalization: Automatic format validation and correction
Per-Key Rate Limiting: Independent 5 req/s limit per API key

🔧 Configuration

API Keys Required

# Alchemy API (required for FundUpdater)
ALCHEMY_API_KEY=your_alchemy_key

# Optional: Use Alchemy proxy for better performance
USE_ALCHEMY_PROXY=true
ALCHEMY_PROXY_URL=http://localhost:3002

# Blockchain explorer APIs (required for scanners)
DEFAULT_ETHERSCAN_KEYS=key1,key2,key3

# Etherscan proxy server
USE_ETHERSCAN_PROXY=true
ETHERSCAN_PROXY_URL=http://localhost:3000

Database Settings

PGHOST=localhost
PGPORT=5432
PGDATABASE=bugchain_indexer
PGUSER=indexer_user
PGPASSWORD=secure_password

Scanner Options

TIMEDELAY_HOURS=4              # Analysis time window
FUNDUPDATEDELAY=7              # Fund update delay (days)
TIMEOUT_SECONDS=7200           # Script timeout

# FundUpdater options
FUND_UPDATE_MAX_BATCH=50000    # Max addresses per batch
ALL_FLAG=true                  # Update all addresses (ignore delay)
HIGH_FUND_FLAG=true            # Target high-value addresses (100K+)

🤖 Automation

Set up automated scanning with cron:

cd scanners/cron
./setup-cron.sh --auto-setup

Default Schedule:

Unified Analysis: Every 4 hours
Fund Updates: Every 6 hours
Data Validation: Weekly (Sundays 2 AM)
Database Optimization: Daily maintenance

📈 Database Optimization

Built-in database optimization tools for high-performance operations:

# Daily optimization (fast)
./run.sh db-optimize-fast

# Weekly maintenance (with VACUUM)
./run.sh db-optimize

# Monthly large database optimization (10GB+)
./run.sh db-optimize-large

# Performance analysis
./run.sh db-analyze

Key Indexes Created:

idx_addresses_api_sort_optimal: (fund DESC, deployed DESC, address ASC)
idx_addresses_address_prefix: Address prefix search with text_pattern_ops
idx_token_metadata_cache_updated: Metadata cache lookup
idx_token_price_cache_updated: Price cache lookup

🔍 API Usage

Backend API Endpoints

Base URL: https://api.bugchain.xyz

# Get addresses with filtering
GET /getAddressesByFilter?limit=50&includeTotal=true&networks=ethereum,polygon

# Query Parameters:
# - limit: Results per page (1-200, default 50)
# - includeTotal: Calculate total count (default false)
# - networks: Comma-separated network list
# - address: Address search (exact or prefix)
# - contractName: Contract name search
# - deployedFrom/deployedTo: Unix timestamp range
# - fundFrom/fundTo: USD value range
# - cursor: Pagination cursor (from previous response)

# Get contract count
GET /getContractCount

# Get network statistics (4-hour cache)
GET /networkCounts

Performance Tips

Use includeTotal=false for fastest queries (0.09s)
With 10+ networks, totalCount is automatically skipped
Network-only filters use fast cache path (0.09s vs 19s)
Address/name filters trigger exact count (slower but accurate)

📋 Requirements

System Requirements

RAM: 4GB+ (8GB+ recommended for parallel processing)
Storage: 50GB+ for database (grows with usage)
Network: Stable internet for RPC and API calls

API Limits

Alchemy API: Varies by plan (Growth: 330 CU/sec)
Etherscan: 5 calls/second (free), 10k calls/day
RPC: Varies by provider (failover supported)

🛠️ Development

Adding New Networks

Check Alchemy API support for the network
Add network config to scanners/config/networks.js:

{
  network: 'newchain',
  rpc: ['https://rpc.newchain.com'],
  etherscanUrl: 'https://api.newchain.com/api',
  alchemyNetwork: 'newchain-mainnet',  // Required for FundUpdater
  contractValidator: '0x...'  // Optional
}

Add tokens to scanners/tokens/newchain.json (for FundUpdater)
Test with NETWORK=newchain ./run.sh unified

Testing

# Test Alchemy API integration
cd scanners
node tests/test-fundupdater-alchemy.js

# Test scanner functionality
NETWORK=ethereum ./run.sh unified auto

🔄 Migration Notes

Moralis → Alchemy Migration

This version has migrated from Moralis to Alchemy API:

Removed: Moralis SDK dependencies
Added: Alchemy Data API v1 for token balances and metadata
Added: Token price caching (7-day TTL)
Improved: Address validation with ethers.getAddress()
Optimized: Contract deployment time fetching via Alchemy RPC (eth_getTransactionByHash, eth_getBlockByNumber)
Performance: 99.5% improvement in multi-network queries

Breaking Changes

Networks without Alchemy support removed (Cronos, Moonriver, etc.)
MORALIS_API_KEY replaced with ALCHEMY_API_KEY
FundUpdater now requires Alchemy API key

📝 License

MIT License - see LICENSE file for details.

🤝 Contributing

Fork the repository
Create feature branch (git checkout -b feature/amazing-feature)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open Pull Request

📞 Support

Documentation: See individual component READMEs
Performance: Built-in monitoring and optimization tools
API Status: Check Alchemy and Etherscan status pages

Built for scale. Optimized for performance. Ready for production.

Name		Name	Last commit message	Last commit date
Latest commit History 38 Commits
contract		contract
docs		docs
scanners		scanners
server		server
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
SETUP.md		SETUP.md

License

h0lydev/BugChainIndexer

Folders and files

Latest commit

History

Repository files navigation