Feature/stats page

.github/workflows/tests.yml

@@ -0,0 +1,46 @@
+name: Tests
+
+on:
+  push:
+    branches: [ "main", "feature/*" ]
+  pull_request:
+    branches: [ "main", "feature/*" ]
+
+jobs:
+  backend-tests:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: backend
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.12
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+        pip install pytest pytest-mock
+    - name: Run tests
+      run: |
+        pytest tests/
+
+  frontend-tests:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: frontend
+    steps:
+    - uses: actions/checkout@v4
+    - name: Use Node.js 22
+      uses: actions/setup-node@v4
+      with:
+        node-version: 22
+        cache: 'npm'
+        cache-dependency-path: frontend/package-lock.json
+    - name: Install dependencies
+      run: npm ci
+    - name: Run tests
+      run: npm run test

.gitignore

@@ -67,6 +67,18 @@ db.sqlite3-journal
 # Flask stuff:
 instance/
 .webassets-cache
+backend/rpc_config.ini
+backend/fee_analysis.db
+
+# Logs
+*.log
+nohup.out
+
+# Next.js
+frontend/.next/
+frontend/node_modules/
+frontend/frontend.log
+frontend/frontend.pd
 
 # Scrapy stuff:
 .scrapy
@@ -217,3 +229,6 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+fee_analysis.db
+
+.DS_Store

Readme.md

@@ -1,4 +1,72 @@
-Bitcoin Core Feerate API
+### Bitcoin Core Fee Rate Estimator
 
-ASAP: https://bitcoincorefeerate.com/fees/2/economical/2
+- A full-stack application for monitoring and validating Bitcoin Core transaction fee estimates against actual block data.
+- Built on top of Bitcoin Core PR #34075
 
+### Overview
+
+This project tracks `estimatesmartfee` from a Bitcoin Core node and compares those estimates with the feerate percentiles of subsequent blocks. It provides a visual interface to verify the accuracy of the node's fee predictions.
+
+#### Key Features
+- **Fee Estimate Tracking**: A background service polls Bitcoin Core every 7 seconds for smart fee estimates.
+- **Historical Accuracy**: Visualizes the accuracy of estimates (within range, overpaid, or underpaid) compared to real block data.
+- **Mempool Diagram**: Real-time visualization of the mempool fee/weight accumulation curve.
+- **Block Statistics**: Direct insights into feerate percentiles for recent blocks.
+
+#### Architecture
+
+- **Backend (Python/Flask)**: Communicates with Bitcoin Core via RPC. Collects estimates into SQLite and serves data via a REST API.
+- **Frontend (Next.js/TypeScript)**: Modern UI using Recharts and D3. Communicates with the backend via a secure API proxy route.
+
+#### Project Structure
+
+```text
+.
+├── backend/            # Flask API, data collector, and SQLite database
+│   ├── src/            # Core logic and RPC services
+│   └── tests/          # Pytest suite for backend validation
+├── frontend/           # Next.js web application
+│   ├── src/app/        # App router and pages
+│   └── src/components/ # D3 and Recharts visualization components
+└── .github/workflows/  # Automated testing workflow
+```
+
+#### How to Use
+
+#### Prerequisites
+- **Bitcoin Core Node**: Access to a node with RPC enabled (`getblockstats` support required).
+- **Python**: 3.12+
+- **Node.js**: 22+
+
+#### 1. Configuration
+- **Backend**: Copy `backend/rpc_config.ini.example` to `backend/rpc_config.ini` and provide RPC credentials.
+
+#### 2. Manual Startup
+**Backend:**
+```bash
+cd backend
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python src/app.py
+```
+
+**Frontend:**
+```bash
+cd frontend
+npm install
+npm run dev
+```
+
+#### 3. Automated Startup
+Use the provided `restart.sh` script to launch both services in the background:
+```bash
+chmod +x restart.sh
+./restart.sh
+```
+
+### Credits
+- **Abubakar Sadiq Ismail**: Bitcoin Core contributor and architecture.
+- **b-l-u-e**: Backend logic and service implementation.
+- **mercie-ux**: Frontend design and visual components.
+- **Gemini & Claude**: AI-assisted development and test automation.

backend/.gitignore

@@ -0,0 +1,8 @@
+# test files
+test_mock.py
+test_secure_connection.py
+test_rpc_ports.py
+test_getbestblockhash.py
+
+rpc_config.ini
+

backend/doc.md

@@ -0,0 +1,50 @@
+# Backend - Bitcoin Core Fees API
+
+This Flask-based REST API interacts with Bitcoin Core RPC and a local SQLite database to provide fee analytics and block statistics.
+
+## Running the Application
+
+### 1. Prerequisites
+Ensure you have a virtual environment set up and dependencies installed:
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+Ensure `rpc_config.ini` is configured with your Bitcoin Core RPC credentials.
+
+### 2. Start the App (Background)
+To start the application in the background:
+
+```bash
+nohup env PYTHONPATH=src .venv/bin/gunicorn --workers 4 --bind 127.0.0.1:5001 app:app > debug.log 2>&1 &
+```
+
+### 3. Monitoring Logs
+To see the logs in real-time:
+```bash
+tail -f debug.log
+```
+
+### 4. Stopping the App
+To stop the process:
+```bash
+pkill -f "gunicorn"
+```
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/blockcount` | GET | Current block height from node. |
+| `/fees/<target>/<mode>/<level>` | GET | `estimatesmartfee` results converted to sat/vB. |
+| `/mempool-diagram` | GET | Analyzed feerate diagram for mempool accumulation. |
+| `/performance-data/<start_block>/` | GET | Block feerate percentiles vs. recorded estimates. |
+| `/fees-sum/<start_block>/` | GET | Aggregated accuracy metrics (within, over, under). |
+
+### Parameters:
+- `target`: Confirmation target (e.g., 2, 7, 144).
+- `mode`: Fee estimation mode (`economical`, `conservative`, `unset`).
+- `level`: Verbosity level for fee estimation.
+- `start_block`: Block height to start range analysis from.

backend/requirements.txt

@@ -0,0 +1,8 @@
+Flask==3.1.3
+Flask-CORS==6.0.2
+Flask-Limiter==4.1.1
+requests==2.32.5
+configparser==6.0.0
+gunicorn==25.1.0
+pytest==9.0.2
+pytest-cov==7.0.0

backend/src/app.py

@@ -0,0 +1,128 @@
+import logging
+import os
+from flask import Flask, jsonify, request
+from flask_cors import CORS
+from flask_limiter import Limiter
+from flask_limiter.util import get_remote_address
+from werkzeug.middleware.proxy_fix import ProxyFix
+import services.rpc_service as rpc_service
+import services.collector_service as collector_service
+import services.database_service as db_service
+
+logging.basicConfig(
+    level=logging.DEBUG,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+logger = logging.getLogger(__name__)
+
+def create_app():
+    app = Flask(__name__)
+    # NOTE: Configure x_for=1 to match your actual proxy depth.
+    # Without this, X-Forwarded-For spoofing can defeat IP-based limiting.
+    app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1)
+    CORS(app)
+
+    # ---------------------------------------------------------------------------
+    # Rate limiting
+    # ---------------------------------------------------------------------------
+    # Uses the real client IP (respects ProxyFix above).
+    # Default: 200 requests/day, 60/hour applied to every endpoint unless
+    # overridden with a per-route @limiter.limit() decorator below.
+    # ---------------------------------------------------------------------------
+    limiter = Limiter(
+        key_func=get_remote_address,
+        app=app,
+        default_limits=["10000 per day", "1000 per hour"],
+        # Store state in memory by default. For multi-worker/multi-process
+        # deployments swap this for a Redis URI:
+        #   storage_uri="redis://localhost:6379"
+        storage_uri="memory://",
+        # Return 429 JSON instead of HTML when limit is hit
+        headers_enabled=True,   # adds X-RateLimit-* headers to responses
+    )
+
+    db_service.init_db()
+    collector_service.start_background_collector()
+
+    # ---------------------------------------------------------------------------
+    # Routes
+    # ---------------------------------------------------------------------------
+
+    @app.route("/fees/<int:target>/<string:mode>/<int:level>", methods=['GET'])
+    @limiter.limit("50 per minute")   # estimatesmartfee is a node RPC call — keep it tight
+    def fees(target, mode, level):
+        VALID_MODES = {"economical", "conservative", "unset"}
+        if mode not in VALID_MODES:
+            return jsonify({"error": f"Invalid mode '{mode}'. Must be one of: {', '.join(VALID_MODES)}"}), 400
+        try:
+            result = rpc_service.estimate_smart_fee(conf_target=target, mode=mode, verbosity_level=level)
+            return jsonify(result)
+        except Exception as e:
+            logger.error(f"/fees RPC failed: {e}", exc_info=True)
+            return jsonify({"error": "Internal server error"}), 500
+
+    @app.route("/mempool-diagram", methods=['GET'])
+    @limiter.limit("50 per minute")   # expensive computation — strict cap
+    def mempool_diagram():
+        try:
+            result = rpc_service.get_mempool_feerate_diagram_analysis()
+            return jsonify(result)
+        except Exception as e:
+            logger.error(f"Mempool diagram RPC failed: {e}", exc_info=True)
+            return jsonify({"error": "Internal server error"}), 500
+
+    @app.route("/performance-data/<int:start_block>/", methods=['GET'])
+    @limiter.limit("50 per minute")   # hits DB + RPC
+    def get_performance_data(start_block):
+        target = request.args.get('target', default=2, type=int)
+        try:
+            data = rpc_service.get_performance_data(start_height=start_block, count=100, target=target)
+            return jsonify(data)
+        except Exception as e:
+            logger.error(f"/performance-data RPC failed: {e}", exc_info=True)
+            return jsonify({"error": "Internal server error"}), 500
+
+    @app.route("/fees-sum/<int:start_block>/", methods=['GET'])
+    @limiter.limit("50 per minute")
+    def get_local_fees_sum(start_block):
+        target = request.args.get('target', default=2, type=int)
+        try:
+            data = rpc_service.calculate_local_summary(target=target)
+            return jsonify(data)
+        except Exception as e:
+            logger.error(f"/fees-sum failed: {e}", exc_info=True)
+            return jsonify({"error": "Internal server error"}), 500
+
+    @app.route("/blockcount", methods=['GET'])
+    @limiter.limit("100 per minute")   # cheap call, slightly more relaxed
+    def block_count():
+        try:
+            result = rpc_service.get_block_count()
+            return jsonify({"blockcount": result})
+        except Exception as e:
+            logger.error(f"/blockcount RPC failed: {e}", exc_info=True)
+            return jsonify({"error": "Internal server error"}), 500
+
+    # ---------------------------------------------------------------------------
+    # Error handlers
+    # ---------------------------------------------------------------------------
+
+    @app.errorhandler(404)
+    def page_not_found(error):
+        return jsonify({"error": "Endpoint not found"}), 404
+
+    @app.errorhandler(429)
+    def rate_limit_exceeded(error):
+        # error.description is the limit string e.g. "30 per 1 minute"
+        logger.warning(f"Rate limit exceeded from {get_remote_address()}: {error.description}")
+        return jsonify({
+            "error": "Too many requests",
+            "message": f"Rate limit exceeded: {error.description}. Please slow down."
+        }), 429
+
+    return app
+
+app = create_app()
+if __name__ == "__main__":
+    port = int(os.environ.get("PORT", 5001))
+    app.run(debug=False, host='0.0.0.0', port=port)