Comprehensive guide for running a Basilica miner node that provides GPU compute resources to the Bittensor network.
What it does: Miner orchestrates validator access to your GPU nodes via SSH. No executor binaries needed.
Minimum Requirements:
- Miner server: Linux with 8+ CPU cores, 16GB RAM, public IP
- GPU node(s): NVIDIA GPU (A100/H100/B200), CUDA ≥12.8, Docker with nvidia runtime
- Bittensor wallet registered on subnet 39 (mainnet) or 387 (testnet)
Quick Setup (5 steps):
# 1. Generate SSH key for node access
ssh-keygen -t ed25519 -f ~/.ssh/miner_node_key -N ""
# 2. Deploy key to GPU nodes
ssh-copy-id -i ~/.ssh/miner_node_key.pub basilica@<gpu_node_ip>
# 3. Copy and edit config from template
cp config/miner.toml.example miner.toml
# Edit miner.toml with your settings:
# - [bittensor] wallet_name, hotkey_name, external_ip
# - [node_management] nodes list with your GPU nodes
# - [ssh_session] miner_node_key_path
# Minimal example configuration:
cat > miner.toml <<EOF
[bittensor]
wallet_name = "your_wallet"
hotkey_name = "your_hotkey"
external_ip = "your_public_ip"
axon_port = 50051
network = "finney"
netuid = 39
chain_endpoint = "wss://entrypoint-finney.opentensor.ai:443"
[database]
url = "sqlite:///opt/basilica/data/miner.db"
[validator_comms]
host = "0.0.0.0"
port = 50051
[node_management]
nodes = [
{ host = "<node_ip>", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 },
]
[ssh_session]
miner_node_key_path = "~/.ssh/miner_node_key"
default_node_username = "basilica"
[validator_assignment]
strategy = "highest_stake"
EOF
# 4. Build and run (with docker compose)
cp ./scripts/miner/compose.prod.yml compose.yml
docker compose up -d
# Check status
docker compose ps
# View logs
docker compose logs -f miner
# 4. Build and run (with self compiled binary)
./scripts/miner/build.sh
./basilica-miner --config miner.toml
# 5. Verify validators can discover nodes
# Check logs for "Node registered" and "Validator authenticated" messagesNeed details? See sections below for architecture explanation, security hardening, troubleshooting, and advanced configuration.
- Overview
- Architecture
- Prerequisites
- Understanding the System
- SSH Key Setup
- GPU Node Preparation
- Miner Configuration
- Deployment Methods
- Validator Access Flow
- Security & Best Practices
- Monitoring
- Troubleshooting
- Advanced Topics
The Basilica miner manages a fleet of GPU nodes and provides validators with direct SSH access to these nodes for verification and rental operations. Unlike traditional architectures that require intermediary agents, Basilica miners act as access control orchestrators, deploying validator SSH keys to nodes on-demand.
-
Miner Server: Linux system with network connectivity (no GPU required)
- 8+ CPU cores, 16GB+ RAM recommended
- Public IP address or port forwarding
- SSH access to your GPU nodes
-
GPU Nodes: One or more servers with:
- NVIDIA GPU (A100, H100, or B200 supported)
- NVIDIA CUDA drivers version ≥12.8
- Linux OS with SSH server
- Docker installed (for container workloads with nvidia runtime)
- All ports need to be open, the NAT or firewall should allow inbound SSH connections from the miner and validator server
- the validator shall be in control of which ports need to have open internet access
- at least 1TB of free disk space recommended (for container images and data)
-
Bittensor Wallet: Registered on subnet 39 (mainnet) or 387 (testnet)
┌──────────────────┐
│ Validator │
│ │
└────────┬─────────┘
│
│ 1. Discovery Request
│ (with SSH public key and signature)
↓
┌──────────────────┐
│ Miner (gRPC) │
│ - Authenticates │
│ - Deploys keys │
│ - Returns nodes │
└────────┬─────────┘
│
│ 2. SSH Key Deployment
│ (miner → nodes)
|────────────────────────────|────────────────────────────|
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ GPU Node 1 │ │ GPU Node 2 │ │ GPU Node N │
│ (SSH endpoint) │ │ (SSH endpoint) │ │ (SSH endpoint) │
└────────▲─────────┘ └────────▲─────────┘ └────────▲─────────┘
│ │ │
└────────────────────────────┴────────────────────────────┘
3. Validator connects directly via SSH
# Update system
sudo apt update && sudo apt upgrade -y
# Install build dependencies
sudo apt install -y \
build-essential \
libssl-dev \
pkg-config \
protobuf-compiler \
git \
curl
# Install Rust (if building from source)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
# Install Docker (optional, for Docker deployment)
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER# Install NVIDIA drivers and CUDA (if not already installed)
# Follow NVIDIA's official installation guide for your GPU model
# Install Docker
curl -fsSL https://get.docker.com | sudo sh
# Install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Verify GPU access
nvidia-smi
docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi# Install btcli if not already installed
pip install bittensor
# Create wallet (if you don't have one)
btcli wallet new_coldkey --wallet.name miner_wallet
btcli wallet new_hotkey --wallet.name miner_wallet --wallet.hotkey default
# Register on subnet (requires TAO for registration fee)
# Mainnet (subnet 39)
btcli subnet register --netuid 39 --wallet.name miner_wallet --wallet.hotkey default
# Testnet (subnet 387)
btcli subnet register --netuid 387 --wallet.name miner_wallet --wallet.hotkey default --subtensor.network testVerify wallet location:
ls ~/.bittensor/wallets/miner_wallet/hotkeys/default
# Should show the hotkey fileNodes are identified by deterministic UUIDs generated from their SSH credentials:
node_id = UUID_v5_namespace(username@host:port)
Key characteristics:
- Same credentials always generate the same node ID
- No database persistence required for node identity
- Node IDs stored in miner's SQLite database
- Path:
~/.bittensor/wallets/{wallet_name}/hotkeys/{hotkey_name}
Example:
[node_management]
nodes = [
{ host = "192.168.1.100", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 }
]Generates: node_id = "a3f2b1c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c" (deterministic)
Basilica uses mutual authentication between validators and miners:
-
Validator sends authentication request:
ValidatorAuthRequest { validator_hotkey: "5G3qVaXz..." (SS58-encoded Bittensor hotkey) signature: Ed25519 signature nonce: "uuid-1234-5678" timestamp: Unix timestamp target_miner_hotkey: "5FHd7..." }
-
Signature payload:
BASILICA_AUTH_V1:{nonce}:{target_miner_hotkey}:{timestamp_seconds} -
Miner verifies:
- Signature is valid for validator's hotkey
- Timestamp is within 5 minutes (prevents replay attacks)
target_miner_hotkeymatches miner's actual hotkey (prevents MITM)
-
Miner signs response:
MinerAuthResponse { authenticated: true session_token: "hex-token-32-bytes" expires_at: timestamp + 3600 seconds miner_hotkey: "5FHd7..." miner_signature: Ed25519 signature response_nonce: "uuid-8765-4321" }
-
Miner signature payload:
MINER_AUTH_RESPONSE:{validator_hotkey}:{response_nonce}:{session_token} -
Validator verifies miner's signature to ensure communication with legitimate miner
When validator discovers nodes:
-
Validator includes SSH public key in discovery request:
DiscoverNodesRequest { validator_hotkey: "5G3qVaXz..." validator_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5..." signature: ... nonce: ... }
-
Miner validates SSH key format:
- Supports:
ssh-rsa,ssh-ed25519,ecdsa-sha2-*,ssh-dss - Rejects malformed keys
- Supports:
-
Miner deploys key to ALL nodes:
# For each node, miner executes via SSH: mkdir -p ~/.ssh && \ echo 'ssh-ed25519 AAAAC3... validator-5G3qVaXz...' >> ~/.ssh/authorized_keys && \ chmod 600 ~/.ssh/authorized_keys
-
Key tagging: Keys are tagged with
validator-{hotkey}for easy revocation:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5... validator-5G3qVaXzKMPDm5AJ3dpzbpUC27kpccBvDwzSWXrq8M6qMmbC -
Miner returns node connection details:
NodeConnectionDetails { node_id: "a3f2b1c4-..." host: "192.168.1.100" port: "22" username: "basilica" additional_opts: "-o StrictHostKeyChecking=no" status: "available" }
-
Validator connects directly:
ssh -i ~/.basilica/ssh/validator_persistent.pem basilica@192.168.1.100
Miners can control which validators receive access to their nodes:
Automatically assigns ALL nodes to the validator with highest stake. No configuration required beyond setting the strategy.
[validator_assignment]
strategy = "highest_stake"Use cases:
- Production deployment (recommended default)
- Maximize security by working with most invested validator
- Simplify operations (automatic validator selection)
Behavior:
- Fetches validators from Bittensor metagraph
- Selects highest-staked validator with validator_permit
validator_hotkeyconfig is ignored with this strategy (usefixed_assignmentif you need a specific validator)
Assign nodes to a specific validator by hotkey. Mainly useful for debugging or testing.
[validator_assignment]
strategy = "fixed_assignment"
validator_hotkey = "5G3qVaXzKMPDm5AJ3dpzbpUC27kpccBvDwzSWXrq8M6qMmbC"Use cases:
- Debugging with a specific known validator
- Testing environments where you need deterministic validator assignment
Validator's perspective:
- Query Bittensor metagraph for miners on subnet
- Extract miner endpoints from axon data
- Connect to miner's gRPC service (port 50051 by default)
- Authenticate using Bittensor hotkey signature
- Send discovery request with SSH public key
- Receive node connection details
- SSH directly to nodes for validation/rental
Miner's perspective:
- Register nodes from config at startup
- Start gRPC server listening on configured port
- Wait for validator connections
- Verify validator signatures and authorization
- Deploy validator SSH keys to nodes automatically
- Return node details to authorized validators
- Monitor node health and update status
Proper SSH key management is critical for security and functionality.
The miner needs an SSH key to access your GPU nodes for key deployment.
# Generate Ed25519 key (recommended for security and performance)
ssh-keygen -t ed25519 -f ~/.ssh/miner_node_key -C "basilica-miner" -N ""
# Set proper permissions (critical for security)
chmod 600 ~/.ssh/miner_node_key
chmod 644 ~/.ssh/miner_node_key.pub
# Verify key was created
ls -la ~/.ssh/miner_node_key*Alternative: RSA key (if Ed25519 not supported):
ssh-keygen -t rsa -b 4096 -f ~/.ssh/miner_node_key -C "basilica-miner" -N ""The miner needs SSH access to deploy validator keys.
For each GPU node:
# Copy public key to node
ssh-copy-id -i ~/.ssh/miner_node_key.pub basilica@192.168.1.100
# Or manually:
cat ~/.ssh/miner_node_key.pub | ssh basilica@192.168.1.100 \
"mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"Verify access:
ssh -i ~/.ssh/miner_node_key basilica@192.168.1.100 "hostname && nvidia-smi --query-gpu=name --format=csv,noheader"Expected output:
gpu-node-1
NVIDIA H100 PCIe
On Miner Server (~/.ssh/config):
# Miner's SSH configuration for GPU nodes
Host gpu-node-*
User basilica
IdentityFile ~/.ssh/miner_node_key
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
ServerAliveInterval 60
ServerAliveCountMax 3
ConnectTimeout 30
On GPU Nodes (/etc/ssh/sshd_config):
# Security hardening
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
# Performance
MaxStartups 30:30:100
MaxSessions 100
# Keep connections alive
ClientAliveInterval 60
ClientAliveCountMax 3
After editing sshd_config:
sudo systemctl restart sshdOn each GPU node:
# Create user for validator access
sudo useradd -m -s /bin/bash basilica
# Add to docker group (for container workloads)
sudo usermod -aG docker basilica
# Optional: Add to sudo group (if validators need elevated privileges)
# sudo usermod -aG sudo basilica
# Set up SSH directory
sudo -u basilica mkdir -p /home/basilica/.ssh
sudo chmod 700 /home/basilica/.ssh
sudo -u basilica touch /home/basilica/.ssh/authorized_keys
sudo chmod 600 /home/basilica/.ssh/authorized_keys# Test as basilica user
sudo -u basilica nvidia-smi
# Test Docker GPU access
sudo -u basilica docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smiEnsure SSH port is accessible:
# Check SSH is running
sudo systemctl status sshd
# Allow SSH through firewall (if using UFW)
sudo ufw allow 22/tcp
sudo ufw enable
# For custom SSH port:
sudo ufw allow 2222/tcpVerify connectivity from miner server:
# From miner server
ssh -i ~/.ssh/miner_node_key basilica@<node-ip> "echo 'Connection successful'"Before adding nodes to miner config, verify:
- NVIDIA drivers installed (
nvidia-smiworks) - Docker installed and configured with NVIDIA runtime
- Dedicated user account created (
basilicaor similar) - User added to
dockergroup - SSH server running and accessible
- Miner's SSH public key deployed to node
- Firewall allows SSH connections from miner
- GPU accessible to Docker containers
- Sufficient disk space for containers/data
Use the configuration template provided in the config directory:
# Copy the example config
cp config/miner.toml.example miner.toml
# Edit with your settings
vim miner.toml[bittensor]
# Wallet configuration (path: ~/.bittensor/wallets/{wallet_name}/hotkeys/{hotkey_name})
wallet_name = "miner_wallet" # Your coldkey/wallet name
hotkey_name = "default" # Your hotkey name
# Network settings
network = "finney" # Options: finney (mainnet), test, local
netuid = 39 # Basilica subnet ID (39=mainnet, 387=testnet)
weight_interval_secs = 300 # Weight setting interval (5 minutes)
# Axon configuration (for Bittensor network registration)
external_ip = "<your public ip>" # YOUR SERVER'S PUBLIC IP
axon_port = 50051
# Advanced settings (usually don't need to change)
max_weight_uids = 256# Find your public IP
curl -4 ifconfig.me[database]
url = "sqlite:///opt/basilica/data/miner.db"
run_migrations = trueEnsure database directory exists:
sudo mkdir -p /opt/basilica/data
sudo chown $USER:$USER /opt/basilica/data[validator_comms]
host = "0.0.0.0" # Internal binding (0.0.0.0 = all interfaces)
port = 50051 # gRPC server port
request_timeout = 30 # Timeout in seconds# UFW
sudo ufw allow 50051/tcp
# Or iptables
sudo iptables -A INPUT -p tcp --dport 50051 -j ACCEPT[node_management]
# List your GPU compute nodes with SSH access details
nodes = [
{ host = "192.168.1.100", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 },
{ host = "192.168.1.101", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 },
]
health_check_interval = 60 # Health check interval in seconds
health_check_timeout = 10 # Health check timeout in seconds
max_retry_attempts = 3
auto_recovery = trueNode configuration fields:
host: IP address or hostname of GPU node (required)port: SSH port, typically 22 (required)username: SSH username on the node (required)hourly_rate_per_gpu: Hourly rental rate in USD per GPU (required)additional_opts(optional): Extra SSH options like"-o StrictHostKeyChecking=no"
[ssh_session]
# Path to your SSH private key for accessing nodes
miner_node_key_path = "~/.ssh/miner_node_key"
# Default username for SSH access to nodes (used as fallback)
default_node_username = "node"Verify key path is correct:
ls -la ~/.ssh/miner_node_key
# Should show: -rw------- (permissions 600)[security]
verify_signatures = true # ALWAYS true for production
# Optional: Ethereum private key for collateral contract (advanced)
# private_key_file = "/opt/basilica/keys/private_key.pem"[metrics]
enabled = true
[metrics.prometheus]
host = "127.0.0.1" # Bind to localhost for security
port = 9090Access metrics:
curl http://localhost:9090/metrics[validator_assignment]
strategy = "highest_stake" # Options: highest_stake, fixed_assignment
# Optional: Assign to specific validator (required for fixed_assignment)
# validator_hotkey = "5G3qVaXzKMPDm5AJ3dpzbpUC27kpccBvDwzSWXrq8M6qMmbC"Choosing a strategy:
- Production: Use
highest_staketo assign all nodes to the top validator - Fixed: Use
fixed_assignmentwith a specificvalidator_hotkeyfor known validators - Development: Use default
highest_stakewithout specific hotkey
Override auto-detected addresses for NAT/proxy scenarios:
[advertised_addresses]
# Only needed if miner is behind NAT/proxy
# grpc_endpoint = "http://203.0.113.45:50051"
# axon_endpoint = "http://203.0.113.45:50051"
# metrics_endpoint = "http://203.0.113.45:9090"Before starting the miner, validate your configuration:
# Validate configuration
cargo run -p basilica-miner -- --config miner.toml config validate
# Expected output:
# Configuration validation passedIf validation fails, check:
- Wallet names match your actual wallet
- External IP is correct
- Database directory exists
- SSH key paths are valid
- Node SSH credentials are accessible
Choose the deployment method that best fits your infrastructure.
Best for: Development, testing, simple setups.
# Clone repository
git clone https://github.com/your-org/basilica.git
cd basilica/basilica
# Build miner binary using build script
./scripts/miner/build.sh
# Binary will be at: ./basilica-minerBuild options:
# Release build (optimized, recommended for production)
./scripts/miner/build.sh --release# Create data directory
sudo mkdir -p /opt/basilica/data
sudo mkdir -p /opt/basilica/config
sudo chown -R $USER:$USER /opt/basilica
# Copy binary and config
sudo cp basilica-miner /opt/basilica/
sudo cp miner.toml /opt/basilica/config/
# Copy Bittensor wallet
sudo mkdir -p /root/.bittensor
sudo cp -r ~/.bittensor/wallets /root/.bittensor/
# Run miner
cd /opt/basilica
sudo ./basilica-miner --config config/miner.toml- SSH key deployment to nodes
- Database access (if in protected directory)
- Network port binding (if port < 1024)
Best for: Production deployments requiring auto-restart and logging.
# Create service file
sudo tee /etc/systemd/system/basilica-miner.service > /dev/null <<EOF
[Unit]
Description=Basilica Miner
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=root
WorkingDirectory=/opt/basilica
ExecStart=/opt/basilica/basilica-miner --config /opt/basilica/config/miner.toml
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=basilica-miner
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/basilica/data /var/log/basilica
[Install]
WantedBy=multi-user.target
EOF# Reload systemd
sudo systemctl daemon-reload
# Enable auto-start on boot
sudo systemctl enable basilica-miner
# Start service
sudo systemctl start basilica-miner
# Check status
sudo systemctl status basilica-miner
# View logs
sudo journalctl -u basilica-miner -f# Stop miner
sudo systemctl stop basilica-miner
# Restart miner
sudo systemctl restart basilica-miner
# Disable auto-start
sudo systemctl disable basilica-miner
# View logs (last 100 lines)
sudo journalctl -u basilica-miner -n 100
# View logs (follow in real-time)
sudo journalctl -u basilica-miner -fBest for: Isolated environments, easy updates, multi-host deployments.
# Build using provided Dockerfile
docker build -f scripts/miner/Dockerfile -t basilica-miner:latest .
# Or pull from registry (if available)
docker pull ghcr.io/your-org/basilica/miner:latest# Create required directories
sudo mkdir -p /opt/basilica/config
sudo mkdir -p /opt/basilica/data
# Copy configuration
sudo cp miner.toml /opt/basilica/config/
# Run miner container
docker run -d \
--name basilica-miner \
--restart unless-stopped \
-v ~/.bittensor:/root/.bittensor:ro \
-v /opt/basilica/config:/opt/basilica/config:ro \
-v /opt/basilica/data:/opt/basilica/data \
-v ~/.ssh:/root/.ssh:ro \
-p 50051:50051 \
-p 9090:9090 \
basilica-miner:latest --config /opt/basilica/config/miner.tomlVolume mappings explained:
~/.bittensor- Bittensor wallet (read-only)/opt/basilica/config- Miner configuration (read-only)/opt/basilica/data- Database and logs (read-write)~/.ssh- SSH keys for node access (read-only)
# View logs
docker logs -f basilica-miner
# Stop container
docker stop basilica-miner
# Start container
docker start basilica-miner
# Remove container
docker rm -f basilica-miner
# Exec into container (debugging)
docker exec -it basilica-miner /bin/bashBest for: Production deployments with automatic updates and monitoring.
# Navigate to miner scripts directory
cd scripts/miner
# Use provided production compose file
cp compose.prod.yml docker-compose.ymlOr create custom compose file:
version: '3.8'
services:
miner:
image: ghcr.io/your-org/basilica/miner:latest
container_name: basilica-miner
restart: unless-stopped
volumes:
- ~/.bittensor:/root/.bittensor:ro
- ./config/miner.toml:/opt/basilica/config/miner.toml:ro
- ./data:/opt/basilica/data
- ~/.ssh:/root/.ssh:ro
- /var/log/basilica:/var/log/basilica
ports:
- "50051:50051"
- "9090:9090"
command: ["--config", "/opt/basilica/config/miner.toml"]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9090/metrics"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
watchtower:
image: nickfedor/watchtower:1.14.0
container_name: basilica-watchtower
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: ["--cleanup", "--interval", "300", "basilica-miner"]# Ensure config exists
sudo cp miner.toml /opt/basilica/config/
# Start services
docker compose up -d
# View logs
docker compose logs -f miner
# Check status
docker compose psWatchtower automatically:
- Checks for image updates every 5 minutes
- Pulls new images when available
- Restarts containers with new images
- Cleans up old images
Understanding how validators access your nodes helps with troubleshooting and security.
Validator queries Bittensor metagraph:
# Validator perspective (simplified)
# Query metagraph for miners on subnet 39
validators_list = bittensor_service.get_neurons(netuid=39, filter="miners")
# Extract miner endpoints
for miner in validators_list:
miner_endpoint = miner.axon.ip + ":" + miner.axon.port
# miner_endpoint = "203.0.113.45:50051"Validator connects to miner's gRPC service:
# Validator connects to miner's gRPC endpoint
grpc_endpoint = "http://203.0.113.45:50051"
miner_client = MinerDiscoveryClient(grpc_endpoint)Validator authenticates with Bittensor signature:
# Validator generates signature
nonce = uuid4()
timestamp = int(time.time())
payload = f"BASILICA_AUTH_V1:{nonce}:{miner_hotkey}:{timestamp}"
signature = validator_keypair.sign(payload.encode())
# Send authentication request
auth_response = miner_client.authenticate_validator(
validator_hotkey=validator_hotkey,
signature=signature,
nonce=nonce,
timestamp=timestamp,
target_miner_hotkey=miner_hotkey
)
# Verify miner's signature in response
session_token = auth_response.session_tokenValidator sends SSH public key:
# Validator's persistent SSH key
ssh_public_key = load_public_key("~/.basilica/ssh/validator_persistent.pem.pub")
# Discover nodes with SSH key
discovery_response = miner_client.discover_nodes(
validator_hotkey=validator_hotkey,
validator_public_key=ssh_public_key,
signature=signature,
nonce=uuid4()
)
# Response contains node connection details
nodes = discovery_response.nodes
# [
# NodeConnectionDetails(node_id="a3f2b1c4-...", host="192.168.1.100", port="22", username="basilica"),
# NodeConnectionDetails(node_id="b4e3c2d5-...", host="192.168.1.101", port="22", username="basilica")
# ]Miner automatically deploys validator's SSH key to all nodes:
# Miner executes on each node (using miner's SSH key)
ssh -i ~/.ssh/miner_node_key basilica@192.168.1.100 << 'EOF'
mkdir -p ~/.ssh
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5... validator-5G3qVaXz...' >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
EOFValidator establishes direct SSH connection:
# Validator connects to node (using validator's SSH key)
ssh -i ~/.basilica/ssh/validator_persistent.pem basilica@192.168.1.100
# Validator can now:
# - Run GPU validation workloads
# - Execute container rentals
# - Monitor GPU statusOn miner:
// Node manager handles discovery request
async fn handle_discover_nodes(&self, request: DiscoverNodesRequest) -> Result<Response> {
// 1. Validate SSH public key format
self.validate_ssh_key(&request.validator_public_key)?;
// 2. Deploy key to all nodes
self.authorize_validator(
&request.validator_hotkey,
&request.validator_public_key
).await?;
// 3. Get node list
let nodes = self.list_nodes().await?;
// 4. Return connection details
Ok(ListNodeConnectionDetailsResponse { nodes })
}SSH key deployment:
async fn authorize_validator(&self, validator_hotkey: &str, ssh_key: &str) -> Result<()> {
for node in self.list_nodes().await? {
// Tag key with validator identifier
let key_entry = format!("{} validator-{}", ssh_key, validator_hotkey);
// Deploy via SSH
let ssh_command = format!(
"mkdir -p ~/.ssh && echo '{}' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys",
key_entry
);
self.ssh_client.execute_command(&node.connection_details, &ssh_command).await?;
}
Ok(())
}Validator authentication ensures:
- Only validators with valid Bittensor hotkeys can discover nodes
- Replay attacks prevented by nonce + timestamp
- Man-in-the-middle attacks prevented by target hotkey verification
- Mutual authentication (both parties verify each other)
SSH key management ensures:
- Only authorized validators' keys deployed to nodes
- Keys tagged with validator identity for audit/revocation
- Miner controls key deployment (nodes remain passive)
- Standard SSH security model applies
DO:
- Generate separate keys for miner and each service
- Use Ed25519 keys (faster, more secure than RSA)
- Set proper permissions (600 for private keys, 644 for public keys)
- Store keys outside of git repositories
- Rotate keys periodically (every 6-12 months)
- Use strong passphrases for keys (if not automated)
DON'T:
- Reuse SSH keys across services
- Store private keys in containers
- Share private keys between systems
- Use weak key types (DSA, RSA <2048 bits)
- Leave default permissions (world-readable)
# Edit /etc/ssh/sshd_config
sudo vim /etc/ssh/sshd_configAdd/modify:
# Disable root login
PermitRootLogin no
# Disable password authentication
PasswordAuthentication no
ChallengeResponseAuthentication no
# Only allow public key authentication
PubkeyAuthentication yes
# Limit authentication attempts
MaxAuthTries 3
# Limit concurrent connections
MaxStartups 30:30:100
MaxSessions 100
# Disable forwarding (if not needed)
AllowTcpForwarding no
X11Forwarding no
# Log authentication attempts
LogLevel VERBOSE
Restart SSH:
sudo systemctl restart sshdOn miner server:
# Allow only necessary ports
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp # SSH (restrict to known IPs in production)
sudo ufw allow 50051/tcp # gRPC/Axon (for validators and Bittensor network)
sudo ufw allow 9090/tcp # Metrics (optional, can be localhost-only)
sudo ufw enableOn GPU nodes:
# Allow SSH from miner only
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from <MINER_IP> to any port 22
sudo ufw enableFor production, consider enabling TLS:
[validator_comms]
tls_enabled = true
tls_cert_path = "/opt/basilica/certs/server.crt"
tls_key_path = "/opt/basilica/certs/server.key"Generate self-signed certificate:
openssl req -x509 -newkey rsa:4096 -nodes \
-keyout /opt/basilica/certs/server.key \
-out /opt/basilica/certs/server.crt \
-days 365 -subj "/CN=miner.basilica.local"Production recommendation:
[validator_assignment]
strategy = "highest_stake"Benefits:
- Automatically selects the most invested validator
- Reduces attack surface
- Simplifies operations and monitoring
- No manual validator tracking required
Track deployed validator keys:
# On GPU node, view authorized_keys
cat ~/.ssh/authorized_keys | grep validator-
# Sample output:
# ssh-ed25519 AAAAC3... validator-5G3qVaXzKMP...
# ssh-ed25519 AAAAC3... validator-5FHd7oPqk...Audit SSH connections:
# View SSH access logs
sudo grep 'Accepted publickey' /var/log/auth.log | tail -20
# View by user
sudo grep 'Accepted publickey for basilica' /var/log/auth.logSet up alerts for:
- Failed authentication attempts
- Unauthorized SSH access
- Unexpected configuration changes
- Database anomalies
- Node connectivity issues
Example: Alert on authentication failures
# Create alert script
sudo tee /opt/basilica/scripts/auth-alert.sh > /dev/null << 'EOF'
#!/bin/bash
FAILED_LOGINS=$(grep "Failed publickey" /var/log/auth.log | wc -l)
if [ $FAILED_LOGINS -gt 10 ]; then
echo "WARNING: $FAILED_LOGINS failed SSH attempts detected" | \
mail -s "Security Alert: SSH Failures" admin@yourdomain.com
fi
EOF
chmod +x /opt/basilica/scripts/auth-alert.sh
# Run hourly via cron
echo "0 * * * * /opt/basilica/scripts/auth-alert.sh" | crontab -Critical data to backup:
- Bittensor wallet (
~/.bittensor/wallets/) - Miner configuration (
/opt/basilica/config/) - SSH keys (
~/.ssh/miner_node_key*) - Database (
/opt/basilica/data/miner.db)
Backup script:
#!/bin/bash
BACKUP_DIR="/backup/basilica/$(date +%Y%m%d)"
mkdir -p $BACKUP_DIR
# Backup wallet
cp -r ~/.bittensor/wallets $BACKUP_DIR/
# Backup config
cp /opt/basilica/config/miner.toml $BACKUP_DIR/
# Backup SSH keys
cp ~/.ssh/miner_node_key* $BACKUP_DIR/
# Backup database
sqlite3 /opt/basilica/data/miner.db ".backup $BACKUP_DIR/miner.db"
# Encrypt backup
tar -czf - $BACKUP_DIR | gpg --encrypt --recipient admin@yourdomain.com > $BACKUP_DIR.tar.gz.gpg
echo "Backup completed: $BACKUP_DIR.tar.gz.gpg"Keep system updated:
# System updates
sudo apt update && sudo apt upgrade -y
# Miner updates
cd /opt/basilica
./scripts/miner/build.sh --release
sudo systemctl restart basilica-minerFor Docker deployments, Watchtower handles automatic updates.
Protect against DoS:
# Limit SSH connection rate with iptables
sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set
sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROPApplication-level rate limiting (future enhancement):
[validator_comms.rate_limit]
enabled = true
requests_per_second = 10
burst_capacity = 20Check miner health:
# Prometheus metrics endpoint (health check)
curl http://localhost:9090/metrics | grep basilica_miner
# Check gRPC server is responding
grpcurl -plaintext localhost:50051 listCheck database connectivity:
# Run miner health check
./basilica-miner --config miner.toml health-check
# Or via database CLI command
./basilica-miner --config miner.toml database healthPrometheus metrics endpoint:
# Access metrics
curl http://localhost:9090/metrics
# Sample metrics:
# basilica_miner_node_count 3
# basilica_miner_validator_connections_total 12
# basilica_miner_ssh_deployments_total 45
# basilica_miner_authentication_requests_total 120Grafana dashboard (if available):
https://basilica-grafana.tplr.ai/
View miner logs:
# Systemd service
sudo journalctl -u basilica-miner -f
# Docker container
docker logs -f basilica-miner
# Binary (if logging to file)
tail -f /opt/basilica/miner.logImportant log patterns to monitor:
# Authentication events
grep "Successfully authenticated validator" /opt/basilica/miner.log
# Node registration events
grep "Registered node" /opt/basilica/miner.log
# SSH key deployment events
grep "Deploying SSH key for validator" /opt/basilica/miner.log
# Errors
grep "ERROR" /opt/basilica/miner.logCheck node connectivity:
# Test SSH access to all nodes
for node in 192.168.1.100 192.168.1.101; do
echo "Testing $node..."
ssh -i ~/.ssh/miner_node_key basilica@$node "nvidia-smi --query-gpu=name,utilization.gpu --format=csv,noheader"
doneMonitor GPU utilization:
# Create monitoring script
cat > /opt/basilica/scripts/monitor-gpus.sh << 'EOF'
#!/bin/bash
while true; do
echo "=== GPU Status $(date) ==="
ssh -i ~/.ssh/miner_node_key basilica@192.168.1.100 nvidia-smi --query-gpu=index,name,utilization.gpu,utilization.memory,temperature.gpu --format=csv
sleep 60
done
EOF
chmod +x /opt/basilica/scripts/monitor-gpus.sh
./opt/basilica/scripts/monitor-gpus.shKey metrics to track:
- Node availability: Percentage of time nodes are accessible
- Validator requests: Number of authentication/discovery requests
- SSH deployments: Number of successful key deployments
- Response times: gRPC endpoint latency
- Error rates: Failed authentications, SSH failures
- Database performance: Query times, connection pool usage
Validators apply a linear uptime multiplier that reaches 1.0 after roughly 14 days of uninterrupted service and resets when downtime exceeds the validator tolerance window. For detailed formulas, examples, and operational guidance, see Node Uptime Ramp-up (Incentives Ramp-up).
Validators actively track executor misbehaviour to protect rentals. Each miner/executor pair has an independent ban state backed by persistent storage.
| Repeated issue | Window | Threshold | Ban duration (baseline) |
|---|---|---|---|
| Light or full validation failures | 1 hour | 2 failures | 30 minutes |
| Any misbehaviour | 6 hours | 3 failures | 12 hours |
| Any misbehaviour | 12 hours | 3 failures | 24 hours |
| Any misbehaviour | 48 hours | 3 failures | 3 days |
| Any misbehaviour | 7 days | 3 failures | 7 days |
- Validation failures (lightweight or full)
- Deployment or startup failures during rentals
- Executor health checks reporting unhealthy state
- Connection errors caused by the miner misrouting a validator
- Banned executors are excluded from discovery/rental routing
- Active bans surface in validator logs and Prometheus metrics
- Validation requests return a specific
executor_bannederror to the miner - When a ban expires, validators automatically clear it; miners can attempt deployments again
- Fix the root cause (ensure executor is reachable, healthy, and has compatible drivers/container images)
- Confirm the ban timer via validator metrics (
validator_executor_ban_active_status) - Wait for the ban duration to elapse (no manual action needed for standard bans)
- After expiry, monitor deployment logs to verify validators reconnect successfully
- Keep executors patched (CUDA, drivers, containers) and aligned with validator expectations
- Automate health checks and restart loops so degraded nodes self-heal quickly
- Validate images locally before exposing them to validators
- Enforce network/firewall rules to avoid intermittent reachability
- Maintain enough GPU capacity to handle assigned validators without overcommit
- Document procedures: who on your team handles ban remediation, monitoring, and redeploys
Error: unable to open database file
Solution:
# Ensure database directory exists
sudo mkdir -p /opt/basilica/data
sudo chown $USER:$USER /opt/basilica/data
# Check database URL in config
# Should be: url = "sqlite:///opt/basilica/data/miner.db"Error: Failed to load hotkey: Invalid format
Solution:
# Verify wallet exists
ls ~/.bittensor/wallets/miner_wallet/hotkeys/default
# Check wallet_name and hotkey_name in config match filesystem
# wallet_name = "miner_wallet"
# hotkey_name = "default"
# Verify wallet format (should be JSON with secretPhrase)
cat ~/.bittensor/wallets/miner_wallet/hotkeys/defaultError: Address already in use (os error 98)
Solution:
# Check what's using the port
sudo lsof -i :50051
# Kill process or change port in config
# [validator_comms]
# port = 50052Error: Failed to connect to node 192.168.1.100: Permission denied (publickey)
Solution:
# Verify miner's public key is on node
ssh basilica@192.168.1.100 'cat ~/.ssh/authorized_keys | grep miner_node_key'
# If not present, deploy it
ssh-copy-id -i ~/.ssh/miner_node_key.pub basilica@192.168.1.100
# Test connection
ssh -i ~/.ssh/miner_node_key basilica@192.168.1.100 'echo "Connection successful"'Error: Connection to 192.168.1.100:22 timed out
Solution:
# Check network connectivity
ping 192.168.1.100
# Check SSH port is open
nc -zv 192.168.1.100 22
# Check firewall on node
ssh basilica@192.168.1.100 'sudo ufw status'
# Allow SSH from miner IP
ssh basilica@192.168.1.100 'sudo ufw allow from <MINER_IP> to any port 22'Error: Permissions 0644 for '~/.ssh/miner_node_key' are too open
Solution:
# Fix key permissions
chmod 600 ~/.ssh/miner_node_key
chmod 644 ~/.ssh/miner_node_key.pubWARN: No validators found matching criteria
Solution:
# Check validator assignment config
# [validator_assignment]
# strategy = "highest_stake"
# validator_hotkey = "..." (optional)
# For testing with a specific validator, use fixed_assignment:
# [validator_assignment]
# strategy = "fixed_assignment"
# validator_hotkey = "5G3qVaXz..."
# Restart miner
sudo systemctl restart basilica-minerERROR: Validator authentication failed: Invalid signature
Solution:
# Verify security config
# [security]
# verify_signatures = true
# For local testing only:
# verify_signatures = false # NEVER use in production
# Check timestamp freshness (should be within 5 minutes)
# Ensure system clocks are synchronized
sudo timedatectl set-ntp trueWARN: No nodes registered - miner will not be able to serve validators
Solution:
# Check node_management config
# [node_management]
# nodes = [
# { host = "192.168.1.100", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 }
# ]
# Verify SSH access to each node
for node in $(grep 'host = ' miner.toml | cut -d'"' -f2); do
echo "Testing $node..."
ssh -i ~/.ssh/miner_node_key basilica@$node 'hostname'
doneERROR: Failed to generate node ID for basilica@192.168.1.100:22
Solution:
# This usually indicates SSH credentials are invalid
# Verify each field in node config:
# - host: Must be accessible IP/hostname
# - port: Must be correct SSH port
# - username: Must exist on node
# Test connection
ssh -p 22 -i ~/.ssh/miner_node_key basilica@192.168.1.100Error: Failed to register on network: Insufficient funds
Solution:
# Check wallet balance
btcli wallet balance --wallet.name miner_wallet
# Ensure sufficient TAO for registration fee
# Transfer TAO to wallet if needed
# Check if already registered
btcli subnet list --netuid 39 | grep <YOUR_HOTKEY>Error: Metadata error: the generated code is not compatible with the node
Solution:
# Regenerate metadata
./scripts/generate-metadata.sh --network finney
# Rebuild miner
./scripts/miner/build.sh --release
# Restart miner
sudo systemctl restart basilica-miner# Run miner with verbose logging
./basilica-miner --config miner.toml -vvv
# Or set in config (not recommended for production)
# [logging]
# level = "debug"# Validate config before starting
./basilica-miner --config miner.toml config validate
# Show effective configuration
./basilica-miner --config miner.toml config show# Test SSH connectivity
./basilica-miner --config miner.toml service test-ssh
# Test database connection
./basilica-miner --config miner.toml database health
# Test Bittensor connection
./basilica-miner --config miner.toml service test-bittensor# Check miner's gRPC endpoint
grpcurl -plaintext localhost:50051 list
# Test from external (validator perspective)
grpcurl -plaintext <MINER_PUBLIC_IP>:50051 list
# Check metrics endpoint
curl http://localhost:9090/metrics | grep basilica_minerIf you're still experiencing issues:
- Check logs for detailed error messages
- Search GitHub issues for similar problems
- Join Discord for community support
- Create GitHub issue with:
- Miner version
- Configuration (redacted sensitive info)
- Full error logs
- Steps to reproduce
You can implement custom assignment strategies by creating a new strategy type:
// Example: Geographic assignment strategy
pub struct GeographicAssignment {
preferred_region: String,
}
#[async_trait]
impl AssignmentStrategy for GeographicAssignment {
async fn select_validators(
&self,
validators: Vec<ValidatorInfo>,
nodes: Vec<RegisteredNode>,
) -> Result<Vec<(ValidatorInfo, Vec<RegisteredNode>)>> {
// Filter validators by region
let regional_validators: Vec<_> = validators
.into_iter()
.filter(|v| self.is_in_region(v))
.collect();
// Assign nodes to regional validators
// ... implementation
}
}For geo-distributed GPU nodes:
[node_management]
nodes = [
# US East
{ host = "us-east-1.example.com", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 },
{ host = "us-east-2.example.com", port = 22, username = "basilica", hourly_rate_per_gpu = 2.50 },
# EU West
{ host = "eu-west-1.example.com", port = 22, username = "basilica", hourly_rate_per_gpu = 2.80 },
{ host = "eu-west-2.example.com", port = 22, username = "basilica", hourly_rate_per_gpu = 2.80 },
# Asia Pacific
{ host = "ap-south-1.example.com", port = 22, username = "basilica", hourly_rate_per_gpu = 2.20 },
]Considerations:
- Network latency between miner and nodes
- Regional compliance requirements
- Validator geographic distribution
- Cost optimization
For enhanced security:
#!/bin/bash
# Rotate miner SSH key monthly
OLD_KEY=~/.ssh/miner_node_key
NEW_KEY=~/.ssh/miner_node_key.new
# Generate new key
ssh-keygen -t ed25519 -f $NEW_KEY -N "" -C "basilica-miner-$(date +%Y%m)"
# Deploy new key to all nodes
for node in $(grep 'host = ' /opt/basilica/config/miner.toml | cut -d'"' -f2); do
ssh-copy-id -i $NEW_KEY.pub basilica@$node
done
# Update miner config
sed -i "s|miner_node_key|miner_node_key.new|g" /opt/basilica/config/miner.toml
# Restart miner
sudo systemctl restart basilica-miner
# Remove old key after verification
sleep 60
rm -f $OLD_KEY $OLD_KEY.pub
mv $NEW_KEY $OLD_KEY
mv $NEW_KEY.pub $OLD_KEY.pubOptimize SSH connections:
# ~/.ssh/config
Host gpu-*
ControlMaster auto
ControlPath ~/.ssh/control-%r@%h:%p
ControlPersist 10m
Compression yes
ServerAliveInterval 60
Optimize database:
[database]
max_connections = 50 # Increase for high validator traffic
connection_timeout = 10 # Faster timeout for quicker failoverSystem tuning:
# Increase file descriptor limits
sudo tee -a /etc/security/limits.conf << EOF
root soft nofile 65536
root hard nofile 65536
EOF
# Apply without reboot
ulimit -n 65536- Deploy your first miner following this guide
- Test validator connectivity and monitor initial interactions
- Set up monitoring with Prometheus/Grafana
- Join the community for support and updates
- Review the Validator Guide to understand how your miner is evaluated
- GitHub Repository: https://github.com/one-covenant/basilica
- Discord: https://discord.gg/Cy7c9vPsNK
- Website: https://www.basilica.ai/
- Validator Guide: docs/validator.md
- Architecture Overview: docs/architecture.md
- API Documentation: docs/api.md
Happy Mining!