1. Initial Setup

A guide for setting up validators, blazingly fast. I have been playing around with different systems, actively switching validator machines, so this acts as a guide for me personally. This is an opinionated guide that covers how to setup a validator, the hardware requirements and settings I like to tune.

Basic Overview

System recommendation refer to Solanahcl list by ferric / StakeWare

Three or more disks are required with the following configuration:

SSD primary OS (~500 GB)
NVMe Ledger (≥2TB)
NVMe Accounts and snapshot (≥2TB)

Base OS: Ubuntu 22.04

1. Initial Setup

Update & Upgrade the System

Ensure your system is up to date with the latest security patches and packages.

sudo apt update && sudo apt upgrade

Check Disk Mount Status

Before proceeding, verify that your disks are properly recognized and mounted:

df -h           # View mounted filesystems and usage
lsblk -f        # Display block devices and mount points

Disk Setup

Directory structure:

Ledger Disk → /mnt/ledger
Account & Snapshot Disk → /mnt/extras
- /mnt/extras/snapshot (For Snapshots)
- /mnt/extras/accounts (For Accounts)

Setup Steps

create sol user

sudo adduser sol

Format the blocks

sudo mkfs -t ext4 /dev/nvme0n1
sudo mkfs -t ext4 /dev/nvme1n1

// with noatime
sudo mount -t xfs -o defaults,noatime,logbufs=8 /dev/nvme2n1 /mnt/ledger

Create directories

sudo mkdir -p /mnt/ledger /mnt/extras/snapshot /mnt/extras/accounts

Spin up directory + give sol user permission

sudo chown -R sol:sol <PATH TO DIR>

Mount to the directory

sudo mount /dev/nvme0n1 <PATH TO DIR>

Add the UUIDs of the disks to the fstab file to retain the mount points after reboots:

sudo nano /etc/fstab

#add the following lines to the fstab file
UUID="COPY THE UUID FROM THE OUTPUT OF lsblk -f" /mnt/ledger ext4 defaults 0 0
UUID="COPY THE UUID FROM THE OUTPUT OF lsblk -f" /mnt/extras/snapshot ext4 defaults 0 0
UUID="COPY THE UUID FROM THE OUTPUT OF lsblk -f" /mnt/extras/accounts ext4 defaults 0 0

Ports Opening

Note: RPC port remains closed, only SSH and gossip ports are opened.

For new machines with UFW disabled:

Add OpenSSH first to prevent lockout if you don't have password access: sudo ufw allow OpenSSH
Open required ports:

sudo ufw allow 8000:8020/tcp

sudo ufw allow 8000:8020/udp

2. System Tuning for best performance

System Performance Optimization

Kernel and Network Tuning

Create and run the following script to optimize system performance:

#!/bin/bash

# Set sysctl performance variables
cat >> /etc/sysctl.conf <<- EOM
# TCP Buffer Sizes (10k min, 87.38k default, 12M max)
net.ipv4.tcp_rmem=10240 87380 12582912
net.ipv4.tcp_wmem=10240 87380 12582912

# TCP Optimization
net.ipv4.tcp_congestion_control=westwood
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=1
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_no_metrics_save=1
net.ipv4.tcp_moderate_rcvbuf=1

# Kernel Optimization
kernel.timer_migration=0
kernel.hung_task_timeout_secs=30
kernel.pid_max=49152

# Virtual Memory Tuning
vm.swappiness=30
vm.max_map_count=2000000
vm.stat_interval=10
vm.dirty_ratio=40
vm.dirty_background_ratio=10
vm.min_free_kbytes=3000000
vm.dirty_expire_centisecs=36000
vm.dirty_writeback_centisecs=3000
vm.dirtytime_expire_seconds=43200

# Solana Specific Tuning
net.core.rmem_max=134217728
net.core.rmem_default=134217728
net.core.wmem_max=134217728
net.core.wmem_default=134217728
EOM

# Reload sysctl settings
sysctl -p

# Set CPU governor to performance mode
echo 'GOVERNOR="performance"' | tee /etc/default/cpufrequtils
echo "performance" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Set performance governor for bare metal (ignore errors)
echo "performance" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor || true

Session File Limits

Choose one of the following configurations:

Service-specific limits in /etc/systemd/system.conf:

[Service]
LimitNOFILE=1000000

System-wide limits in /etc/systemd/system.conf:

[Manager]
DefaultLimitNOFILE=1000000

Disable swap

Swap can negatively affect validator performance.

Turn off swap

sudo swapoff -a #disables swap
sudo rm /swapfile #removes swap file

Ensure swap is disabled

btop
# or
free -h

3. Validator Setup

Installing rust and other dependencies

Install useful rust tools on the root system

sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang cmake make libprotobuf-dev protobuf-compiler

Switch to the sol user

sudo su - sol

Install rust on the sol user

curl https://sh.rustup.rs -sSf | sh
source $HOME/.cargo/env
rustup component add rustfmt
rustup update

Installing Agave/Jito Client

Clone the Agave/Jito client repository:

git clone https://github.com/jito-foundation/jito-solana.git --recurse-submodules

Navigate to the jito/agave directory

cd jito-solana

Git checkout to the version you want to user and install submodules, eg: v2.2.14-jito

git checkout tags/$VERSION
git submodule update --init --recursive

CI_COMMIT=$(git rev-parse HEAD) scripts/cargo-install-all.sh --validator-only ~/.local/share/solana/install/releases/"$VERSION"

Post-Installation Setup

Create symlink for Jito client (if used):

ln -sf /home/sol/.local/share/solana/install/releases/$VERSION/bin /home/sol/.local/share/solana/install/active_release

Grant execution permissions to the install script:

chmod +x bin/start.sh

Add the start script to this file, Use the start script here, specifically configured for a voting validator node. Note that the configuration includes modifications to support RPC functionality. additional flag for RPC node here
Add the following to your .bashrc or .bash_profile:

# Environment Setup
. "$HOME/.cargo/env"
export PATH="/home/sol/.local/share/solana/install/active_release/bin:$PATH"

# Helpful Aliases
alias catchup='solana catchup --our-localhost'
alias monitor='agave-validator --ledger /mnt/ledger monitor'
alias logtail='tail -f /home/sol/logs/solana-validator.log'

Additional Resources

Installation script source: ice-staking repository

4. Service Management

Solana validator service

Create a systemd service file for the Solana validator:

[Unit]
Description=Solana Validator
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=sol
LimitNOFILE=1000000
LogRateLimitIntervalSec=0
Environment="SOLANA_METRICS_CONFIG=host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password"
Environment="PATH=/home/sol/bin:/home/sol/.local/share/solana/install/active_release/bin:/home/sol/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
ExecStart=/home/sol/bin/ice-staking/start/start.sh mainnet-beta

[Install]
WantedBy=multi-user.target

Service Management Commands

Start Service

sudo systemctl enable --now sol

Stop Service

sudo systemctl stop sol

Restart Service

sudo systemctl restart sol

5. Monitoring Setup (optional)

We'll be using Betterstack Heartbeat for tracking uptime and incident reporting. Create a new Heartbeat and copy its url,

Clone the github repo

git clone https://github.com/brewlabshq/arise-status.git

Checkout to the latest release

git checkout tags/v0.1.0    #replace with the latest version

Setup the .env file

cp .env.example .env
nano .env

Update the .env file with the arise heartbeat url and name

PING_URL="https://arise-heartbeat-url"
SERVICE_NAME="My validator Mainnet"

Build and run

cargo build -r 
cargor run -r

After successful setup you can see a log "RPC Alive, status posted successfully".

We'll be creating the service for arise monitoring so that the monitoring stays alive even if the system reboots

Arise monitoring service

Create a new service file for arise

[Unit]
Description=Arise Heartbeat
After=network.target
StartLimitIntervalSec=0


[Service]
Type=simple
Restart=always
RestartSec=1
User=sol
LogRateLimitIntervalSec=0
Environment="PATH=/home/sol/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
ExecStart=/home/sol/arise-status/start.sh


[Install]
WantedBy=multi-user.target

Enable the service

sudo systemctl enable arise --now

Confirm the service is running successfully

sudo systemctl status arise

Hot-Swap Validator Setup Guide

Overview

This guide describes how to set up two servers for hot-swapping to maintain 100% uptime during system changes. The process follows the Identity Transition methodology by Pumpkin.

Identity Keypair Configuration

Required Keypairs

Unstaked Keypair (unstaked.json)
- Functions as a burner keypair
- Maintains zero SOL balance to prevent voting capabilities
Staked Keypair (staked.json)
- Serves as the primary staked keypair
- Used for validator transitions when needed

Transferring Keypairs

Transfer the keypairs to your validator server using SCP:

scp <source_files> ice-ams:

Note: Customize the SSH configuration according to your setup. Ensure proper permissions are set for the sol user after transfer.

Log Rotation Configuration

Create and implement log rotation for validator logs:

cat > logrotate.sol <<EOF
/home/sol/logs/solana-validator.log {
    rotate 7
    daily
    missingok
    postrotate
        systemctl kill -s USR1 sol.service
    endscript
}
EOF

sudo cp logrotate.sol /etc/logrotate.d/sol
systemctl restart logrotate.service

Create and implement log rotation for Arise Monitoring logs:

cat > logrotate.arise <<EOF
/home/sol/arise-output.log {
    rotate 7
    hourly
    missingok
    postrotate
        systemctl kill -s USR1 arise.service
    endscript
}
EOF

sudo cp logrotate.arise /etc/logrotate.d/arise
systemctl restart logrotate.service

After this check the log file snapshot download should have started

logtail #custom alias for tail -f logs/solana-validator.logs

Solana Validator Operations Guide

Metrics & Monitoring Solutions

1. Built-in Dashboard Options

Solana Metrics Dashboard
- Official solution from Solana Labs
- Access via URL specified in service file
- Provides real-time validator performance metrics

2. Third-Party Solutions

Stakeconomy's SolanaMonitoring
- Repository: github.com/stakeconomy/solanamonitoring
- Community-maintained monitoring solution
- Features:
  - Performance tracking
  - Health checks

3. Custom Monitoring Stack

Grafana + InfluxDB Setup
- Fully customizable metrics visualization
- Time-series data storage
- Benefits:
  - Custom dashboards
  - Historical data analysis

Active Monitoring Tools

1. Solana Watcher

Official monitoring tool by Solana Labs
Documentation: docs.solanalabs.com/operations/best-practices/monitoring
Features:
- Automated health checks
- System alerts

2. Stakewiz Update bot Integration

Telegram notification system
Real-time alerts and updates

Security Best Practices

1. Firewall Configuration

Only open required ports
Implement port-specific rules
Regular audit of open ports
Use UFW (Uncomplicated Firewall) for simple management

2. User Management

✅ Run validator with non-root user
❌ Avoid running as root
❌ Validator user should not have sudo privileges
Create a dedicated service account for validator operations

3. SSH Security

Disable password authentication
Use SSH keys exclusively
Consider:
- Custom SSH port
- Key-based authentication only
- Rate limiting for failed attempts

4. Keypair Security

Secure storage of validator keypairs
Best practices:
- Encrypted backups
- Access control logs

Credits

Solana Labs / docs
Tim Garcia / youtube
Overclock / setup guide
Ferric / StakeWare
Pumpkin's Pool / Pumpkin's pool

Name		Name	Last commit message	Last commit date
Latest commit History 28 Commits
.github/workflows		.github/workflows
LICENSE		LICENSE
README.md		README.md
_config.yml		_config.yml

License

brewlabshq/validator-jumpstart

Folders and files

Latest commit

History

Repository files navigation

Basic Overview

1. Initial Setup

Update & Upgrade the System

Check Disk Mount Status

Disk Setup

Setup Steps

Ports Opening

2. System Tuning for best performance

System Performance Optimization

Kernel and Network Tuning

Session File Limits

Disable swap

3. Validator Setup

Installing rust and other dependencies

Installing Agave/Jito Client

Post-Installation Setup

Additional Resources

4. Service Management

Solana validator service

Service Management Commands

Start Service

Stop Service

Restart Service

5. Monitoring Setup (optional)

Arise monitoring service

Hot-Swap Validator Setup Guide

Overview

Identity Keypair Configuration

Required Keypairs

Transferring Keypairs

Log Rotation Configuration

Solana Validator Operations Guide

Metrics & Monitoring Solutions

1. Built-in Dashboard Options

2. Third-Party Solutions

3. Custom Monitoring Stack

Active Monitoring Tools

1. Solana Watcher

2. Stakewiz Update bot Integration

Security Best Practices

1. Firewall Configuration

2. User Management

3. SSH Security

4. Keypair Security

Credits

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Packages