Skip to content

Access Log Analytics

Jump to bottom Edit New page
EvgeniiMekhanik edited this page Jan 9, 2026 · 14 revisions

Tempesta FW uses an mmap()ed memory area to deliver client access logs to the user-space in a zero-copy fashion. tfw_logger is a user-space daemon spawning a worker thread per a CPU core and gathers access log records from the mmap()ed area with no locks and no copying. The received from the kernel access log records are grouped and sent to the ClikHouse database. Thanks to the batches, ClickHouse can absorb millions of records per second on a modest hardware node.

ClickHouse is a powerful analytical (column-oriented DBMS) for online analytical processing (OLAP). We have chosen ClickHouse because

The ClickHouse advanced analytic queries over the stored data facilitate web performance analytics and security incidents (e.g. application DDoS attacks) response.

Architecture

HTTP Request → Tempesta FW → mmap buffer → tfw_logger → ClickHouse
                    ↓
                /dev/tempesta_mmap_log
                    ↓
            [Worker Thread 1] [Worker Thread 2] ... [Worker Thread N]
                    ↓
                ClickHouse Database

Always keep your ClickHouse database separate from the edge servers that may come under a DDoS attack - this ensures you can analyze the incident in near real time.

Configuration

Tempesta FW Configuration

For the current development version 0.9 add to your tempesta_fw.conf:

access_log mmap logger_config=/path/to/tfw_logger.json;

For the stable version 0.8 use:

access_log mmap mmap_host=localhost mmap_log=/var/log/tempesta_access.log;

Do not use dmesg for access_log! This can lead to kernel hung under heavy load and should used only for debug purposes!

When using tfw_logger, you can configure the size of the per-CPU memory-mapped buffers used for storing access log events before they are transmitted to external storage:

mmap_log_buffer_size <SIZE>;

SIZE specifies the buffer size in bytes for each CPU. The value must be a power of 2 and a multiple of 4KB (page size). The allowed range is from 4KB to 128MB. Defaults: 1M.

Examples:

mmap_log_buffer_size 4M;
mmap_log_buffer_size 512K;
mmap_log_buffer_size 16M;

Larger buffer sizes can improve performance under high load by reducing the frequency of buffer flushes to external storage, but will consume more memory. The optimal size depends on your traffic patterns and available system memory.

tfw_logger Configuration

0.9 (current)

Create a separate JSON configuration file (e.g., /etc/tempesta/tfw_logger.json):

{
    "log_path": "/var/log/tempesta/tfw_logger.log",
    "access_log": {
        "plugin_path": "/opt/tempesta/access_log.so",
        "host": "localhost",
        "port": 9000,
        "user": "tempesta_user",
        "password": "secure_password",
        "db_name": "default",
        "table_name": "access_log",
        "max_events": 10000,
    }
}

0.8 (stable)

In 0.8 tfw_logger is configured by attributes for the access_log configuration option in the main Tempesta FW configuration tempesta_fw.conf:

access_log mmap mmap_host=localhost mmap_usser=tempesta_user mmap_password=secure_password mmap_log=/var/log/tempesta_access.log;

Configuration Options

Option Description Default Example
log_path Path to tfw_logger log file /var/log/tempesta/tfw_logger.log
plugin_path Path to access logger plugin - /opt/tempesta/access_log.so
access_log.host ClickHouse server hostname localhost clickhouse.example.com
access_log.port ClickHouse native protocol port 9000 9000
access_log.db_name ClickHouse database name default custom_default
access_log.table_name ClickHouse table name access_log custom_access_log
access_log.user ClickHouse username (optional) default tempesta_user
access_log.password ClickHouse password (optional) - secure_password
access_log.max_events Batch size for inserts 1000 500

Buffer Size Guidelines

  • Small deployments: 4MB (4194304)
  • Medium traffic: 16MB (16777216)
  • High traffic: 64MB+ (67108864)
  • Enterprise: 256MB+ (268435456)

Buffer size must be multiple of page size and ≥4096 bytes.

Access Log Schema

tfw_logger creates the following ClickHouse table structure:

CREATE TABLE IF NOT EXISTS access_log.access_log (
    timestamp DateTime64(3, 'UTC'),
    address IPv6,
    method UInt8,
    version UInt8,
    status UInt16, 
    response_content_length UInt64,
    response_time UInt32,
    vhost String,
    uri String,
    referer String,
    user_agent String,
    tft UInt64,
    tfh UInt64,
    dropped_events UInt64
) ENGINE = MergeTree()
ORDER BY timestamp;

Field Descriptions

Field Type Description
timestamp DateTime64(3) Request timestamp with millisecond precision
addr IPv6 Client IP address (IPv4 mapped to IPv6)
method UInt8 HTTP method (GET=1, POST=2, etc.)
version UInt8 HTTP method (GET=1, POST=2, etc.)
status UInt16 HTTP response status code
response_content_length UInt64 Response content length in bytes
response_time UInt32 Response time in milliseconds
vhost String Host header value
uri String Request URI path and query
referer String Referer header
user_agent String User-Agent header
tft UInt64 TF TLS hash
tfh UInt64 TF HTTP hash
dropped_events UInt64 Number of dropped events (monitoring)

Field method is a numerical value (see tfw_http_meth_t in http.h):

1: COPY
2: DELETE
3: GET
4: HEAD
5: LOCK
6: MKCOL
7: MOVE
8: OPTIONS
9: PATCH
10: POST
11: PROPFIND
12: PROPPATCH
13: PUT
14: TRACE
15: UNLOCK
16: PURGE
17: UNKNOWN

Field version is also a numerical value:

0: INVALID
1: HTTP 0.9
2: HTTP 1.0
3: HTTP 1.1
4: HTTP 2

Installation and Usage

1. Build tfw_logger 

cd tempesta/logger
make build

2. Create Configuration (0.9 only)

# Generate default configuration
./tfw_logger --generate --config /etc/tempesta/tfw_logger.json

# Edit configuration as needed
sudo nano /etc/tempesta/tfw_logger.json

3. Setup ClickHouse

-- Create database and user
CREATE DATABASE access_log;
CREATE USER tempesta_user IDENTIFIED BY 'secure_password';
GRANT ALL ON access_log.* TO tempesta_user;

4. Configure Tempesta FW

Add to tempesta_fw.conf FOR 0.9:

access_log mmap logger_config=/etc/tempesta/tfw_logger.json;

or for 0.8:

access_log mmap mmap_host=localhost mmap_log=/var/log/tempesta_access.log;

5. Start Services

# Start Tempesta FW (creates mmap device)
sudo ./scripts/tempesta.sh --start

# tfw_ogger will be started automatically by tempesta.sh
# Or start manually:
sudo ./logger/tfw_logger --config /etc/tempesta/tfw_logger.json

Command Line Interface (for 0.9 only)

# Show help
./tfw_logger --help

# Start with specific configuration 
./tfw_logger --config /etc/tempesta/tfw_logger.json

# Override configuration options
./tfw_logger --config config.json --host clickhouse.example.com --port 9001 --table custom_access_log

# Test configuration
./tfw_logger --config config.json --help

CLI Options

Option Description
--help, -h Show help message
--generate, -g Generate default configuration file
--config, -c PATH Path to JSON configuration file
--host, -H HOST ClickHouse hostname (override)
--port, -P PORT ClickHouse port (override)
--database, -d DATABASE ClickHouse database name (override)
--table, -t TABLE ClickHouse table name (override)
--user, -u USER ClickHouse username (override)
--password, -p PASS ClickHouse password (override)
--log-path, -l PATH Log file path (override)

Performance Tuning

CPU Affinity

tfw_logger automatically sets CPU affinity for worker threads:

  • Each worker thread is bound to a specific CPU core
  • Number of threads automatically matches available CPU cores (respects affinity/cgroups)
  • Improves cache locality and reduces context switching

Buffer Sizing

Larger buffers reduce syscall overhead but increase memory usage:

Traffic Level Buffer Size Memory Usage
Low (< 1K RPS) 4MB ~4MB per worker
Medium (1K-10K RPS) 16MB ~16MB per worker
High (10K-100K RPS) 64MB ~64MB per worker
Enterprise (100K+ RPS) 256MB+ ~256MB+ per worker

ClickHouse Optimization

-- Optimize table for high-frequency inserts
ALTER TABLE access_log.access_log 
MODIFY SETTING merge_with_ttl_timeout = 3600;

-- Create materialized views for common queries
CREATE MATERIALIZED VIEW access_log.hourly_stats
ENGINE = SummingMergeTree()
ORDER BY (toStartOfHour(timestamp), status)
AS SELECT
    toStartOfHour(timestamp) as hour,
    status,
    count() as requests,
    avg(response_time) as avg_response_time
FROM access_log.access_log
GROUP BY hour, status;

Monitoring

Health Checks

# Check if tfw_logger is running
ps aux | grep tfw_logger

# Check log output
tail -f /var/log/tempesta/tfw_logger.log

# Verify ClickHouse connectivity
clickhouse-client --query "SELECT count() FROM access_log.access_log"

Metrics

Monitor these key metrics:

-- Request rate
SELECT 
    toStartOfMinute(timestamp) as minute,
    count() as requests_per_minute
FROM access_log.access_log 
WHERE timestamp >= now() - INTERVAL 1 HOUR
GROUP BY minute 
ORDER BY minute;

-- Error rates  
SELECT 
    status,
    count() as count,
    count() * 100.0 / sum(count()) OVER () as percentage
FROM access_log.access_log
WHERE timestamp >= now() - INTERVAL 1 HOUR
GROUP BY status
ORDER BY count DESC;

-- Dropped events (buffer overruns)
SELECT max(dropped_events) as max_dropped 
FROM access_log.access_log
WHERE timestamp >= now() - INTERVAL 1 HOUR;

Troubleshooting

Common Issues

tfw_logger won't start

# Check if Tempesta FW is running
sudo ./scripts/tempesta.sh --status

# Verify mmap device exists
ls -la /dev/tempesta_mmap_log

# Check permissions
sudo chmod 666 /dev/tempesta_mmap_log  # Temporary fix

Permission denied on mmap device

# Run tfw_logger with appropriate permissions
sudo ./tfw_logger --config config.json

# Or fix device permissions permanently
sudo chown tempesta:tempesta /dev/tempesta_mmap_log

ClickHouse connection failed

# Test ClickHouse connectivity
clickhouse-client --host localhost --port 9000 --query "SELECT 1"

# Check user permissions
clickhouse-client --query "SHOW GRANTS FOR tempesta_user"

High memory usage

# Reduce buffer size in configuration
{
    "clickhouse": {
        "max_events": 500     # Reduce batch size
    }
}

Log Analysis

Common log patterns:

# Successful startup
grep "Starting Tempesta FW Logger" /var/log/tempesta/tfw_logger.log

# Worker thread info
grep "worker threads started" /var/log/tempesta/tfw_logger.log  

# ClickHouse connectivity
grep "ClickHouse" /var/log/tempesta/tfw_logger.log

# Error patterns
grep -i error /var/log/tempesta/tfw_logger.log

Integration Examples

Basic Analytics Dashboard

-- Top pages by requests
SELECT 
    uri,
    count() as requests,
    avg(response_time) as avg_response_time_us
FROM access_log.access_log
WHERE timestamp >= now() - INTERVAL 1 DAY
GROUP BY uri
ORDER BY requests DESC
LIMIT 10;

-- Status code distribution
SELECT 
    status,
    count() as requests,
    count() * 100.0 / sum(count()) OVER () as percentage
FROM access_log.access_log  
WHERE timestamp >= now() - INTERVAL 1 DAY
GROUP BY status;

-- Traffic by hour
SELECT 
    toStartOfHour(timestamp) as hour,
    count() as requests,
    uniq(addr) as unique_visitors
FROM access_log.access_log
WHERE timestamp >= now() - INTERVAL 1 DAY  
GROUP BY hour
ORDER BY hour;

Alerting Queries

-- High error rate alert
SELECT count() as error_count
FROM access_log.access_log
WHERE timestamp >= now() - INTERVAL 5 MINUTE
  AND status >= 500;

-- Slow response time alert  
SELECT count() as slow_requests
FROM access_log.access_log
WHERE timestamp >= now() - INTERVAL 5 MINUTE
  AND response_time > 1000000; -- > 1 second

Quick Test

# Build and test
cd tempesta/logger
make build test
Add a custom footer

Contents

Clone this wiki locally