Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
2a46c23
Add PostgreSQL advanced eventslog schema and variable scaffolding
renecannao Feb 18, 2026
acbd7d0
Implement PostgreSQL eventslog circular buffer and sink pipeline
renecannao Feb 18, 2026
b100235
Add PGSQL eventslog dump commands and periodic disk sync scheduling
renecannao Feb 18, 2026
b4e85f1
Expose PostgreSQL eventslog metrics in stats and Prometheus
renecannao Feb 18, 2026
1edf1f8
Add PostgreSQL advanced logging TAP coverage and rewrite architecture…
renecannao Feb 18, 2026
20d6282
build(clean): ignore and remove generated MySQL parser artifacts
renecannao Feb 18, 2026
2586995
tap(pgsql): add end-to-end test for eventslog automatic buffer-to-dis…
renecannao Feb 18, 2026
97e5cfe
docs(doxygen): expand PostgreSQL logger API documentation for new adv…
renecannao Feb 18, 2026
6f5a22f
docs(internal): update PGSQL advanced logging architecture validation…
renecannao Feb 18, 2026
eba795d
fix(pgsql-eventslog): make dump-to-memory sizing protocol-agnostic ac…
renecannao Feb 18, 2026
0aa3c5e
fix(pgsql-eventslog): harden event ownership and keep newest rows in …
renecannao Feb 18, 2026
6db0898
fix(admin): improve PGSQL eventslog dump failure reporting and stats …
renecannao Feb 18, 2026
ab7d918
test(pgsql-eventslog): improve autodump assertion diagnostics and run…
renecannao Feb 18, 2026
82a28a2
fix(pgsql-eventslog): close remaining minor review findings in logger…
renecannao Feb 18, 2026
2a1b199
test(tap): register pgsql_query_logging_autodump-t in test groups
renecannao Feb 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,12 @@ test/tap/tests/test_cluster_sync_config/test_cluster_sync.cnf
.aider*
GEMINI.md

# generated MySQL parser artifacts
lib/MySQL_Lexer.yy.c
lib/MySQL_Parser.output
lib/MySQL_Parser.tab.c
lib/MySQL_Parser.tab.h

# Database discovery output files
discovery_*.md
database_discovery_report.md
Expand Down
210 changes: 210 additions & 0 deletions doc/internal/pgsql_advanced_query_logging_architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
# PostgreSQL Advanced Query Logging Architecture

## Document Status
- Status: Implemented (as-built)
- Scope: PostgreSQL advanced events logging parity with MySQL, including buffer, SQLite sinks, dump commands, scheduler sync, and metrics
- Branch: `v3.0_pgsql_advanced_logging`

## 1. Objective and Delivered Outcome
The PostgreSQL logging pipeline now supports advanced query events logging with the same operational model already used by MySQL:
- query event capture at request completion
- in-memory circular buffering
- manual dump from buffer to `stats` and/or `stats_history`
- optional periodic auto-dump to disk
- logger metrics in `stats_pgsql_global` and Prometheus

This implementation is additive. Existing PostgreSQL file-based events log and audit log behavior remains available.

## 2. As-Built Runtime Architecture

### 2.1 Capture path
1. PostgreSQL sessions call query logging at request completion.
2. `PgSQL_Logger::log_request()` builds a `PgSQL_Event` from session/backend state.
3. If file logging is enabled, the event is written to events log file.
4. If `pgsql-eventslog_buffer_history_size > 0`, the event is deep-copied and inserted into the PostgreSQL events circular buffer.

Implemented in:
- `lib/PgSQL_Logger.cpp`
- `include/PgSQL_Logger.hpp`

### 2.2 Buffering model
A dedicated `PgSQL_Logger_CircularBuffer` provides:
- thread-safe insertion/drain using mutex
- bounded size via runtime variable
- event drop accounting when the buffer is full or resized smaller

Runtime resizing is wired in PostgreSQL thread variable refresh:
- `PgSQL_Thread::refresh_variables()` applies `eventslog_buffer_history_size` changes to the live circular buffer.

Implemented in:
- `include/PgSQL_Logger.hpp`
- `lib/PgSQL_Logger.cpp`
- `lib/PgSQL_Thread.cpp`

### 2.3 Drain and persistence pipeline
`PgSQL_Logger::processEvents(SQLite3DB* statsdb, SQLite3DB* statsdb_disk)` drains the buffer and persists events to:
- memory table: `stats_pgsql_query_events` when `statsdb != nullptr`
- disk table: `history_pgsql_query_events` when `statsdb_disk != nullptr`

Behavior:
- batched SQLite inserts with prepared statements
- in-memory retention bound by `pgsql-eventslog_table_memory_size`
- query digest serialized as hex text (`0x...`), matching MySQL table style
- `sqlstate` and textual `error` persisted for failed queries

Implemented in:
- `lib/PgSQL_Logger.cpp`

## 3. Data Model

### 3.1 Memory table
`stats.stats_pgsql_query_events`

Columns:
- `id`
- `thread_id`
- `username`
- `database`
- `start_time`
- `end_time`
- `query_digest`
- `query`
- `server`
- `client`
- `event_type`
- `hid`
- `extra_info`
- `affected_rows`
- `rows_sent`
- `client_stmt_name`
- `sqlstate`
- `error`

Implemented in:
- `include/ProxySQL_Admin_Tables_Definitions.h`
- `lib/Admin_Bootstrap.cpp`

### 3.2 Disk history table
`stats_history.history_pgsql_query_events`

Columns match `stats_pgsql_query_events`.

Indexes:
- `idx_history_pgsql_query_events_start_time` on `start_time`
- `idx_history_pgsql_query_events_query_digest` on `query_digest`

Implemented in:
- `include/ProxySQL_Statistics.hpp`
- `lib/ProxySQL_Statistics.cpp`

## 4. Admin Interface and Control Surface

### 4.1 Dump commands
PostgreSQL-specific dump commands are now available:
- `DUMP PGSQL EVENTSLOG FROM BUFFER TO MEMORY`
- `DUMP PGSQL EVENTSLOG FROM BUFFER TO DISK`
- `DUMP PGSQL EVENTSLOG FROM BUFFER TO BOTH`

Command handling executes `GloPgSQL_Logger->processEvents(...)` with the selected sink targets.

These commands are exposed by the shared Admin module and are available from both Admin protocol endpoints:
- MySQL protocol on port `6032`
- PostgreSQL protocol on port `6132`

Implemented in:
- `lib/Admin_Handler.cpp`

### 4.2 Runtime/config variables
PostgreSQL thread variables used by advanced logging:
- `pgsql-eventslog_buffer_history_size`
- `pgsql-eventslog_table_memory_size`
- `pgsql-eventslog_buffer_max_query_length`

Admin scheduling variable:
- `admin-stats_pgsql_eventslog_sync_buffer_to_disk`

Implemented in:
- `include/PgSQL_Thread.h`
- `include/proxysql_structs.h`
- `include/proxysql_admin.h`
- `lib/ProxySQL_Admin.cpp`

## 5. Scheduler Integration

### 5.1 PostgreSQL auto-dump to disk
Admin main loop now periodically flushes PostgreSQL buffered events to `history_pgsql_query_events` when:
- `stats_pgsql_eventslog_sync_buffer_to_disk > 0`
- timer interval is elapsed

### 5.2 MySQL symmetry fix
The same scheduler loop now also invokes MySQL buffered events dump to disk based on `stats_mysql_eventslog_sync_buffer_to_disk`, ensuring symmetric behavior across both protocols.

Implemented in:
- `lib/ProxySQL_Admin.cpp`

## 6. Metrics Architecture

### 6.1 Logger internal metrics
PostgreSQL logger tracks:
- memory/disk copy counts
- total copy time (memory/disk)
- get-all-events calls/time/count
- total events copied (memory/disk)
- circular buffer added/dropped totals
- circular buffer current size

Implemented in:
- `include/PgSQL_Logger.hpp`
- `lib/PgSQL_Logger.cpp`

### 6.2 Stats table exposure
Metrics are exported to `stats_pgsql_global` with `PgSQL_Logger_` prefix.

Implemented in:
- `lib/ProxySQL_Admin_Stats.cpp`

### 6.3 Prometheus exposure
Prometheus metric family `proxysql_pgsql_logger_*` is exposed through the serial metrics updater path.

Implemented in:
- `lib/PgSQL_Logger.cpp`
- `lib/ProxySQL_Admin.cpp`

## 7. Compatibility and Semantics
- Existing `DUMP EVENTSLOG ...` remains MySQL behavior for compatibility.
- New PostgreSQL syntax is explicit: `DUMP PGSQL EVENTSLOG ...`.
- PostgreSQL event error fields use `sqlstate + error` (textual message).
- PostgreSQL event table uses `database` column naming.

## 8. Code Touchpoint Summary
- Logger core:
- `include/PgSQL_Logger.hpp`
- `lib/PgSQL_Logger.cpp`
- Thread/runtime integration:
- `lib/PgSQL_Thread.cpp`
- Admin commands and scheduler:
- `lib/Admin_Handler.cpp`
- `lib/ProxySQL_Admin.cpp`
- Table definitions/bootstrap:
- `include/ProxySQL_Admin_Tables_Definitions.h`
- `include/ProxySQL_Statistics.hpp`
- `lib/Admin_Bootstrap.cpp`
- `lib/ProxySQL_Statistics.cpp`
- Stats/Prometheus metrics:
- `lib/ProxySQL_Admin_Stats.cpp`

## 9. Validation and Acceptance Mapping
Implemented acceptance validation via TAP test:
- `test/tap/tests/pgsql_query_logging_memory-t.cpp`
- `test/tap/tests/pgsql_query_logging_autodump-t.cpp`

Coverage:
- table schema shape validation (`stats_pgsql_query_events`, `history_pgsql_query_events`)
- buffer dump command execution
- success/error event accounting in memory and history tables
- `sqlstate` capture for representative PostgreSQL errors
- non-empty textual `error` capture for error rows
- scheduler-driven periodic dump to `history_pgsql_query_events` via `admin-stats_pgsql_eventslog_sync_buffer_to_disk`

TAP group registration:
- `test/tap/groups/groups.json` (`pgsql_query_logging_memory-t`, `pgsql_query_logging_autodump-t`)
Loading
Loading