diff --git a/docs/global_variables/mysql_variables.md b/docs/global_variables/mysql_variables.md index 97478d6e39..2b11964779 100644 --- a/docs/global_variables/mysql_variables.md +++ b/docs/global_variables/mysql_variables.md @@ -14,7 +14,9 @@ NOTE: You can click on the variable name to jump to its definition | ------------------------------------------------------------------------------------------------------- | --------------------------- | | [mysql-add_ldap_user_comment](#mysql-add_ldap_user_comment) | | | [mysql-auditlog_filename](#mysql-auditlog_filename) | | -| [mysql-auditlog_filesize](#mysql-auditlog_filename) | 100MB | +| [mysql-auditlog_filesize](#mysql-auditlog_filesize) | 100MB | +| [mysql-auditlog_flush_timeout](#mysql-auditlog_flush_timeout) | 1000 | +| [mysql-auditlog_flush_size](#mysql-auditlog_flush_size) | 4096 | | [mysql-aurora_max_lag_ms_only_read_from_replicas](#mysql-aurora_max_lag_ms_only_read_from_replicas) | 2 | | [mysql-auto_increment_delay_multiplex](#mysql-auto_increment_delay_multiplex) | 5 | | [mysql-auto_increment_delay_multiplex_timeout_ms](#mysql-auto_increment_delay_multiplex_timeout_ms) | 10000 | @@ -69,6 +71,9 @@ NOTE: You can click on the variable name to jump to its definition | [mysql-eventslog_filesize](#mysql-eventslog_filesize) | 104857600 | | [mysql-eventslog_format](#mysql-eventslog_format) | 1 | | [mysql-eventslog_stmt_parameters](#mysql-eventslog_stmt_parameters) | 0 | +| [mysql-eventslog_flush_timeout](#mysql-eventslog_flush_timeout) | 1000 | +| [mysql-eventslog_flush_size](#mysql-eventslog_flush_size) | 4096 | +| [mysql-eventslog_rate_limit](#mysql-eventslog_rate_limit) | 1 | | [mysql-firewall_whitelist_enabled](#mysql-firewall_whitelist_enabled) | false | | [mysql-firewall_whitelist_errormsg](#mysql-firewall_whitelist_errormsg) | Firewall blocked this query | | [mysql-forward_autocommit](#mysql-forward_autocommit) | false | @@ -218,6 +223,35 @@ progressive number. Files are automatically rotated based on the value of [mysql This variable defines the maximum file size of the audit log file (see [mysql-auditlog_filename][5]) when the current file will be closed and a new file will be created. The default value is 104857600 (100MB) +### `mysql-auditlog_flush_timeout` + +| | | | +| -------------------- | ----------- | ---------------------------- | +| **System Variable** | **Name** | mysql-auditlog_flush_timeout | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (milliseconds) | +| | **Default** | 1000 | +| | **Minimum** | 0 | +| | **Maximum** | | + +This variable defines the flush timeout to write the buffered audit log entries to the +audit log file (see [Audit Log][3]). The default value is 1000 (1 second). + +### `mysql-auditlog_flush_size` + +| | | | +| -------------------- | ----------- | -------------------------- | +| **System Variable** | **Name** | mysql-auditlog_flush_size | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (bytes) | +| | **Default** | 4096 | +| | **Minimum** | 0 | +| | **Maximum** | | + +This variable defines the flush size to write the buffered audit log entries to the +audit log file (see [Audit Log][3]). If the size of the buffer goes beyond this value, +the contents of the buffer are written to the file. The default value is 4096. + ### `mysql-aurora_max_lag_ms_only_read_from_replicas` | | | | @@ -1166,6 +1200,50 @@ See also [Query Logging][34] Due to the potential overhead of logging prepared statements parameters, especially in `JSON` format and in the presence of large blobs, this variable controls if parameters need to be logged or not. +### `mysql-eventslog_flush_timeout` + +| | | | +| -------------------- | ----------- | ----------------------------- | +| **System Variable** | **Name** | mysql-eventslog_flush_timeout | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (milliseconds) | +| | **Default** | 1000 | +| | **Minimum** | 0 | +| | **Maximum** | | + +This variable defines the flush timeout to write the buffered event log entries to the +events log file (see [Query Logging][34]). The default value is 1000 (1 second). + +### `mysql-eventslog_flush_size` + +| | | | +| -------------------- | ----------- | -------------------------- | +| **System Variable** | **Name** | mysql-eventslog_flush_size | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (bytes) | +| | **Default** | 4096 | +| | **Minimum** | 0 | +| | **Maximum** | | + +This variable defines the flush size to write the buffered event log entries to the +events log file (see [Query Logging][34]). If the size of the buffer goes beyond this value, +the contents of the buffer are written to the file. The default value is 4096. + +### `mysql-eventslog_rate_limit` + +| | | | +| -------------------- | ----------- | -------------------------- | +| **System Variable** | **Name** | mysql-eventslog_rate_limit | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer | +| | **Default** | 1 | +| | **Minimum** | 1 | +| | **Maximum** | | + +This variable defines the sampling rate for event logs (see [Query Logging][34]). To log all +events, set this variable to 1. If the variable is set to N, where N > 1, the number of logs written to the file +will be approximately equal to 1/N. The default value is 1. + ### `mysql-firewall_whitelist_enabled` | | | | diff --git a/docs/global_variables/pgsql_variables.md b/docs/global_variables/pgsql_variables.md index 65be167846..c346e4815f 100644 --- a/docs/global_variables/pgsql_variables.md +++ b/docs/global_variables/pgsql_variables.md @@ -14,6 +14,8 @@ NOTE: You can click on the variable name to jump to its definition | --------------------------------------------------------------------- | ------------- | | [pgsql-auditlog_filename](#pgsql-auditlog_filename) | NULL | | [pgsql-auditlog_filesize](#pgsql-auditlog_filesize) | 104857600 | +| [pgsql-auditlog_flush_timeout](#pgsql-auditlog_flush_timeout) | 1000 | +| [pgsql-auditlog_flush_size](#pgsql-auditlog_flush_size) | 4096 | | [pgsql-aurora_max_lag_ms_only_read_from_replicas](#pgsql-aurora_max_lag_ms_only_read_from_replicas) | 2 | | [pgsql-authentication_method](#pgsql-authentication_method) | 3 | | [pgsql-auto_increment_delay_multiplex](#pgsql-auto_increment_delay_multiplex) | 5 | @@ -43,6 +45,9 @@ NOTE: You can click on the variable name to jump to its definition | [pgsql-eventslog_filename](#pgsql-eventslog_filename) | NULL | | [pgsql-eventslog_filesize](#pgsql-eventslog_filesize) | 104857600 | | [pgsql-eventslog_format](#pgsql-eventslog_format) | 1 | +| [pgsql-eventslog_flush_timeout](#pgsql-eventslog_flush_timeout) | 1000 | +| [pgsql-eventslog_flush_size](#pgsql-eventslog_flush_size) | 4096 | +| [pgsql-eventslog_rate_limit](#pgsql-eventslog_rate_limit) | 1 | | [pgsql-firewall_whitelist_enabled](#pgsql-firewall_whitelist_enabled) | false | | [pgsql-free_connections_pct](#pgsql-free_connections_pct) | 10 | | [pgsql-handle_unknown_charset](#pgsql-handle_unknown_charset) | 1 | @@ -135,6 +140,40 @@ Maximum size of the PostgreSQL audit log file. **Description**: Specifies the size limit for individual PostgreSQL audit log files before they are rotated. + +### `pgsql-auditlog_flush_timeout` + +Timeout for writing logs to PostgreSQL audit log file. + +| | | | +| -------------------- | ----------- | ---------------------------- | +| **System Variable** | **Name** | pgsql-auditlog_flush_timeout | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (milliseconds) | +| | **Default** | 1000 | +| | **Minimum** | 0 | +| | **Maximum** | | + +**Description**: Specifies the timeout to write the buffered audit log entries to the +audit log file. The default value is 1000 (1 second). + +### `pgsql-auditlog_flush_size` + +Size threshold for writing logs to PostgreSQL audit log file. + +| | | | +| -------------------- | ----------- | -------------------------- | +| **System Variable** | **Name** | pgsql-auditlog_flush_size | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (bytes) | +| | **Default** | 4096 | +| | **Minimum** | 0 | +| | **Maximum** | | + +**Description**: This variable defines the flush size to write the buffered audit log entries to the +audit log file. If the size of the buffer goes beyond this value, +the contents of the buffer are written to the file. The default value is 4096. + ### `pgsql-aurora_max_lag_ms_only_read_from_replicas` **Description**: Not supported. @@ -458,6 +497,56 @@ Format of the PostgreSQL events log. **Description**: Sets the logging format for PostgreSQL events. Currently, only format `1` (Binary) is supported. +### `pgsql-eventslog_flush_timeout` + +Timeout for writing logs to the PostgreSQL events log. + +| | | | +| -------------------- | ----------- | ----------------------------- | +| **System Variable** | **Name** | pgsql-eventslog_flush_timeout | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (milliseconds) | +| | **Default** | 1000 | +| | **Minimum** | 0 | +| | **Maximum** | | + +**Description**: This variable defines the flush timeout to write the buffered event log entries to the +events log file. The default value is 1000 (1 second). + +### `pgsql-eventslog_flush_size` + +Size threshold for writing logs to the PostgreSQL events log. + +| | | | +| -------------------- | ----------- | -------------------------- | +| **System Variable** | **Name** | pgsql-eventslog_flush_size | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer (bytes) | +| | **Default** | 4096 | +| | **Minimum** | 0 | +| | **Maximum** | | + +**Description**: This variable defines the flush size to write the buffered event log entries to the +events log file. If the size of the buffer goes beyond this value, +the contents of the buffer are written to the file. The default value is 4096. + +### `pgsql-eventslog_rate_limit` + +Sampling rate for the PostgreSQL events log. + +| | | | +| -------------------- | ----------- | -------------------------- | +| **System Variable** | **Name** | pgsql-eventslog_rate_limit | +| | **Dynamic** | Yes | +| **Permitted Values** | **Type** | Integer | +| | **Default** | 1 | +| | **Minimum** | 1 | +| | **Maximum** | | + +**Description**: This variable defines the sampling rate for event logs. To log all +events, set this variable to 1. If the variable is set to N, where N > 1, the number of logs written to the file +will be approximately equal to 1/N. The default value is 1. + ### `pgsql-firewall_whitelist_enabled` Enables the firewall whitelist for PostgreSQL. diff --git a/docs/query_logging/query_logging.md b/docs/query_logging/query_logging.md index 593268a585..7572e2fe93 100644 --- a/docs/query_logging/query_logging.md +++ b/docs/query_logging/query_logging.md @@ -47,7 +47,7 @@ If you changed the variable `mysql-eventslog_format` to `2` (logging queries in and/or you would only like to log queries matching certain criteria, you need additional query rules in the following manner: If you don't trust Bob, you can log all of Bob's queries: -```sql +```sqlhttps://proxysql.com/documentation/global-variables/mysql-variables/#mysql-auditlog_filename INSERT INTO mysql_query_rules (rule_id, active, username, log, apply) VALUES (1, 1, 'Bob', 1, 0); ``` @@ -130,6 +130,48 @@ Example of JSON logging: {"client":"127.0.0.1:39960","digest":"0x98A2503010E9E4C8","duration_us":1492,"endtime":"2019-07-14 18:06:04.013779","endtime_timestamp_us":1563091564013779,"event":"COM_STMT_EXECUTE","hostgroup_id":0,"query":"SELECT id,id2 FROM test1 WHERE id < ?","rows_affected":0,"rows_sent":4099,"schemaname":"test","server":"127.0.0.1:3306","starttime":"2019-07-14 18:06:04.012287","starttime_timestamp_us":1563091564012287,"thread_id":4,"username":"sbtest"} ``` +## Sampling + +A new variable `mysql-eventslog_rate_limit` is introduced to control the sampling rate for event logs. When set to `1` (default), +all events are logged to the file. When set to a value greater than `1`, the number of events logged to the file will be +approximately equal to `1/N`, where `N` is the value set for the variable. + +**To log all events (default)** + +```sql +SET mysql-eventslog_rate_limit=1; +LOAD MYSQL VARIABLES TO RUNTIME; +SAVE MYSQL VARIABLES TO DISK; +``` + +**To log events at a certain sampling rate** + +```sql +SET mysql-eventslog_rate_limit=100; +LOAD MYSQL VARIABLES TO RUNTIME; +SAVE MYSQL VARIABLES TO DISK; +``` + +## Logging at high RPS + +Logging queries at high throughput often requires tuning `mysql-eventslog_flush_timeout` and `mysql-eventslog_flush_size`. +These variables determine when the log entries stored in temporary buffers are flushed to the file on disk. + +**Time based flush** + +`mysql-eventslog_flush_timeout` triggers a flush when the time elapsed since the last flush exceeds the specified timeout value. + +**Size based flush** + +`mysql-eventslog_flush_size` triggers a flush when the buffer size (in bytes) exceeds the specified limit. Note that each worker thread maintains its own temporary buffer for log entries. Therefore, this variable should be tuned based on the available hardware resources. Increasing this value will increase memory consumption, roughly calculated as the number of worker threads multiplied by the flush size. + +```sql +SET mysql-eventslog_flush_timeout=5000; +SET mysql-eventslog_flush_size=16*1024; +LOAD MYSQL VARIABLES TO RUNTIME; +SAVE MYSQL VARIABLES TO DISK; +``` + ## Related Issues and Feature Requests Here's some related discussion on this feature.