Unit Testing Phase 2.5: Protocol Encoding/Decoding Unit Tests

[renecannao]

Parent Issue

Part of #5472 — Unit Testing Framework: Milestone 2

Depends On

• Unit Testing Phase 2.1: Test Infrastructure Foundation (Glo* Stubs & Harness) #5473 — Phase 2.1: Test Infrastructure Foundation

Why Protocol Tests

Protocol encoding/decoding is the interface between ProxySQL and every client and backend. Bugs here cause silent data corruption, connection drops, or security vulnerabilities. These functions operate on byte buffers and have well-defined specifications (MySQL Client/Server Protocol, PostgreSQL Frontend/Backend Protocol), making them ideal for unit testing.

Key challenge: Many protocol functions live on MySQL_Protocol / PgSQL_Protocol classes that reference a MySQL_Session or MySQL_Data_Stream. Some encoding/decoding is inline or standalone, but others need session context. This phase may require identifying which functions can be tested as-is and which need minimal stubs.

Scope

Test File

test/tap/tests/unit/protocol_unit-t.cpp

Test Cases

MySQL length-encoded integers:

• mysql_decode_length() — 1-byte, 2-byte, 3-byte, 8-byte encoded values
• mysql_decode_length_ll() — 64-bit variant
• Boundary values: 0, 250, 251, 0xFFFF, 0xFFFFFF, max uint64
• Encoding + decoding roundtrip: encode a value, decode it, verify match

MySQL packet framing:

• Packet header parsing (4 bytes: 3-byte length + 1-byte sequence number)
• Multi-packet handling (payloads > 16MB split across packets)
• Sequence number validation

MySQL handshake:

• Server greeting packet construction → verify protocol version, server version, thread ID, auth plugin data, capability flags
• Client handshake response parsing → verify username, auth response, database, auth plugin name
• SSL request packet detection (capability flag check)
• Auth switch request construction/parsing

MySQL command packets:

• COM_QUERY packet parsing → extract query string
• COM_STMT_PREPARE parsing
• COM_STMT_EXECUTE parsing (with parameter binding)
• COM_QUIT, COM_PING, COM_INIT_DB handling

MySQL response packets:

• OK packet construction → verify affected_rows, last_insert_id, status_flags, warnings
• Error packet construction → verify error_code, sql_state, error_message
• EOF packet (old-style and new-style)
• Column definition packets
• Resultset row encoding (text and binary protocol)

MySQL compression:

• Compressed packet framing (7-byte header)
• Compression/decompression roundtrip (if testable without connection)

PostgreSQL protocol (parallel coverage):

• Startup message construction/parsing
• Authentication message types (AuthenticationOk, AuthenticationMD5Password, AuthenticationSASL)
• Query/Parse/Bind/Execute message encoding/decoding
• ReadyForQuery parsing (transaction status)
• ErrorResponse/NoticeResponse parsing
• RowDescription/DataRow encoding/decoding
• CommandComplete parsing

Edge cases:

• Malformed packets (truncated, oversized, invalid type bytes)
• Packets with NULL fields
• Unicode/multibyte characters in query strings
• Maximum field sizes

Technical Notes

Some protocol functions are methods on MySQL_Protocol which requires a MySQL_Data_Stream. Approach options:

1. Test only standalone/static functions first (decode functions, packet constructors)
2. Create a minimal MySQL_Data_Stream stub backed by an in-memory buffer instead of a socket
3. Refactor pure encoding/decoding into free functions (production code change — minimize)

Acceptance Criteria

• All test cases pass without a running ProxySQL instance or network
• Tests complete in under 3 seconds
• No memory leaks under ASAN
• MySQL length-encoded integer encoding/decoding tested at all boundary values
• At least one roundtrip test per major packet type (encode → decode → verify)
• Malformed packet handling tested (no crashes, graceful errors)
• Both MySQL and PostgreSQL protocols covered

[renecannao]

Closed by #5488