Refactor eloq substrate init function

[liunyl]

Feature Summary

In order to support converged db, we need to extract the data substrate related initialization into a separate function.

Motivation

Proposed Solution

In the init function, it should read a config file that has all configs related to data substrate. This config file should not have any database engine related configs.
It should initialize store_hd, log_service and tx_service in the right order. As the three core component of data substrate, they should be able to be accessed globally.
After data substrate initialization is done, database engine can do their own initialization and pass in its engine specific configs to data substrate (i.e. catalog factory)

[liunyl]

Research: Data Substrate Initialization Analysis for Converged Database

Date: 2025-09-25T06:40:15Z
Researcher: Claude
Git Commit: N/A (not a git repository)
Branch: N/A (not a git repository)
Repository: converged

Research Question

What needs to be extracted into a shared data substrate initialization function to enable eloqkv and eloqsql to share the same data substrate components (tx_service, store_hd, log_service)? The data substrate will support multiple database engines (eloqsql, eloqkv, eloqdoc) simultaneously and use a unified configuration system.

Summary

Both eloqkv and eloqsql currently initialize their own data substrate components during their respective init functions. The data substrate consists of three core components: tx_service, store_hd (storage handler), and log_service (log agent). The new approach will use a unified configuration file + gflags pattern, with the data substrate reading from a separate config file without any engine-specific patterns. The TxService will accept multiple catalog factories (array of size 3) to support multiple engines simultaneously, with engine-specific dependencies wrapped in compile-time macros.

Detailed Findings

Current Initialization Functions

eloqsql Initialization (ha_eloq.cc:2397)

• Function: eloq_init_func(void *p)
• Location: eloqsql/storage/eloq/ha_eloq.cc:2397-3283
• Configuration System: MySQL system variables with eloq_ prefix
• Key Components Initialized:
  • Storage handler (storage_hd) - lines 2558, 2573, 2750
  • Log agent (log_agent) - line 3095
  • Tx service (tx_service) - lines 3209, 3227

eloqkv Initialization (redis_service.cpp)

• Function: RedisServiceImpl::Init() (main initialization)
• Location: eloqkv/src/redis_service.cpp
• Configuration System: gflags with FLAGS_ prefix + INI config files
• Key Components Initialized:
  • Storage handler (store_hd_) - lines 961, 978, 1138
  • Log agent (log_agent) - line 1374
  • Tx service (tx_service_) - line 1437

Data Substrate Components Analysis

1. TxService

Purpose: Core transaction management service
Initialization Pattern:

• Both projects create std::unique_ptr<TxService>
• Requires catalog factory, system handler, configuration map, and other dependencies
• Key Dependencies:
  • CatalogFactory (engine-specific)
  • SystemHandler (engine-specific: MariaSystemHandler vs nullptr)
  • DataStoreHandler (store_hd)
  • LogAgent (log_service)
  • Configuration map with ~15 parameters

Configuration Parameters (from both projects):

std::map<std::string, uint32_t> tx_service_conf{
    {"core_num", core_num},
    {"checkpointer_interval", ckpt_interval},
    {"node_memory_limit_mb", node_memory_limit_mb},
    {"node_log_limit_mb", node_log_limit_mb},
    {"checkpointer_delay_seconds", ckpt_delay},
    {"collect_active_tx_ts_interval_seconds", active_tx_interval},
    {"realtime_sampling", realtime_sampling},
    {"rep_group_cnt", replica_num},
    {"enable_key_cache", enable_key_cache},
    {"enable_shard_heap_defragment", enable_heap_defragment},
    {"bthread_worker_num", bthread_worker_num},
    {"max_standby_lag", max_standby_lag},
    {"kickout_data_for_test", kickout_data_for_test}
};

2. Store Handler (store_hd)

Purpose: Data storage abstraction layer
Implementation Types:

• EloqDS::DynamoHandler - DynamoDB backend
• EloqKV::RocksDBHandlerImpl - RocksDB backend
• EloqDS::DataStoreServiceClient - Data store service client
• EloqDS::BigTableHandler - BigTable backend

Initialization Dependencies:

• Storage-specific configuration (endpoints, credentials, paths)
• Bootstrap mode detection
• Connection establishment via Connect() method
• Critical: Requires SetTxService() call after tx_service creation

3. Log Service (log_agent)

Purpose: Write-ahead logging and replication
Implementation:

• MyEloqLogAgent (eloqsql) vs EloqKV::EloqLogAgent (eloqkv)
• Both inherit from same base class
• Key Dependencies:
  • Replica number configuration
  • Log service initialization via InitTxLogService()
  • RocksDB configuration for log storage

Configuration Pattern Differences

Current eloqsql Configuration System

• Format: MySQL system variables
• Prefix: eloq_
• Examples:
  • eloq_local_ip, eloq_core_num, eloq_txlog_service_list
  • eloq_enable_tx_metrics, eloq_enable_log_service_metrics
• Access: Direct variable access
• Data Path: mysql_real_data_home_ptr

Current eloqkv Configuration System

• Format: gflags + INI files
• Prefix: FLAGS_ for command line, config file sections
• Examples:
  • FLAGS_txlog_service_list, FLAGS_core_num
  • config_reader.GetString("cluster", "txlog_service_list")
• Access: Command line flags + INIReader
• Data Path: eloq_data_path (configurable)

New Unified Configuration System

• Format: Unified config file + gflags
• Data Substrate Config: Separate config file with no engine-specific patterns
• Engine Integration: Engines pass their specific configurations to data substrate
• Multi-Engine Support: Single data substrate instance supports multiple engines

Initialization Sequence Analysis

Current eloqsql Sequence:

1. Parse MySQL system variables
2. Initialize storage handler (storage_hd)
3. Initialize log agent (log_agent)
4. Create tx_service with all dependencies
5. Start tx_service
6. Set tx_service reference in storage handler

Current eloqkv Sequence:

1. Parse gflags and INI config
2. Initialize log service (if not standalone)
3. Initialize storage handler (store_hd_)
4. Create tx_service with all dependencies
5. Set tx_service reference in storage handler
6. Start tx_service

Key Dependencies and Order Requirements

Critical Dependencies:

1. Storage Handler → TxService: Storage handler needs tx_service reference via SetTxService()
2. Log Service → TxService: Log agent passed to tx_service constructor
3. Catalog Factory → Multi-Engine: Array of catalog factories for multiple engines (size 3)
4. System Handler → Engine: Engine-specific system handlers (wrapped in compile macros)

Multi-Engine Considerations:

• Catalog Factory Array: TxService accepts CatalogFactory* catalog_factory[3] for eloqsql, eloqkv, eloqdoc
• Compile-Time Engine Selection: Engine-specific code wrapped in #ifdef macros
• Bootstrap Mode: Unified bootstrap detection across all engines
• Engine Registration: Engines register themselves with data substrate during initialization

Architecture Insights

Shared vs Engine-Specific Components

Shared Components (extracted to data substrate):

• TxService core logic
• DataStoreHandler implementations
• LogAgent implementations
• Unified configuration parsing logic
• Metrics initialization
• Multi-engine catalog factory management

Engine-Specific Components (wrapped in compile macros):

• CatalogFactory implementations (#ifdef ELOQSQL MariaCatalogFactory, #ifdef ELOQKV engine-specific, etc.)
• SystemHandler implementations (#ifdef ELOQSQL MariaSystemHandler, #ifdef ELOQKV nullptr, etc.)
• Engine registration and initialization logic
• Engine-specific data path resolution

Configuration Unification Approach

1. Unified Config File: Single data substrate config file with standardized naming
2. Gflags Integration: Use gflags for command-line overrides and defaults
3. Engine-Specific Configs: Engines pass their specific configurations to data substrate
4. No Engine Dependencies: Data substrate config file contains no engine-specific patterns

Potential Issues with Multi-Engine Data Substrate Initialization

1. Multi-Engine Catalog Factory Management

Issue: TxService needs to support multiple catalog factories simultaneously
Impact: Current implementation expects single catalog factory
Solution: Modify TxService to accept CatalogFactory* catalog_factory[3] array

2. Compile-Time Engine Selection

Issue: Engine-specific code needs to be conditionally compiled
Impact: Data substrate must compile with or without specific engines
Solution: Wrap all engine-specific dependencies in #ifdef ELOQSQL, #ifdef ELOQKV, #ifdef ELOQDOC macros

3. Engine Registration and Initialization Order

Issue: Multiple engines need to register with data substrate in correct order
Impact: Initialization sequence becomes more complex
Solution: Implement engine registration system with dependency management

4. Unified Configuration Management

Issue: Single config file must serve multiple engines without conflicts
Impact: Configuration namespace and parameter management
Solution: Use unified config file with engine-specific parameter passing

5. Resource Sharing and Isolation

Issue: Multiple engines sharing same data substrate resources
Impact: Potential resource conflicts and performance isolation
Solution: Implement resource partitioning and isolation mechanisms

Code References

eloqsql Key Files:

• eloqsql/storage/eloq/ha_eloq.cc:2397 - Main initialization function
• eloqsql/storage/eloq/ha_eloq.cc:3209,3227 - TxService creation
• eloqsql/storage/eloq/ha_eloq.cc:2558,2573,2750 - Storage handler creation
• eloqsql/storage/eloq/ha_eloq.cc:3095 - Log agent creation
• eloqsql/storage/eloq/ha_eloq.cc:3253 - SetTxService call

eloqkv Key Files:

• eloqkv/src/redis_service.cpp - Main initialization logic
• eloqkv/src/redis_service.cpp:1437 - TxService creation
• eloqkv/src/redis_service.cpp:961,978,1138 - Storage handler creation
• eloqkv/src/redis_service.cpp:1374 - Log agent creation
• eloqkv/src/redis_service.cpp:1469 - SetTxService call

Shared Components:

• eloqkv/tx_service/include/tx_service.h - TxService interface
• eloqsql/storage/eloq/tx_service/include/tx_service.h - TxService interface (identical)
• eloqkv/tx_service/include/store/data_store_handler.h - Storage handler interface
• eloqsql/storage/eloq/tx_service/include/store/data_store_handler.h - Storage handler interface (identical)

Recommendations

1. Create Multi-Engine Data Substrate Initialization Function

class DataSubstrate {
public:
    struct Config {
        // Unified configuration structure (no engine-specific patterns)
        std::string data_path;
        std::map<std::string, uint32_t> tx_service_config;
        std::string txlog_service_list;
        // ... other unified config parameters
    };
    
    struct EngineConfig {
        CatalogFactory* catalog_factory;
        SystemHandler* system_handler;
        std::string engine_name;
        // Engine-specific parameters
    };
    
    static std::unique_ptr<DataSubstrate> Initialize(
        const Config& config,
        const std::vector<EngineConfig>& engines
    );
};

2. Compile-Time Engine Selection

#ifdef ELOQSQL
    #include "eloq_catalog_factory.h"
    #include "eloq_system_handler.h"
#endif

#ifdef ELOQKV
    #include "eloqkv_catalog_factory.h"
#endif

#ifdef ELOQDOC
    #include "eloqdoc_catalog_factory.h"
#endif

3. Multi-Engine TxService Modification

// Modify TxService constructor to accept multiple catalog factories
TxService(
    CatalogFactory* catalog_factory[3],  // Array for eloqsql, eloqkv, eloqdoc
    SystemHandler* system_handlers[3],   // Array for each engine
    const std::map<std::string, uint32_t>& config,
    // ... other parameters
);

4. Hard-Coded Engine Registration System

class DataSubstrate {
private:
    void RegisterEngines() {
        std::vector<EngineConfig> engines;
        
#ifdef ELOQSQL
        // Register eloqsql engine
        EngineConfig eloqsql_config;
        eloqsql_config.catalog_factory = &maria_catalog_factory;
        eloqsql_config.system_handler = &MariaSystemHandler::Instance();
        eloqsql_config.engine_name = "eloqsql";
        engines.push_back(eloqsql_config);
#endif

#ifdef ELOQKV
        // Register eloqkv engine
        EngineConfig eloqkv_config;
        eloqkv_config.catalog_factory = &eloqkv_catalog_factory;
        eloqkv_config.system_handler = nullptr;  // eloqkv doesn't use system handler
        eloqkv_config.engine_name = "eloqkv";
        engines.push_back(eloqkv_config);
#endif

#ifdef ELOQDOC
        // Register eloqdoc engine
        EngineConfig eloqdoc_config;
        eloqdoc_config.catalog_factory = &eloqdoc_catalog_factory;
        eloqdoc_config.system_handler = &EloqDocSystemHandler::Instance();
        eloqdoc_config.engine_name = "eloqdoc";
        engines.push_back(eloqdoc_config);
#endif
        
        // Initialize with registered engines
        InitializeWithEngines(engines);
    }
};

5. Phased Multi-Engine Initialization

1. Phase 1: Read unified data substrate config file (INI format)
2. Phase 2: Initialize storage handler and log service
3. Phase 3: Hard-code engine registration (eloqsql first, then eloqkv)
4. Phase 4: Create tx_service with multi-engine catalog factory array
5. Phase 5: Establish cross-references and start services

Open Questions - Resolved

1. Configuration File Format: ✅ Use INI format - Standardized and easy to parse
2. Engine Initialization Order: ✅ Order does not matter - Use eloqsql then eloqkv
3. Resource Isolation: ✅ Managed by data substrate - No need to worry about isolation
4. Compile Macro Strategy: ✅ Use #ifdef ELOQKV, #ifdef ELOQDOC, #ifdef ELOQSQL - Current approach is correct
5. Engine Discovery: ✅ Hard code registration in init function - Wrap registration code for each engine with compile-time macros

Implementation Notes

Key Design Decisions

1. Unified Config + Gflags: Data substrate uses unified config file with gflags for overrides
2. Multi-Engine Support: Single data substrate instance supports eloqsql, eloqkv, and eloqdoc simultaneously
3. Compile-Time Selection: Engine-specific code wrapped in #ifdef macros for conditional compilation
4. No Backward Compatibility: Clean break from existing configuration systems
5. Manual Testing: Initial testing will be done manually without automated test suite

Configuration File Structure (INI Format)

[data_substrate]
data_path = /path/to/data
core_num = 8
checkpointer_interval = 10
node_memory_limit_mb = 8192
txlog_service_list = 127.0.0.1:8001,127.0.0.1:8002,127.0.0.1:8003
enable_tx_metrics = true
enable_log_service_metrics = true

[storage]
storage_type = rocksdb
rocksdb_storage_path = /path/to/rocksdb
enable_cache_replacement = true

[log_service]
log_service_data_path = /path/to/log_service
rocksdb_scan_threads = 4

Engine Registration Order

1. eloqsql - First engine to register
2. eloqkv - Second engine to register
3. eloqdoc - Third engine (if compiled in)

Related Research

• Original ticket: GitHub Issue #128
• Multi-engine architecture enables converged database with shared data substrate

[liunyl]

Data Substrate Initialization Implementation Plan

Overview

This plan implements a unified data substrate initialization system that supports multiple database engines (eloqsql, eloqkv, eloqdoc) simultaneously. The data substrate will use a unified INI configuration file with gflags support, and all engine-specific code will be wrapped in compile-time macros.

Current State Analysis

What exists now:

• Separate initialization functions in eloqsql/storage/eloq/ha_eloq.cc:2397 and eloqkv/src/redis_service.cpp
• Duplicate data substrate components (tx_service, store_hd, log_service) in both projects
• Different configuration systems (MySQL variables vs gflags+INI)
• Engine-specific catalog factories and system handlers

What's missing:

• Unified data substrate initialization function
• Multi-engine catalog factory support in TxService
• Unified configuration system
• Compile-time engine selection mechanism

Desired End State

After this plan is complete:

• Single DataSubstrate class that can be initialized with multiple engines
• Unified INI configuration file with no engine-specific patterns
• TxService modified to accept catalog factory array for multiple engines
• All engine-specific code wrapped in #ifdef ELOQSQL, #ifdef ELOQKV, #ifdef ELOQDOC macros
• Modular initialization with separate helper functions for each component

Key Discoveries:

• Both projects use identical data substrate components but with different initialization patterns
• TxService currently expects single catalog factory but needs to support array of 3 engines
• Configuration systems are fundamentally different and need unification
• Engine-specific dependencies (CatalogFactory, SystemHandler) must be passed as parameters

What We're NOT Doing

• Maintaining backward compatibility with existing configuration systems
• Implementing automated testing (manual testing only)
• Adding resource isolation mechanisms (managed by data substrate)
• Supporting dynamic engine discovery (hard-coded registration only)

Implementation Approach

Create a modular DataSubstrate class with separate helper functions for each component initialization. All helper functions will be implemented in the same data_substrate.cpp file to avoid file fragmentation while maintaining modularity. Use compile-time macros for engine-specific code and unified INI configuration.

File Structure

data_substrate/
├── include/
│   └── data_substrate.h          # Main class definition
├── src/
│   └── data_substrate.cpp        # All initialization functions
└── config/
    └── sample_data_substrate.ini # Sample configuration

Note: All helper functions (configuration loading, storage handler init, log service init, engine registration, TxService init, service startup) are implemented as separate functions within the same data_substrate.cpp file to maintain modularity while reducing file fragmentation.

Phase 1: Create Data Substrate Foundation

Overview

Create the core DataSubstrate class structure and configuration system.

Changes Required:

1. Create Data Substrate Header

File: data_substrate/include/data_substrate.h
Changes: Create main DataSubstrate class with configuration structures

#pragma once

#include <memory>
#include <string>
#include <vector>
#include <map>
#include <INIReader.h>

// Forward declarations
namespace txservice {
    class TxService;
    class DataStoreHandler;
    class LogAgent;
}

namespace txlog {
    class LogServer;
}

class DataSubstrate {
public:
    struct Config {
        // Unified configuration structure (no engine-specific patterns)
        std::string data_path;
        std::string txlog_service_list;
        std::string storage_type;
        std::string rocksdb_storage_path;
        std::string log_service_data_path;
        
        // TxService configuration
        std::map<std::string, uint32_t> tx_service_config;
        
        // Storage configuration
        bool enable_cache_replacement;
        
        // Log service configuration
        uint32_t rocksdb_scan_threads;
        uint32_t txlog_group_replica_num;
        
        // Metrics configuration
        bool enable_tx_metrics;
        bool enable_log_service_metrics;
    };
    
    struct EngineConfig {
        class CatalogFactory* catalog_factory;
        class SystemHandler* system_handler;
        std::string engine_name;
    };
    
    static std::unique_ptr<DataSubstrate> Initialize(
        const std::string& config_file_path
    );
    
    // Component accessors
    txservice::TxService* GetTxService() const { return tx_service_.get(); }
    txservice::DataStoreHandler* GetStoreHandler() const { return store_hd_.get(); }
    txlog::LogServer* GetLogServer() const { return log_server_.get(); }
    
    ~DataSubstrate();

private:
    DataSubstrate() = default;
    
    // Helper functions for component initialization
    bool LoadConfiguration(const std::string& config_file_path);
    bool InitializeStorageHandler();
    bool InitializeLogService();
    bool RegisterEngines();
    bool InitializeTxService();
    bool StartServices();
    
    // Component instances
    std::unique_ptr<txservice::TxService> tx_service_;
    std::unique_ptr<txservice::DataStoreHandler> store_hd_;
    std::unique_ptr<txlog::LogServer> log_server_;
    std::unique_ptr<txservice::LogAgent> log_agent_;
    
    // Configuration
    Config config_;
    std::vector<EngineConfig> engines_;
    
    // Engine-specific instances (wrapped in compile macros)
#ifdef ELOQSQL
    static MariaCatalogFactory maria_catalog_factory_;
#endif
#ifdef ELOQKV
    static EloqKvCatalogFactory eloqkv_catalog_factory_;
#endif
#ifdef ELOQDOC
    static EloqDocCatalogFactory eloqdoc_catalog_factory_;
#endif
};

2. Create Configuration Loading Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement configuration loading from INI file in the same file as main class

#include "data_substrate.h"
#include <gflags/gflags.h>
#include <INIReader.h>

bool DataSubstrate::LoadConfiguration(const std::string& config_file_path) {
    INIReader config_reader(config_file_path);
    
    if (config_reader.ParseError() < 0) {
        return false;
    }
    
    // Load data substrate configuration
    config_.data_path = config_reader.Get("data_substrate", "data_path", "");
    config_.txlog_service_list = config_reader.Get("data_substrate", "txlog_service_list", "");
    config_.enable_tx_metrics = config_reader.GetBoolean("data_substrate", "enable_tx_metrics", true);
    config_.enable_log_service_metrics = config_reader.GetBoolean("data_substrate", "enable_log_service_metrics", true);
    
    // Load storage configuration
    config_.storage_type = config_reader.Get("storage", "storage_type", "rocksdb");
    config_.rocksdb_storage_path = config_reader.Get("storage", "rocksdb_storage_path", "");
    config_.enable_cache_replacement = config_reader.GetBoolean("storage", "enable_cache_replacement", true);
    
    // Load log service configuration
    config_.log_service_data_path = config_reader.Get("log_service", "log_service_data_path", "");
    config_.rocksdb_scan_threads = config_reader.GetInteger("log_service", "rocksdb_scan_threads", 4);
    config_.txlog_group_replica_num = config_reader.GetInteger("log_service", "txlog_group_replica_num", 3);
    
    // Load TxService configuration
    config_.tx_service_config["core_num"] = config_reader.GetInteger("data_substrate", "core_num", 8);
    config_.tx_service_config["checkpointer_interval"] = config_reader.GetInteger("data_substrate", "checkpointer_interval", 10);
    config_.tx_service_config["node_memory_limit_mb"] = config_reader.GetInteger("data_substrate", "node_memory_limit_mb", 8192);
    // ... other tx_service parameters
    
    return true;
}

Success Criteria:

Automated Verification:

• DataSubstrate class compiles successfully
• Configuration loading works with sample INI file
• All helper function declarations are present
• Compile macros are properly defined

Manual Verification:

• Configuration file can be parsed without errors
• Default values are applied correctly
• gflags integration works for command-line overrides

---

Phase 2: Implement Storage Handler Initialization

Overview

Create helper function to initialize storage handler with unified configuration.

Changes Required:

1. Create Storage Handler Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement storage handler initialization logic in the same file

#include "data_substrate.h"
#include "store_handler/rocksdb_handler.h"
#include "store_handler/dynamo_handler.h"
#include "store_handler/data_store_service_client.h"

bool DataSubstrate::InitializeStorageHandler() {
    if (config_.storage_type == "rocksdb") {
        // Initialize RocksDB handler
        EloqShare::RocksDBConfig rocksdb_config;
        rocksdb_config.storage_path = config_.rocksdb_storage_path;
        rocksdb_config.enable_cache_replacement = config_.enable_cache_replacement;
        
        store_hd_ = std::make_unique<EloqKV::RocksDBHandlerImpl>(
            rocksdb_config,
            false, // bootstrap - will be determined by engine
            config_.enable_cache_replacement
        );
    }
    else if (config_.storage_type == "dynamodb") {
        // Initialize DynamoDB handler
        std::string endpoint = config_reader.Get("storage", "dynamodb_endpoint", "");
        std::string region = config_reader.Get("storage", "dynamodb_region", "");
        std::string access_key = config_reader.Get("storage", "aws_access_key_id", "");
        std::string secret_key = config_reader.Get("storage", "aws_secret_key", "");
        std::string keyspace = config_reader.Get("storage", "keyspace_name", "eloq_data");
        
        store_hd_ = std::make_unique<EloqDS::DynamoHandler>(
            keyspace, endpoint, region, access_key, secret_key,
            false, // bootstrap
            false, // ddl_skip_kv
            config_.tx_service_config["core_num"] * 2
        );
    }
    else if (config_.storage_type == "data_store_service") {
        // Initialize Data Store Service client
        // Implementation for DSS client
    }
    
    if (!store_hd_) {
        return false;
    }
    
    return store_hd_->Connect();
}

Success Criteria:

Automated Verification:

• Storage handler initialization compiles successfully
• All storage types (rocksdb, dynamodb, dss) are supported
• Connection establishment is properly handled

Manual Verification:

• RocksDB storage handler connects successfully
• DynamoDB storage handler connects successfully (if configured)
• Error handling works for invalid configurations

---

Phase 3: Implement Log Service Initialization

Overview

Create helper function to initialize log service with unified configuration.

Changes Required:

1. Create Log Service Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement log service initialization logic in the same file

#include "data_substrate.h"
#include "log_service/log_server.h"
#include "log_service/log_agent.h"

bool DataSubstrate::InitializeLogService() {
    // Parse txlog service list
    std::vector<std::string> txlog_ips;
    std::vector<uint16_t> txlog_ports;
    
    if (!config_.txlog_service_list.empty()) {
        std::string token;
        std::istringstream txlog_ip_list_stream(config_.txlog_service_list);
        while (std::getline(txlog_ip_list_stream, token, ',')) {
            size_t c_idx = token.find_first_of(':');
            if (c_idx != std::string::npos) {
                txlog_ips.emplace_back(token.substr(0, c_idx));
                txlog_ports.emplace_back(std::stoi(token.substr(c_idx + 1)));
            }
        }
    }
    
    // Create log agent
    log_agent_ = std::make_unique<txservice::LogAgent>(
        static_cast<uint32_t>(config_.txlog_group_replica_num)
    );
    
    // Initialize log server if no standalone service provided
    if (txlog_ips.empty()) {
        // Initialize local log server
        std::string log_path = config_.data_path + "/log_service";
        
        log_server_ = std::make_unique<txlog::LogServer>(
            0, // node_id - will be set by engine
            0, // port - will be set by engine
            log_path,
            config_.rocksdb_scan_threads
        );
        
        if (!log_server_->Start()) {
            return false;
        }
    }
    
    return true;
}

Success Criteria:

Automated Verification:

• Log service initialization compiles successfully
• Log agent creation works correctly
• Log server initialization handles both standalone and local modes

Manual Verification:

• Local log server starts successfully
• Standalone log service connection works
• Log agent is properly configured

---

Phase 4: Implement Engine Registration System

Overview

Create helper function to register engines with compile-time macro support.

Changes Required:

1. Create Engine Registration Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement hard-coded engine registration in the same file

#include "data_substrate.h"

#ifdef ELOQSQL
#include "eloq_catalog_factory.h"
#include "eloq_system_handler.h"
#endif

#ifdef ELOQKV
#include "eloqkv_catalog_factory.h"
#endif

#ifdef ELOQDOC
#include "eloqdoc_catalog_factory.h"
#include "eloqdoc_system_handler.h"
#endif

bool DataSubstrate::RegisterEngines() {
    engines_.clear();
    
#ifdef ELOQSQL
    // Register eloqsql engine
    EngineConfig eloqsql_config;
    eloqsql_config.catalog_factory = &maria_catalog_factory_;
    eloqsql_config.system_handler = &MariaSystemHandler::Instance();
    eloqsql_config.engine_name = "eloqsql";
    engines_.push_back(eloqsql_config);
#endif

#ifdef ELOQKV
    // Register eloqkv engine
    EngineConfig eloqkv_config;
    eloqkv_config.catalog_factory = &eloqkv_catalog_factory_;
    eloqkv_config.system_handler = nullptr;  // eloqkv doesn't use system handler
    eloqkv_config.engine_name = "eloqkv";
    engines_.push_back(eloqkv_config);
#endif

#ifdef ELOQDOC
    // Register eloqdoc engine
    EngineConfig eloqdoc_config;
    eloqdoc_config.catalog_factory = &eloqdoc_catalog_factory_;
    eloqdoc_config.system_handler = &EloqDocSystemHandler::Instance();
    eloqdoc_config.engine_name = "eloqdoc";
    engines_.push_back(eloqdoc_config);
#endif
    
    return !engines_.empty();
}

Success Criteria:

Automated Verification:

• Engine registration compiles with different macro combinations
• All three engines can be registered when macros are enabled
• Registration works with single engine enabled

Manual Verification:

• eloqsql engine registers correctly when ELOQSQL macro is defined
• eloqkv engine registers correctly when ELOQKV macro is defined
• eloqdoc engine registers correctly when ELOQDOC macro is defined

---

Phase 5: Modify TxService for Multi-Engine Support

Overview

Modify TxService to accept catalog factory array and system handler array for multiple engines.

Changes Required:

1. Update TxService Constructor

File: tx_service/include/tx_service.h
Changes: Modify constructor to accept arrays for multiple engines

class TxService {
public:
    // Modified constructor for multi-engine support
    TxService(
        CatalogFactory* catalog_factory[3],  // Array for eloqsql, eloqkv, eloqdoc
        SystemHandler* system_handlers[3],   // Array for each engine
        const std::map<std::string, uint32_t>& config,
        uint32_t node_id,
        uint32_t native_ng_id,
        const std::unordered_map<uint32_t, std::vector<NodeConfig>>* ng_configs,
        uint32_t cluster_config_version,
        DataStoreHandler* store_hd,
        LogAgent* log_agent,
        bool enable_mvcc,
        bool skip_redo_log,
        bool skip_kv,
        bool enable_cache_replacement,
        bool auto_redirect,
        MetricsRegistry* metrics_registry,
        const CommonLabels& common_labels
    );
    
private:
    CatalogFactory* catalog_factories_[3];
    SystemHandler* system_handlers_[3];
    // ... other members
};

2. Update TxService Implementation

File: tx_service/src/tx_service.cpp
Changes: Update constructor implementation to handle multiple engines

TxService::TxService(
    CatalogFactory* catalog_factory[3],
    SystemHandler* system_handlers[3],
    const std::map<std::string, uint32_t>& config,
    // ... other parameters
) {
    // Copy catalog factories
    for (int i = 0; i < 3; i++) {
        catalog_factories_[i] = catalog_factory[i];
        system_handlers_[i] = system_handlers[i];
    }
    
    // Initialize with primary catalog factory (eloqsql)
    catalog_factory_ = catalog_factories_[0];  // eloqsql
    system_handler_ = system_handlers_[0];     // eloqsql
    
    // ... rest of initialization
}

Success Criteria:

Automated Verification:

• TxService compiles with new constructor signature
• All existing TxService functionality remains intact
• Multi-engine catalog factory array is properly handled

Manual Verification:

• TxService can be created with multiple catalog factories
• Primary engine (eloqsql) is used for main operations
• Other engines are accessible when needed

---

Phase 6: Implement TxService Initialization Helper

Overview

Create helper function to initialize TxService with all registered engines.

Changes Required:

1. Create TxService Initialization Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement TxService initialization with multi-engine support in the same file

#include "data_substrate.h"
#include "tx_service/tx_service.h"

bool DataSubstrate::InitializeTxService() {
    if (engines_.empty()) {
        return false;
    }
    
    // Prepare catalog factory and system handler arrays
    CatalogFactory* catalog_factories[3] = {nullptr, nullptr, nullptr};
    SystemHandler* system_handlers[3] = {nullptr, nullptr, nullptr};
    
    // Fill arrays with registered engines
    for (size_t i = 0; i < engines_.size() && i < 3; i++) {
        catalog_factories[i] = engines_[i].catalog_factory;
        system_handlers[i] = engines_[i].system_handler;
    }
    
    // Create TxService with multi-engine support
    tx_service_ = std::make_unique<txservice::TxService>(
        catalog_factories,
        system_handlers,
        config_.tx_service_config,
        0, // node_id - will be set by engine
        0, // native_ng_id - will be set by engine
        nullptr, // ng_configs - will be set by engine
        0, // cluster_config_version - will be set by engine
        store_hd_.get(),
        log_agent_.get(),
        false, // enable_mvcc
        false, // skip_redo_log
        false, // skip_kv
        config_.enable_cache_replacement,
        true, // auto_redirect
        nullptr, // metrics_registry - will be set by engine
        {} // common_labels - will be set by engine
    );
    
    if (!tx_service_) {
        return false;
    }
    
    // Set tx_service reference in storage handler
    if (store_hd_) {
        store_hd_->SetTxService(tx_service_.get());
    }
    
    return true;
}

Success Criteria:

Automated Verification:

• TxService initialization compiles successfully
• Multi-engine arrays are properly populated
• Storage handler reference is correctly set

Manual Verification:

• TxService is created with all registered engines
• Storage handler can access TxService
• All engines are properly initialized

---

Phase 7: Implement Service Startup Helper

Overview

Create helper function to start all services in correct order.

Changes Required:

1. Create Service Startup Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement service startup sequence in the same file

#include "data_substrate.h"

bool DataSubstrate::StartServices() {
    if (!tx_service_) {
        return false;
    }
    
    // Start TxService
    // Note: Engine-specific parameters will be provided by engines
    std::vector<std::string> txlog_ips;
    std::vector<uint16_t> txlog_ports;
    
    // Parse txlog service list for startup
    if (!config_.txlog_service_list.empty()) {
        std::string token;
        std::istringstream txlog_ip_list_stream(config_.txlog_service_list);
        while (std::getline(txlog_ip_list_stream, token, ',')) {
            size_t c_idx = token.find_first_of(':');
            if (c_idx != std::string::npos) {
                txlog_ips.emplace_back(token.substr(0, c_idx));
                txlog_ports.emplace_back(std::stoi(token.substr(c_idx + 1)));
            }
        }
    }
    
    // Start TxService (engine-specific parameters will be provided by engines)
    int start_result = tx_service_->Start(
        0, // node_id - will be set by engine
        0, // native_ng_id - will be set by engine
        nullptr, // ng_configs - will be set by engine
        0, // cluster_config_version - will be set by engine
        &txlog_ips,
        &txlog_ports,
        nullptr, // hm_ip - will be set by engine
        nullptr, // hm_port - will be set by engine
        nullptr, // hm_bin_path - will be set by engine
        config_.tx_service_config,
        std::move(log_agent_),
        config_.data_path,
        "" // cluster_config_file - will be set by engine
    );
    
    return start_result >= 0;
}

Success Criteria:

Automated Verification:

• Service startup compiles successfully
• TxService start method is called correctly
• Error handling is properly implemented

Manual Verification:

• TxService starts successfully
• All services are running after startup
• Error conditions are handled gracefully

---

Phase 8: Implement Main DataSubstrate Initialization

Overview

Create the main initialization function that orchestrates all helper functions.

Changes Required:

1. Create Main Initialization Function

File: data_substrate/src/data_substrate.cpp
Changes: Implement main initialization orchestration

#include "data_substrate.h"

std::unique_ptr<DataSubstrate> DataSubstrate::Initialize(
    const std::string& config_file_path
) {
    auto data_substrate = std::make_unique<DataSubstrate>();
    
    // Phase 1: Load configuration
    if (!data_substrate->LoadConfiguration(config_file_path)) {
        return nullptr;
    }
    
    // Phase 2: Initialize storage handler
    if (!data_substrate->InitializeStorageHandler()) {
        return nullptr;
    }
    
    // Phase 3: Initialize log service
    if (!data_substrate->InitializeLogService()) {
        return nullptr;
    }
    
    // Phase 4: Register engines
    if (!data_substrate->RegisterEngines()) {
        return nullptr;
    }
    
    // Phase 5: Initialize TxService
    if (!data_substrate->InitializeTxService()) {
        return nullptr;
    }
    
    // Phase 6: Start services
    if (!data_substrate->StartServices()) {
        return nullptr;
    }
    
    return data_substrate;
}

DataSubstrate::~DataSubstrate() {
    if (tx_service_) {
        tx_service_->Shutdown();
    }
    
    // Cleanup other components
    store_hd_ = nullptr;
    log_server_ = nullptr;
    log_agent_ = nullptr;
}

Success Criteria:

Automated Verification:

• Main initialization function compiles successfully
• All helper functions are called in correct order
• Error handling prevents partial initialization

Manual Verification:

• DataSubstrate initializes successfully with valid config
• All components are properly initialized
• Cleanup works correctly on destruction

---

Phase 9: Create Sample Configuration File

Overview

Create sample configuration file for testing and documentation.

Changes Required:

1. Create Sample Configuration

File: data_substrate/config/sample_data_substrate.ini
Changes: Create comprehensive sample configuration

[data_substrate]
data_path = /tmp/eloq_data
core_num = 8
checkpointer_interval = 10
node_memory_limit_mb = 8192
node_log_limit_mb = 1024
checkpointer_delay_seconds = 5
collect_active_tx_ts_interval_seconds = 2
realtime_sampling = true
txlog_service_list = 127.0.0.1:8001,127.0.0.1:8002,127.0.0.1:8003
txlog_group_replica_num = 3
enable_tx_metrics = true
enable_log_service_metrics = true
enable_cache_hit_rate = true
enable_kv_metrics = true
enable_remote_request_metrics = true
collect_tx_duration_round = 1
busy_round_threshold = 10

[storage]
storage_type = rocksdb
rocksdb_storage_path = /tmp/eloq_data/rocksdb
enable_cache_replacement = true
rocksdb_max_write_buffer_number = 3
rocksdb_max_background_jobs = 4
rocksdb_target_file_size_base = 67108864

[log_service]
log_service_data_path = /tmp/eloq_data/log_service
rocksdb_scan_threads = 4
logserver_snapshot_interval = 3600
enable_txlog_request_checkpoint = true
notify_checkpointer_threshold_size = 1GB

[metrics]
enable_metrics = true
metrics_port = 9090

Success Criteria:

Automated Verification:

• Sample configuration file is valid INI format
• All required parameters are included
• Default values are reasonable

Manual Verification:

• Configuration file can be loaded by DataSubstrate
• All parameters are properly parsed
• Default values work correctly

---

Testing Strategy

Unit Tests:

• Configuration loading with various INI file formats
• Storage handler initialization for different storage types
• Log service initialization with and without standalone service
• Engine registration with different macro combinations
• TxService initialization with multiple engines

Integration Tests:

• End-to-end DataSubstrate initialization
• Multi-engine TxService operation
• Storage handler and TxService interaction
• Log service and TxService interaction

Manual Testing Steps:

1. Create sample configuration file
2. Initialize DataSubstrate with eloqsql only
3. Initialize DataSubstrate with eloqkv only
4. Initialize DataSubstrate with both eloqsql and eloqkv
5. Verify all components are accessible
6. Test service startup and shutdown
7. Verify error handling with invalid configurations

Performance Considerations

• Configuration loading should be fast (single INI file read)
• Component initialization should be parallel where possible
• Memory usage should be minimal for unused engines
• Startup time should be comparable to current implementations

Migration Notes

• No backward compatibility required
• Existing engines will need to be modified to use new DataSubstrate
• Configuration files will need to be converted to new INI format
• Engine-specific initialization code will be moved to compile macros

References

• Original ticket: thoughts/shared/research/2025-09-25-ENG-128-data-substrate-initialization.md
• eloqsql initialization: eloqsql/storage/eloq/ha_eloq.cc:2397
• eloqkv initialization: eloqkv/src/redis_service.cpp
• TxService interface: tx_service/include/tx_service.h
• Storage handler interface: tx_service/include/store/data_store_handler.h

[liunyl]

Init function has been refactored into a separate data substrate init.
Eloqkv and eloqsql can be built with the new data substrate.
ConvergeDB can build and run eloqkv and eloqsql together.

[liunyl]

In progress:
clean up eloqkv code and merge into main. Maintain backward compatibility and update build script/doc.
TODO:
same for eloqsql
Refactor code to clarify start up process in convergedb