Remove ZetaSQL and Modernize Build Toolchain

[czgdp1807]

Overview

This PR removes the ZetaSQL dependency, modernizes the build toolchain to support Ubuntu 24.04, GCC 13, Python 3.13, and enables native builds on macOS ARM64 (Apple Silicon). The changes improve long-term maintainability, reduce build complexity, and eliminate Docker dependencies while maintaining backward compatibility across Linux and macOS platforms.

ZetaSQL Removal

Removed ZetaSQL integration and the filter_query functionality to:

• Eliminate complex dependency chain (20+ transitive dependencies)
• Simplify build process and reduce build times
• Enable toolchain modernization without ZetaSQL compatibility constraints
• Reduce maintenance burden for rarely-used advanced filtering feature

Changes:

• Removed com_google_zetasql from WORKSPACE
• Disabled filter_query in query executors (returns UnimplementedError)
• Removed filter query tests from metadata_store_test.py
• Commented out filter query dependencies in BUILD files

Impact: Users relying on filter_query should migrate to standard list operations with client-side filtering.

Toolchain Updates Completed

Bazel Dependencies

• rules_foreign_cc: 0.9.0 -> 0.12.0 (removed obsolete patch)
• Bazel Skylib: 1.5.0 -> 1.7.1
• Google Test: 1.12.1 -> 1.15.2
• pybind11: 2.10.1 -> 2.13.6 (adds Python 3.13 support)
• SQLite: 3.39.2 -> 3.47.2
• zlib: 1.2.12 -> 1.3.1 (fixes fdopen macro conflicts on newer systems)
• upb: Added patch to fix C23 extension errors on modern Clang
• Abseil: 20230802.1 (kept for compatibility - see below)

Build Configuration (.bazelrc)

• Cross-Platform Support: Added platform-specific configurations for Linux and macOS ARM64
• macOS ARM64:
  • CPU target: darwin_arm64
  • Linker flags: -Wl,-undefined,dynamic_lookup for proper Python extension symbol resolution
  • C23 extension handling: -Wno-error=c23-extensions for modern Clang compatibility
  • Conda integration: CMAKE_ICONV_FLAG environment variable for libmysqlclient iconv configuration
• Linux-Specific: GNU linker flags (--no-as-needed) applied only on Linux
• C/C++ Standards:
  • C11 for all C code (--conlyopt=-std=c11)
  • C++17 for all C++ code including host tools (--cxxopt/-std=c++17, --host_cxxopt="-std=c++17")
• Compiler Warnings:
  • C-only flags (-Wno-array-parameter, -Wno-implicit-function-declaration) via --conlyopt
  • Permissive mode for backward compatibility (-fpermissive)
  • Error suppression for legacy code (-Wno-error)

Platform-Specific Fixes

• libmysqlclient.BUILD:
  • Added darwin_arm64 config for Apple Silicon
  • Linux: Explicit uint/ushort/ulong type mappings for GCC 13 / Ubuntu 24.04
  • macOS: -liconv linking for character encoding support via conda libiconv
  • Added _GNU_SOURCE to fix implicit function declarations
• MySQL iconv: Conditional CMake configuration via CMAKE_ICONV_FLAG environment variable

CI/CD - Conda-Based Native Builds

Replaced Docker-based CI with native conda builds for better platform support and maintainability.

• Removed: .github/workflows/build.yml and .github/workflows/test.yml (Docker-based)
• Added:
  • .github/workflows/conda-build.yml: Native build workflow with micromamba
  • .github/workflows/conda-test.yml: Native test workflow
  • ci/environment-macos.yml: macOS-specific conda environment (libiconv, delocate)
  • ci/environment.yml: Linux-specific conda environment (GCC 8.x, auditwheel, manylinux2014 sysroot)
• Matrix:
  • Platforms: ubuntu-latest, macos-latest (x86_64 and ARM64 via GitHub-hosted runners)
  • Python: 3.9, 3.10, 3.11, 3.13
• Features:
  • OS-conditional environment files using GitHub Actions ternary expressions
  • Bazelisk 1.20.0 for consistent Bazel version management
  • Platform-specific wheel repair: auditwheel (Linux manylinux2014), delocate (macOS)

Updates Intentionally Avoided

Bazel 6.5.0 (7.4.1 attempted)

Reason: gRPC 1.46.3 incompatible with Bazel 7.x
Error: 'apple_common.multi_arch_split' value has no field or method 'platform_type'
Blocker: Requires gRPC upgrade to 1.50+ which has breaking API changes

Abseil 20230802.1 (20240722.0 attempted)

Reason: API changes break existing code
Error: 'StrCat' is not a member of 'absl'
Blocker: Requires comprehensive codebase refactoring for new Abseil API

Protobuf 3.21.9 (3.25.5 attempted)

Reason: Bazel rules removed in newer versions
Error: file '@com_google_protobuf//:protobuf.bzl' does not contain symbol 'cc_proto_library'
Blocker: Requires migration to new Protobuf Bazel rules (protobuf-22.x or 4.x)

PostgreSQL 12.1 (16.6 and 17.2 attempted)

Reason: File structure completely changed between major versions
Error: missing input file '@postgresql//:src/port/thread.c' (and others)
Blocker: Requires complete rewrite of postgresql.BUILD for new file structure

Next Steps

Future Improvements

1. PostgreSQL 16.x/17.x Migration: Refactor postgresql.BUILD for modern PostgreSQL versions
2. Bazel 7.x Migration: Coordinate with gRPC upgrade (requires significant effort)
3. Protobuf 4.x Migration: Plan migration path to modern Protobuf rules
4. Python 3.13 Wheels: Wait for manylinux image to add Python 3.13 support
5. Dependency Cleanup: Remove GCC warning suppressions after upstream fixes
6. macOS ARM64 Runner: Migrate to macos-14 when available for native ARM64 CI builds

Testing Checklist

• Conda-based builds complete successfully (Python 3.9, 3.10, 3.11, 3.13)
• Unit tests pass in CI
• Native Linux (Ubuntu 24.04) builds verified
• Native macOS ARM64 (Apple Silicon) builds verified
• Cross-platform CI/CD workflows operational
• Wheel repair works correctly (auditwheel on Linux, delocate on macOS)

Compatibility

• Breaking Changes: filter_query feature disabled
• Python Support: 3.9, 3.10, 3.11, 3.13 (wheels for 3.9-3.11)
• Platform Support: Linux x86_64 (manylinux2014), macOS x86_64, macOS ARM64 (Apple Silicon)
• Build System: Native conda-based builds (no Docker required)
• Backward Compatible: All other APIs unchanged

---

Tested on:

• Ubuntu 24.04 (native + GitHub Actions)
• macOS ARM64 (Apple M1/M2/M3)
• macOS x86_64 (via GitHub Actions macos-latest)

Build System: Bazel 6.1.0 via Bazelisk 1.20.0

[czgdp1807]

Build Performance Improvement

Before vs After

Branch	Build Time
master (with ZetaSQL)	8m 4.9s
upgrade_deps (without ZetaSQL)	2m 6.8s

Improvement: 5m 58s faster (74% reduction) – ~3.8x speedup

Performance Analysis

The dramatic build time improvement is almost due to ZetaSQL removal:

Primary Factor: ZetaSQL Dependency Chain (~6 minutes saved)

• Eliminated 20+ transitive dependencies that previously required compilation
• ZetaSQL's heavy dependencies dominated build time
• Removed complex C++ template instantiations from ZetaSQL's query processing layer
• No longer building ZetaSQL's analyzer, evaluator, and parser components

Toolchain Modernization

The dependency upgrades (Bazel Skylib, rules_foreign_cc, Google Test, SQLite) likely contributed ** to build time improvement**. These updates primarily provide:

• Long-term maintainability improvements
• Modern language feature support (C++17, Python 3.13)

Conclusion

ZetaSQL removal is responsible for virtually all of the 74% build time reduction. The toolchain modernization delivers value through improved maintainability and future compatibility.

Machine Information

czgdp1807@qgpu1:~/ml-metadata$ uname -a
Linux qgpu1 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec  5 13:09:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
czgdp1807@qgpu1:~/ml-metadata$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          43 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   48
  On-line CPU(s) list:    0-47
Vendor ID:                AuthenticAMD
  Model name:             AMD Ryzen Threadripper 2970WX 24-Core Processor
    CPU family:           23
    Model:                8
    Thread(s) per core:   2
    Core(s) per socket:   24
    Socket(s):            1
    Stepping:             2
    Frequency boost:      enabled
    CPU(s) scaling MHz:   74%
    CPU max MHz:          3000.0000
    CPU min MHz:          2200.0000
    BogoMIPS:             5987.95
czgdp1807@qgpu1:~/ml-metadata$ grep MemTotal /proc/meminfo | awk '{print $2/1024/1024 " GB
"}'
125.726 GB

[czgdp1807]

Native build on linux is now possible with setup_and_build.sh. Please let me know if you face any issues while using it.