Skip to content

Conversation

@czgdp1807
Copy link

Summary

This PR modernizes the CI pipeline by adding builds with conda/mamba GitHub Actions workflows, enabling builds with better reproducibility and manylinux2014 compatibility. In future, Docker based based, build.yml and test.yml will be removed.

Changes

New Files

  • ci/environment.yml: Conda environment with GCC 8.5.0, sysroot_linux-64=2.17, and all build/test dependencies
  • .github/workflows/conda-build.yml: Workflow for building manylinux2014 wheels across Python 3.9-3.11
  • .github/workflows/conda-test.yml: Workflow for running pytest tests on built wheels

Key Improvements

  1. manylinux2014 Compatibility: Uses GCC 8.5.0 + sysroot_linux-64=2.17 to match CentOS 7 devtoolset-8 (equivalent to the Docker environment)
  2. Simplified Maintenance: Declarative environment.yml replaces complex Dockerfile configuration
  3. Native Execution: Runs directly on GitHub-hosted ubuntu-latest runners instead of inside Docker containers

Technical Details

  • Compiler Toolchain: GCC 8.5.0 with glibc 2.17 headers ensures binary compatibility with manylinux2014
  • Dependency Management: Uses mamba-org/setup-micromamba@v1 for environment setup
  • Wheel Repair: auditwheel repair --plat manylinux2014_x86_64 ensures PyPI compatibility
  • Bazel Management: Uses Bazelisk 1.20.0 for consistent Bazel 6.1.0 version management

Removed Dependencies (to-be-done after merging this PR)

  • No longer requires Docker or docker-compose for local development
  • Eliminates need for maintaining Dockerfiles and container configurations

Benefits

  • Developer Experience: Local builds easily reproducible with micromamba env create -f ci/environment.yml
  • Maintenance: Single environment.yml file vs multiple Docker configuration files
  • Debugging: Easier to troubleshoot builds running natively on runners
  • Compatibility: Maintains exact manylinux2014 compliance for PyPI distribution

Testing

Build workflow produces manylinux2014_x86_64 wheels for Python 3.9-3.11
Test workflow validates wheels install and pass all pytest tests
Wheels are uploadable to PyPI with automated trusted publishing

This is ready to merge because all CI tests are passing. Refer - czgdp1807#4

Migration Notes

This intends to replace the existing Docker-based build process. Future PRs can add macOS support by:

  1. Creating ci/environment-macos.yml with Clang compilers
  2. Expanding the matrix.os to include macos-latest
  3. Using delocate for macOS wheel repair

Add conda/micromamba-based GitHub Actions workflows for building and
testing ml-metadata wheels. Uses GCC 8.5.0 to match manylinux2014
compatibility, supports Python 3.9-3.11, and includes automated PyPI
publishing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant