Workflow to build python CLI snapshot for nightly publishing (sdist + wheels) #3036

HonahX · 2025-11-13T01:06:25Z

This PR include a github workflow, "Python Client Build Artifacts", to build wheels for multiple os/platform for nightly build and release automation in the future. It will produce a zip containing
contain sdist and wheels built for macos and linux (TODO: support windows)

> tar -tzf python-client-artifacts-20141056849.zip
apache_polaris-1.2.0-cp310-cp310-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp310-cp310-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0-cp310-cp310-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp311-cp311-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp311-cp311-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp311-cp311-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0-cp312-cp312-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp312-cp312-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp312-cp312-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0-cp313-cp313-macosx_14_0_arm64.whl
apache_polaris-1.2.0-cp313-cp313-macosx_15_0_arm64.whl
apache_polaris-1.2.0-cp313-cp313-manylinux_2_39_x86_64.whl
apache_polaris-1.2.0.tar.gz

The expected usage for this workflow is:

Triggered for every python client changes to verify that changes will not break wheel building
Called by other workflow (e.g. nightly build) like

python-client-snapshot-build:
    if: github.repository == 'apache/polaris'
    uses: ./.github/workflows/python-client-build-artifacts.yml

python-client-snapshot-publish:
  runs-on: ubuntu-latest
  needs: python-client-snapshot-build
  if: github.repository == 'apache/polaris'
  steps:
    - name: Download artifacts
      uses: actions/download-artifact@v4
      with:
        name: ${{ needs.python-client-snapshot-build.outputs.artifact-name }}
        path: dist/

    - name: List artifacts
      run: ls -la dist/

   - name: Publish to TestPyPI
     id: publish-testpypi
     continue-on-error: true
     uses: pypa/gh-action-pypi-publish@release/v1
     with:
       repository-url: https://test.pypi.org/legacy/
       skip-existing: true
       verbose: true
....

Called by maintainer directly for test purpose.

The build of a release candidate that include proper sig, checksum, is out-of-scope of this PR and should be included in separate PR/workflow for python client release automation

cc: @binarycat0 @MonkeyCanCode @flyrain @snazy

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

client/python/generate_clients.py

.github/workflows/python-client-release-pypi-artifact.yml

.github/workflows/python-client-build-artifacts.yml

.github/workflows/python-client-release-svn-artifact.yml

.github/workflows/python-release-build.yml

MonkeyCanCode · 2025-11-19T05:05:15Z

@HonahX very nice work. We are almost there to have package in PyPI.

snazy

Is there a particular reason to not extent the release-automation workflows?

We have to follow the ASF release policy. It seems there are some steps missing in this PR:

checksums
gpg signatures
RC staging
staging the source tarball and artifacts to vote on
eventually publishing exactly the artifacts that have been voted on

HonahX · 2025-11-20T03:30:28Z

Hi @snazy Thanks for highlighting these important elements of the final release candidate for python CLI. Given the number of missing pieces in the CLI release process, my goal is to separate the pipeline into 2 milestone:

Have a formalized way to build release artifacts (wheels) that will later be released to PyPI for users to install.
Have a formalized way to build and upload release candidate that include proper signature and checksum of release artifacts (release automation pipeline)

This PR aims to focus on the first milestone in order to have additional CI to ensure that all changes are compatible with the release process and unblock nightly build. The signature/checksum/release candidate are goals in the second milestone. This way we could first determine what to release and start testing the artifacts and then finishing the rest part. WDYT?

snazy · 2025-11-26T04:51:04Z

@HonahX is this PR a draft PR or not? The summary says yes, the PR status says no.

HonahX · 2025-11-26T05:46:27Z

Hi @snazy It's ready for review. I checked the current PR description and it does not seem to imply this as a draft. Could you please point me to the part of the summary that may lead to the wrong implication here? Just want to eliminate all potential confusion.

snazy

The PR adds completely new release workflows for separate Python releases.

We already have release workflows in place, so I do not see a point having a separate set of workflows.

Technically it's missing a lot of the requirements for release artifacts for ASF projects and the ASF release process.

A huge amount of time and work already went into the existing release automation, because it is not an easy task to get it right.

HonahX · 2025-12-02T03:24:56Z

Hi @snazy, I think there might be some confusion in the original PR summary and the workflow name. I should not put the "release" in the workflow title as the intention was not to make this a separate release procedure than the existing release automation. Ultimately, the python client release process should be part of the release automation.

I've updated the PR to simplify the workflow file and updated the summary to clarify the intention. We will focus on build the right wheels and sdist and make nightly build working first. Sorry for any confusion above. Please take a look again and let me know WDYT!

snazy · 2025-12-02T18:09:37Z

I can take another look, but due to time constraints likely not before next week.

flyrain · 2025-12-04T02:24:24Z

.github/workflows/python-client-build-artifacts.yml

+        uses: pypa/cibuildwheel@v3.1.4
+        with:
+          output-dir: wheelhouse
+          package-dir: "./client/python"
+          config-file: "{package}/pyproject.toml"
+        env:
+          # Ignore 32 bit architectures
+          CIBW_ARCHS: "auto64"
+          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
+          CIBW_REPAIR_WHEEL_COMMAND_MACOS: "" # Disable wheel repair since there will be no binary in the wheel
+          CIBW_REPAIR_WHEEL_COMMAND_LINUX: ""
+          CIBW_ENVIRONMENT: POLARIS_CLI_SKIP_CLIENT_GENERATION="true" # java is not available in linux containers during wheel build, skip client generation


Curious: does this command archive the same result?

run: | make client-build FORMAT=wheel

It should be mostly the same for MacOs/windows, but for linux it will help build both manylinux and musllinux wheel to have broad compatibility.

flyrain

LGTM. Thanks @HonahX !

snazy

I think we do not need the overhead of CPython.
Without that, I suspect the Python package build can be moved into the Makefile and the calling workflows would just need to setup the necessary Python versions.

Then the existing nightly-build workflow can be extended with one job to setup Python, call that Makefile target and publish to test.pypi.org.

snazy · 2025-12-08T10:39:07Z

.github/workflows/python-client-build-artifacts.yml

+          make client-build FORMAT=sdist
+
+      - name: Build wheels
+        uses: pypa/cibuildwheel@v3.1.4


Actions in ASF workflows must reference the Git SHA of an ASF-approved version of the action, see this policy.

The action docs mention that the wheels can contain libraries from the build container.
Which ones are included and what are the licenses for those libraries?

Good catch. We definitely should use the approved version here: https://github.com/apache/infrastructure-actions/blob/7a58482c11f14b79ea238d4a57093077cc18673c/approved_patterns.yml#L174C3-L174C61

The action docs mention that the wheels can contain libraries from the build container.

The dependencies may be brought by the wheel repair operation. In our case, we explicitly disabled the wheel repair:

CIBW_REPAIR_WHEEL_COMMAND_MACOS: "" CIBW_REPAIR_WHEEL_COMMAND_LINUX: ""

so no additional dependencies will be brought in.

But based on other comments and discussion, cibuildwheel may not be the best way here, let me try another approach first

Switched to use make client-build FORMAT=wheel directly instead of cibuildwheel

.github/workflows/python-client-build-artifacts.yml

snazy · 2025-12-08T10:46:22Z

.github/workflows/python-client-build-artifacts.yml

+    strategy:
+      matrix:
+        os: [ ubuntu-24.04, macos-14, macos-15] # current build script is not compatible with windows


Is there anything that's performance critical in the Python package?
I'd prefer to not build native code.

Yeah, ideally we only need a universal wheel since everything is pure python. However, since we need to leverage a custom build script to do open api client generation, poetry thinks it needs to generate platform-specific wheels. This is a known limitation and seems we have no native way to override this behavior now.

Should the situation changed in the future, we should definitely switch to universal wheel if possible.

snazy · 2025-12-08T10:47:41Z

.github/workflows/python-client-build-artifacts.yml

+          if [ -n "${{ inputs.version }}" ]; then
+            ARTIFACT_NAME="python-client-artifacts-${{ inputs.version }}"


What's the reason for this special case?

It could make the output zip file name indicate that a custom version has been provided hence the built sdist and wheels will have its package version changed. Just for convenience.

client/python/generate_clients.py

.github/workflows/python-client-build-artifacts.yml

snazy · 2026-01-07T10:14:47Z

I honestly do not think that this code is reusable from actual release automation, as many things like signing and checksums are not considered. Can you simplify this by focusing only on snapshot publishing?

- Remove version input - auto-generate PEP 440 compliant dev version Format: {base_version}.devYYYYMMDDHHMMSS+commitshort - Generate version once and share across all build jobs - Include version in final merged artifact name - Rename workflow to "Python Client Build Snapshot Artifacts"

- Add client-get-version Makefile target to get current version - Use make client-get-version instead of grep/sed parsing - Simpler version format: {version}.dev{timestamp} - Add artifact-name output for nightly workflow integration

HonahX · 2026-01-09T00:50:06Z

Hi @snazy I've removed the version input option and make it the snapshot version we want to tag our nightly build, e.g. 0.9.0.dev20231215

The workflow file is targeted to be used in 2 places:

PR that involves client change to make sure all wheels are successfully built
Integration to the nightly.yaml.

For the nightly build, the example usage is like

...

python-client-snapshot-build:
    if: github.repository == 'apache/polaris'
    uses: ./.github/workflows/python-client-build-artifacts.yml

  python-client-snapshot-publish:
    runs-on: ubuntu-latest
    needs: python-client-snapshot-build
    if: github.repository == 'apache/polaris'
    steps:
      - name: Download artifacts
        uses: actions/download-artifact@v4
        with:
          name: ${{ needs.python-client-snapshot-build.outputs.artifact-name }}
          path: dist/

      - name: List artifacts
        run: ls -la dist/

     - name: Publish to TestPyPI
       id: publish-testpypi
       continue-on-error: true
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
         repository-url: https://test.pypi.org/legacy/
         skip-existing: true
         verbose: true

In this way we do not need to put the multi-OS wheel built logic in the nightly publishing workflow. The artifacts downloaded in python-client-snapshot-publish will look like

Contents of dist/:
total 6676
drwxr-xr-x 2 runner runner   4096 Jan  8 04:46 .
drwxr-xr-x 3 runner runner   4096 Jan  8 04:46 ..
-rw-r--r-- 1 runner runner 546036 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp310-cp310-macosx_14_0_arm64.whl
-rw-r--r-- 1 runner runner 546038 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp310-cp310-macosx_15_0_arm64.whl
-rw-r--r-- 1 runner runner 546041 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp310-cp310-manylinux_2_39_x86_64.whl
-rw-r--r-- 1 runner runner 546036 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp311-cp311-macosx_14_0_arm64.whl
-rw-r--r-- 1 runner runner 546037 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp311-cp311-macosx_15_0_arm64.whl
-rw-r--r-- 1 runner runner 546040 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp311-cp311-manylinux_2_39_x86_64.whl
-rw-r--r-- 1 runner runner 546037 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp312-cp312-macosx_14_0_arm64.whl
-rw-r--r-- 1 runner runner 546038 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp312-cp312-macosx_15_0_arm64.whl
-rw-r--r-- 1 runner runner 546039 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp312-cp312-manylinux_2_39_x86_64.whl
-rw-r--r-- 1 runner runner 546039 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp313-cp313-macosx_14_0_arm64.whl
-rw-r--r-- 1 runner runner 546038 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp313-cp313-macosx_15_0_arm64.whl
-rw-r--r-- 1 runner runner 546040 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602-cp313-cp313-manylinux_2_39_x86_64.whl
-rw-r--r-- 1 runner runner 237998 Jan  8 04:46 apache_polaris-1.4.0.dev20260108043602.tar.gz

Please let me know WDYT

HonahX · 2026-01-11T03:00:13Z

If we migrate to uv: #3410, we may no longer need this as the new backend can build universal wheel even using custom build script

MonkeyCanCode · 2026-01-11T07:04:11Z

If we migrate to uv: #3410, we may no longer need this as the new backend can build universal wheel even using custom build script

Exactly.

snazy · 2026-01-12T12:08:47Z

Hi @HonahX , I might not have phrased my previous reply as clear as I wanted it to be. While I think the recent cleanup puts this PR in a much better shape (thanks for that!), I think the workflow should build the artifacts and also publish those directly to test-pypi. This is because (chained) GitHub workflow invocations have some restrictions, but more importantly to keep the (not so complex) logic in a single workflow (file).

(Edit: lost an important word during writing)

HonahX · 2026-01-14T17:42:44Z

@snazy Thanks for the explanation. I think the previous complexity here is that we need to build wheels for different operating system and that's why I want a separate workflow for that to prevent over-complicating the publishing workflow. But with the uv migration happened recently, we can build universal wheel and we do not need this anymore. Thanks for reviewing! I will close this one.

github-project-automation bot added this to Basic Kanban Board Nov 13, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Nov 13, 2025

HonahX linked an issue Nov 14, 2025 that may be closed by this pull request

Python CLI Publication: Github workflow to build artifacts #3047

Open

HonahX changed the title ~~[Draft] Workflows to build python CLI release and pypi artifacts~~ Workflows to build python CLI release and pypi artifacts Nov 19, 2025

HonahX commented Nov 19, 2025

View reviewed changes

client/python/generate_clients.py Outdated Show resolved Hide resolved

HonahX marked this pull request as ready for review November 19, 2025 04:41

MonkeyCanCode self-requested a review November 19, 2025 04:48

MonkeyCanCode reviewed Nov 19, 2025

View reviewed changes

snazy reviewed Nov 19, 2025

View reviewed changes

HonahX force-pushed the honahx-python-release-2 branch from 9247338 to 36a45e4 Compare November 25, 2025 21:41

snazy requested changes Nov 26, 2025

View reviewed changes

HonahX changed the title ~~Workflows to build python CLI release and pypi artifacts~~ Workflows to build python CLI artifacts (sdist + wheels) Dec 2, 2025

HonahX changed the title ~~Workflows to build python CLI artifacts (sdist + wheels)~~ Workflow to build python CLI artifacts (sdist + wheels) Dec 2, 2025

HonahX requested review from MonkeyCanCode, binarycat0 and snazy December 2, 2025 17:34

flyrain reviewed Dec 4, 2025

View reviewed changes

flyrain previously approved these changes Dec 4, 2025

View reviewed changes

MonkeyCanCode previously approved these changes Dec 4, 2025

View reviewed changes

snazy reviewed Dec 8, 2025

View reviewed changes

HonahX dismissed stale reviews from MonkeyCanCode and flyrain via 7facfa4 December 11, 2025 17:01

HonahX force-pushed the honahx-python-release-2 branch from abe51b0 to 2578b89 Compare December 11, 2025 18:09

HonahX requested a review from snazy December 11, 2025 18:10

HonahX added 8 commits January 7, 2026 13:56

Add artifact build pipeline

79fdef4

fix license

2b1d3dc

Fix format

46d1e77

Fix nit and format issue

87d6409

Simplify build file

de01a6c

Remove the need of cibuildwheel

6557cce

Remove unnecessary step

927782e

HonahX force-pushed the honahx-python-release-2 branch from 2578b89 to f5c52e3 Compare January 8, 2026 00:15

HonahX added 3 commits January 8, 2026 16:55

Simplify snapshot version generation

96ef5b5

- Add client-get-version Makefile target to get current version - Use make client-get-version instead of grep/sed parsing - Simpler version format: {version}.dev{timestamp} - Add artifact-name output for nightly workflow integration

Fix the github action commits

eca2d97

Fix version issue

15c4274

HonahX changed the title ~~Workflow to build python CLI artifacts (sdist + wheels)~~ Workflow to build python CLI snapshot for nightly publishing (sdist + wheels) Jan 9, 2026

HonahX mentioned this pull request Jan 11, 2026

(feat)Python CLI: Switch from Poetry to UV for python package management #3410

Merged

6 tasks

HonahX closed this Jan 14, 2026

github-project-automation bot moved this from PRs In Progress to Done in Basic Kanban Board Jan 14, 2026

		if [ -n "${{ inputs.version }}" ]; then
		ARTIFACT_NAME="python-client-artifacts-${{ inputs.version }}"

Workflow to build python CLI snapshot for nightly publishing (sdist + wheels) #3036

Workflow to build python CLI snapshot for nightly publishing (sdist + wheels) #3036

Uh oh!

Conversation

HonahX commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MonkeyCanCode commented Nov 19, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX commented Nov 20, 2025

Uh oh!

snazy commented Nov 26, 2025

Uh oh!

HonahX commented Nov 26, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX commented Dec 2, 2025

Uh oh!

snazy commented Dec 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snazy commented Jan 7, 2026

Uh oh!

HonahX commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HonahX commented Jan 11, 2026

Uh oh!

MonkeyCanCode commented Jan 11, 2026

Uh oh!

snazy commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HonahX commented Jan 14, 2026

Uh oh!

Reviewers

HonahX commented Nov 13, 2025 •

edited

Loading

HonahX Dec 11, 2025 •

edited

Loading

HonahX commented Jan 9, 2026 •

edited

Loading

snazy commented Jan 12, 2026 •

edited

Loading