Add GitHub workflows for automated tag-based releases #498

jesuino · 2025-11-12T20:00:26Z

Could we also have a daily publish for Kale backend at least? This way we could use the latest published dev version without having to build the backend locally for developing for the frontend!

In principle I agree that having a nightly version would simplify fronted dev. Not sure I have seen projects doing this directly to PyPI. After a quick look online it seems to me that PyPI might cap due to size limits at some point, so nightly might not be recommended. Some folks do this using test.pipy.org (https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152)

We could think of publishing a nightly there, although there doesn't seems to be an automated garbage collection mechanism as well, not sure if at some point we will need to manually remove old versions.

Worth looking into what other projects are doing!

jesuino · 2025-11-12T19:59:17Z

I tried to follow these steps quite a few times and I noticed that the pipeline generated by Kale will still refer to localhost(127.0.0.1) devpi. But it fails due the fact that KFP is adding the pip index-url automatically to trusted host (see this issue).

In any case, if we fix that, we would still have to figure out how to fix the problem with an accessible devpi for KFP - in this case I think that the best solution would be running devpi on a pod in kubeflow namespace. (great thing that you made INDEX_URL a parameter!)

Does this make sense?

I feel like there are two options:

for super simple local dev setups using a minimal k8s setup for KFP (e.g. via kind), exposing somehow the local devpi to the local k8s

indeed running devpi inside the cluster for testing scenario involving clusters running remotely or other use cases more difficult to manage.

We should probably discuss this more in a separate issue, since this is more about the local dev experience rather than publishing a new release. The initial PR was clearly incomplete and cannot cover all use cases.

It makes sense to me! Let's discuss somewhere else

-Original file line number
+Diff line change
@@ -0,0 +1,21 @@
+    name: Create GitHub release
+    on:
+      push:
+        tags:
+          - 'v*'
+    permissions:
+      contents: write
+    jobs:
+      create-release:
+        runs-on: ubuntu-latest
+        steps:
+          - name: Create release with generated notes
+            uses: softprops/action-gh-release@v2
+            with:
+              tag_name: ${{ github.ref_name }}
+              generate_release_notes: true
+            env:
+              GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

-Original file line number
+Diff line change
@@ -0,0 +1,120 @@
+    name: Publish release artifacts
+    on:
+      release:
+        types: [published]
+      workflow_dispatch:
+        inputs:
+          tag:
+            description: 'Git ref (tag or branch) to publish or test'
+            required: true
+            type: string
+          mode:
+            description: 'Where to publish artifacts'
+            required: true
+            default: release
+            type: choice
+            options:
+              - release
+              - test
+    permissions:
+      contents: read
+    jobs:
+      publish-backend:
+        if: ${{ github.event_name != 'release' || !github.event.release.prerelease }}
+        runs-on: ubuntu-latest
+        env:
+          RELEASE_MODE: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.mode || 'release' }}
+          RELEASE_TAG: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.tag || github.event.release.tag_name }}
+        steps:
+          - name: Check out release tag
+            uses: actions/checkout@v4
+            with:
+              ref: ${{ env.RELEASE_TAG }}
+              fetch-depth: 0
+          - name: Set up Python
+            uses: actions/setup-python@v5
+            with:
+              python-version: '3.11'
+          - name: Install build tooling
+            run: |
+              python -m pip install --upgrade pip
+              python -m pip install build
+          - name: Build backend distribution
+            working-directory: backend
+            run: python -m build
+          - name: Publish backend to PyPI
+            if: env.RELEASE_MODE == 'release'
+            uses: pypa/gh-action-pypi-publish@release/v1
+            with:
+              packages_dir: backend/dist
+              password: ${{ secrets.PYPI_API_TOKEN }}
+          - name: Publish backend to TestPyPI
+            if: env.RELEASE_MODE == 'test'
+            uses: pypa/gh-action-pypi-publish@release/v1
+            with:
+              packages_dir: backend/dist
+              repository-url: https://test.pypi.org/legacy/
+              password: ${{ secrets.TEST_PYPI_API_TOKEN }}
+      publish-labextension:
+        if: ${{ github.event_name != 'release' || !github.event.release.prerelease }}
+        needs: publish-backend
+        runs-on: ubuntu-latest
+        env:
+          RELEASE_MODE: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.mode || 'release' }}
+          RELEASE_TAG: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.tag || github.event.release.tag_name }}
+          NODE_AUTH_TOKEN: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.mode == 'test' && secrets.TEST_NPM_TOKEN || secrets.NPM_TOKEN }}
+        steps:
+          - name: Check out release tag
+            uses: actions/checkout@v4
+            with:
+              ref: ${{ env.RELEASE_TAG }}
+              fetch-depth: 0
+          - name: Set up Node.js
+            uses: actions/setup-node@v4
+            with:
+              node-version: '20'
+              registry-url: 'https://registry.npmjs.org'
+          - name: Set up Python
+            uses: actions/setup-python@v5
+            with:
+              python-version: '3.11'
+          - name: Install JupyterLab build tooling
+            working-directory: labextension
+            run: |
+              python -m pip install --upgrade pip
+              python -m pip install 'jupyterlab>=4.2,<5.0'
+          - name: Install Node dependencies
+            working-directory: labextension
+            run: |
+              corepack enable
+              jlpm install --frozen-lockfile
+          - name: Build labextension assets
+            working-directory: labextension
+            run: jlpm build:prod
+          - name: Publish labextension to npm
+            if: env.RELEASE_MODE == 'release'
+            working-directory: labextension
+            run: npm publish
+          - name: Exercise npm publish workflow (dry run)
+            if: env.RELEASE_MODE == 'test'
+            working-directory: labextension
+            run: |
+              npm publish --dry-run
+              mkdir -p dist
+              npm pack --pack-destination dist

-Original file line number
+Diff line change
@@ -0,0 +1,126 @@
+    # Contributing to the Kale Backend
+    This guide walks through setting up a local development environment for the Kale
+    backend, explains how the package version flows through the system, and shows how
+    to iterate safely using a temporary Python package index.
+    > The backend lives under `backend/`. All commands below assume you run them from
+    > the repository root unless noted otherwise.
+    ## 1. Prerequisites
+    - Python 3.10 (managed via `pyenv`, `conda`, or your system package manager).
+    - Recent `pip` and `virtualenv`/`venv`.
+    - `node`/`yarn` are only required when working on the labextension (not covered here).
+    - Optional but recommended: `devpi-server` and `devpi-client` for local package
+      publishing (see “Local package index”).
+    ## 2. Create a virtual environment
+    ```bash
+    python -m venv .venv
+    source .venv/bin/activate
+    python -m pip install --upgrade pip
+    ```
+    ## 3. Install the backend in editable mode
+    ```bash
+    cd backend
+    pip install -e .[dev]
+    ```
+    The `pip install -e .[dev]` step relies on `backend/pyproject.toml`:
+    - Dependencies are declared in `[project] dependencies`.
+    - `setuptools_scm` infers the package version from git tags, so the version number
+      evolves automatically (`1.2.3` → `1.2.3.devN+gHASH` for untagged commits). You do
+      *not* have to edit any file to bump versions during development.
+    ## 4. Understanding version detection in code generation
+    Kale exposes its version at runtime via `kale.__version__`. When a notebook is
+    compiled into a Kubeflow Pipelines (KFP) component:
+    - `kale.compiler.Compiler._get_package_list_from_imports` always includes a Kale
+      dependency. If a concrete version is available (`__version__ != "0+unknown"`),
+      the component pins to `kubeflow-kale==<version>`. Otherwise it falls back to an
+      unpinned `kubeflow-kale`.
+    - `kale.common.utils.compute_pip_index_urls` determines which PyPI indexes are
+      baked into the generated component. It honours environment variables in the
+      following order:
+. `KALE_PIP_INDEX_URLS` – comma separated list, highest priority.
+. `KALE_DEV_MODE` + optional `KALE_DEVPI_SIMPLE_URL` – enables your local devpi.
+. Default fallback – `https://pypi.org/simple`.
+    Because compiled workflows freeze both the Kale version and the index list,
+    updating your local package copy requires re‑publishing the wheel/sdist that the
+    pipeline will install. That is why a disposable package index (devpi) is useful –
+    you can iterate without uploading experimental builds to the public PyPI.
+    ## 5. Working with the local devpi index
+    The `backend/scripts/` directory provides helpers for spinning up a local devpi
+    registry and publishing the freshly built Kale package:
+    | Script | Purpose | Typical usage |
+    | ------ | ------- | ------------- |
+    | `devpi-up.sh` | Start (or reuse) a devpi-server instance on `HOST:PORT` and create a volatile `root/<index>` derived from the public PyPI mirror. Prints the environment variables needed for pip. | `./scripts/devpi-up.sh` |
+    | `devpi-publish.sh` | Build Kale (`python -m build`), infer the newest artifact version from `dist/`, remove any existing release with the same version from the target devpi index, then upload the artifacts. | `./scripts/devpi-publish.sh` |
+    | `devpi-main.sh` | Convenience wrapper that calls the two scripts above and exports `KALE_DEV_MODE=1` + `KALE_DEVPI_SIMPLE_URL` in the current shell. | `source ./scripts/devpi-main.sh` |
+    Recommended loop when iterating on compiled pipelines:
+    ```bash
+    source .venv/bin/activate
+    cd backend
+    # Start devpi and export KALE_DEV_MODE / KALE_DEVPI_SIMPLE_URL / pip overrides.
+    source scripts/devpi-main.sh
+    # Make your code changes…
+    # Rebuild and publish to the local index (removes the previous dev build).
+    ./scripts/devpi-publish.sh
+    # Re-run notebook compilation so the generated component pulls from devpi.
+    kale --help  # or your usual CLI invocation pointing at the notebook
+    # For example you can run this quick test to validate the the generated DSL points
+    # for the correct Kale version
+    # kale --nb ./examples/base/candies_sharing.ipynb --dev
+    ```
+    > Note: the scripts rely on `devpi-server`, `devpi-client`, and `python -m build`.
+    > Install them once in your virtualenv: `pip install devpi-server devpi-client build`.
+    ## 6. Running tests
+    ```bash
+    cd backend
+    python -m pytest backend/kale/tests/unit_tests
+    ```
+    Integration tests under `backend/kale/tests/e2e/` perform notebook → DSL
+    compilation and compare against golden files in `backend/kale/tests/assets/`.
+    If you alter template output, update the generated files accordingly.
+    ## 7. Formatting and linting
+    - Python formatting is enforced via the existing code style (PEP 8). Many generated
+      files run through `autopep8`.
+    - The labextension has independent tooling (`jlpm lint`, `jlpm prettier`). Run
+      those only if you modify the frontend package.
+    ## 8. Typical development checklist
+. Activate your virtual environment.
+. `pip install -e .[dev]`.
+. Optionally start `devpi` via `source scripts/devpi-main.sh`.
+. Implement and re-publish using `scripts/devpi-publish.sh` when you need a new
+       dev build.
+. Update fixtures (`backend/kale/tests/assets/kfp_dsl/*.py`) whenever template
+       output changes.
+. Run unit and (when relevant) e2e tests.
+. Before committing, ensure `git status` shows only intentional changes.
+    Happy hacking! If anything in this guide is unclear, open an issue or PR with
+    improvements. Contributions are welcome.***

-Original file line number
+Diff line change
@@ -0,0 +1,5 @@
+    # Kubeflow Kale Backend
+    This package contains the Kubeflow Kale Python backend. For general
+    project documentation and usage instructions, please refer to the
+    repository root [README](../README.md).

-Original file line number
+Diff line change
@@ Expand Up / @@ -12,6 +12,27 @@ @@
     # See the License for the specific language governing permissions and
     # limitations under the License.
+    # --- Detect package version at runtime ---
+    # Use:
+    # from kale import __version__ as KALE_VERSION
+    # use KALE_VERSION wherever a display/log/version check is needed
+    try:
+        from importlib.metadata import version as _pkg_version, PackageNotFoundError
+    except Exception:  # Py<3.8 fallback if needed
+        from importlib_metadata import (  # type: ignore
+            version as _pkg_version,
+            PackageNotFoundError,
+        )
+    try:
+        __version__ = _pkg_version("kubeflow-kale")
+    except PackageNotFoundError:
+        # this might happen when a developer tried to test Kale locally from source
+        # without installing it first.
+        __version__ = "0+unknown"
+    # -----------------------------------------
     from typing import NamedTuple, Any
@@ Expand All / @@ -34,6 +55,18 @@ class Artifact(NamedTuple): @@
     from .processors import NotebookProcessor, NotebookConfig, PythonProcessor
     from kale.common import logutils
-    __all__ = ["PipelineParam", "Artifact",'NotebookProcessor', 'Step', 'StepConfig', 'Pipeline', 'PipelineConfig', 'VolumeConfig', 'Compiler', 'marshal']
+    __all__ = [
+        "PipelineParam",
+        "Artifact",
+        "NotebookProcessor",
+        "Step",
+        "StepConfig",
+        "Pipeline",
+        "PipelineConfig",
+        "VolumeConfig",
+        "Compiler",
+        "marshal",
+    ]
     logutils.get_or_create_logger(module=__name__, name="kale")
     del logutils

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GitHub workflows for automated tag-based releases #498

Diff view

Diff view

There are no files selected for viewing

jesuino Nov 12, 2025

Uh oh!

StefanoFioravanzo Nov 20, 2025 •

edited

Loading

Uh oh!

jesuino Nov 12, 2025

Uh oh!

StefanoFioravanzo Nov 20, 2025

Uh oh!

jesuino Nov 25, 2025

Uh oh!

Uh oh!

-Original file line number
+Diff line change
@@ Expand Up / @@ -12,9 +12,8 @@ @@
     #  See the License for the specific language governing permissions and
     #  limitations under the License.
+    import os
     import argparse
-    # import os
-    # import warnings
     from argparse import RawTextHelpFormatter
     from kale.processors import NotebookProcessor
@@ Expand Down Expand Up / @@ -60,6 +59,26 @@ def main(): @@
         general_group.add_argument('--run_pipeline', action='store_const',
                                    const=True)
         general_group.add_argument('--debug', action='store_true')
+        general_group.add_argument(
+            '--dev',
+            action='store_true',
+            help='Bake local dev index (devpi) into generated components.',
+        )
+        general_group.add_argument(
+            '--pip-index-urls',
+            type=str,
+            help=('Comma-separated PEP 503 simple indexes to bake into components.'
+                  'Overrides --dev/KALE_DEV_MODE. Example: '
+                  '"http://127.0.0.1:3141/root/dev/+simple/,'
+                  'https://pypi.org/simple"'),
+        )
+        general_group.add_argument(
+            '--devpi-simple-url',
+            type=str,
+            default=None,
+            help=('Devpi simple URL to use when --dev is set. '
+                  'Default: http://127.0.0.1:3141/root/dev/+simple/'),
+        )
         metadata_group = parser.add_argument_group('Notebook Metadata Overrides',
                                                    METADATA_GROUP_DESC)
@@ Expand All / @@ -84,6 +103,13 @@ def main(): @@
                                     help='The access mode for the created volumes')
         args = parser.parse_args()
+        if args.pip_index_urls:
+            os.environ["KALE_PIP_INDEX_URLS"] = args.pip_index_urls
+        elif args.dev:
+            os.environ["KALE_DEV_MODE"] = "1"
+            if args.devpi_simple_url:
+                os.environ["KALE_DEVPI_SIMPLE_URL"] = args.devpi_simple_url
         # get the notebook metadata args group
         mt_overrides_group = next(
             filter(lambda x: x.title == 'Notebook Metadata Overrides',
@@ Expand Down @@

Add GitHub workflows for automated tag-based releases #498

Are you sure you want to change the base?

Add GitHub workflows for automated tag-based releases #498

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

jesuino Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

StefanoFioravanzo Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jesuino Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

StefanoFioravanzo Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

jesuino Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

StefanoFioravanzo Nov 20, 2025 •

edited

Loading