Add GitHub workflows for automated tag-based releases#498
Add GitHub workflows for automated tag-based releases#498StefanoFioravanzo wants to merge 13 commits intomainfrom
Conversation
* detect the installed Kale version once and reuse it during DSL generation * centralize pip index selection logic with override/dev-mode precedence and tests * refresh generated pipeline fixtures to drop hard-coded dev builds and test PyPI * clean up packaging metadata, tweak devpi publishing, and document contributor workflow in CONTRIBUTE.md
Signed-off-by: Stefano Fioravanzo <stefano.fioravanzo@gmail.com>
|
@ederign @adrielparedes please see this PR as an initial proposal for an automated release workflow |
jesuino
left a comment
jesuino
left a comment
There was a problem hiding this comment.
Hello @StefanoFioravanzo Thank you so much for working on this.
I tried to run in dev mode, but I faced problems that I clarified in my comment.
Another thing I noticed is that `.venv/' is not in the gitignore. I was using conda and didn't notice it.
| - The labextension has independent tooling (`jlpm lint`, `jlpm prettier`). Run | ||
| those only if you modify the frontend package. | ||
|
|
||
| ## 8. Typical development checklist |
There was a problem hiding this comment.
I tried to follow these steps quite a few times and I noticed that the pipeline generated by Kale will still refer to localhost(127.0.0.1) devpi. But it fails due the fact that KFP is adding the pip index-url automatically to trusted host (see this issue).
In any case, if we fix that, we would still have to figure out how to fix the problem with an accessible devpi for KFP - in this case I think that the best solution would be running devpi on a pod in kubeflow namespace. (great thing that you made INDEX_URL a parameter!)
Does this make sense?
There was a problem hiding this comment.
I feel like there are two options:
- for super simple local dev setups using a minimal k8s setup for KFP (e.g. via
kind), exposing somehow the local devpi to the local k8s - indeed running devpi inside the cluster for testing scenario involving clusters running remotely or other use cases more difficult to manage.
We should probably discuss this more in a separate issue, since this is more about the local dev experience rather than publishing a new release. The initial PR was clearly incomplete and cannot cover all use cases.
There was a problem hiding this comment.
It makes sense to me! Let's discuss somewhere else
| @@ -0,0 +1,120 @@ | |||
| name: Publish release artifacts | |||
There was a problem hiding this comment.
Could we also have a daily publish for Kale backend at least? This way we could use the latest published dev version without having to build the backend locally for developing for the frontend!
There was a problem hiding this comment.
In principle I agree that having a nightly version would simplify fronted dev. Not sure I have seen projects doing this directly to PyPI. After a quick look online it seems to me that PyPI might cap due to size limits at some point, so nightly might not be recommended. Some folks do this using test.pipy.org (https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152)
We could think of publishing a nightly there, although there doesn't seems to be an automated garbage collection mechanism as well, not sure if at some point we will need to manually remove old versions.
Worth looking into what other projects are doing!
|
@StefanoFioravanzo is this PR ready to move from draft to review? |
|
Hey @hmtosi there we a few things that @jesuino pointed out that I wanted to look at. Also, before moving forward with this I wanted to make sure that our workflow for testing a local version against a remote k8s cluster is flawless. I had started working on a proposal early last week but I didn't find the time to complete it :/ |
3 participants
Warning
This PR is built on top of #491. Disregard all commits but the last two ones. The only changes relevant to this PR are in the
.github/workflowsfolderThis PR is dependent on #491 because it introduces a new pyproject.toml for the backend that changes the way the backend builds.
Summary
v*tags with generated notes so tagging alone publishes the release page.some details
.github/workflows/create-release.yml:1tag-driven release job that callssoftprops/action-gh-release@v2with generated notes..github/workflows/publish-release.yml:5listens to both real releases andworkflow_dispatch, wiring inputs for the target ref and release/test mode while still skipping prereleases..github/workflows/publish-release.yml:41builds the backend once and conditionally uploads to PyPI or TestPyPI (TEST_PYPI_API_TOKEN) depending on the mode..github/workflows/publish-release.yml:78builds the labextension in the same run and either publishes to npm with the prod token or performs a dry-run plusnpm packusing the staging token.