Releases: opst/knitfab
v1.6.1
- Date: 2025-03-03
v1.6.1 is minor bugfix release.
Bugfix
Web-Console: Lineage Graph: Run-to-Log edges are visible
At v1.6.0, Edges in Lineage Graph was not visible dependeing on a case.
We have fixed it.
Upgrade Path
Knitfab System
Download the installer, and run
installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
v1.6.1-beta
v1.6.1-beta
- Date: 2025-02-28
v1.6.1 is minor bugfix release.
Bugfix
Web-Console: Lineage Graph: Run-to-Log edges are visible
At v1.6.0, Edges in Lineage Graph was not visible dependeing on a case.
We have fixed it.
How to Try
Knitfab System
Download the installer from branch develop/v1.6.1, and run
BRANCH=develop/v1.6.1 CHART_VERSION=v1.6.1-beta installer.sh --install
in the directory where you have installed Knitfab.
As mentioned above, you may need to renew & replace TLS certificates to access the Web Console.
CLI knit
Download from assets of this release.
What's Changed
- Update version string -> v1.6.1-beta by @takaoka-youta-opst in #270
- Update version string -> v1.6.1 by @takaoka-youta-opst in #271
- rename files by @takaoka-youta-opst in #272
- Make the missing edge visible. by @takaoka-youta-opst in #273
- Feature/#267/web console version number clickable by @takaoka-youta-opst in #274
- Feature/#275/release v1.6.1 beta by @takaoka-youta-opst in #277
Full Changelog: v1.6.0...v1.6.1-beta
Contributors
Assets 9
v1.6.0
- Date: 2025-02-26
v1.6.0 introduces a new feature: Web Console!
Important Changes
Web Console
Knitfab frontend server, also known as WebAPI server, publishes Knitfab Web Console on its root (https://<DOMAIN or HOST>:<PORT>/).
You can access Web Console with your web browser (we tested with Google Chrome).
The Web Console has:
- "Data" tab: list and filter Data in your Knitfab
- "Plan" tab: list and filter Plans in your Knitfab
- "Run" tab: list and filter Runs in your Knitfab
Each item in the lists -- Data, Plan and Run --, has a button that shows a graph.
For Data and Run, it opens Lineage Graph (similar to knit data lineage) related to the Data or Run.
For Plan, it opens Plan Graph (similar to knit plan graph) related to the Plan.
Note
For Knitfab admins who have installed Knitfab with installer-generated TLS certificates:
For your users to access the Web Console, you need to renew & replace TLS certificates.
Follow the next section.
And, these TLS certificates are not authorized with any publicly trusted CA.
Your users may see a browser warning page the first time they access it.
Installer: Renew TLS certificates
With this release, our Installer now supports renewing TLS certificates.
Upgrade the Installer with
curl -LO https://raw.githubusercontent.com/opst/knitfab/refs/heads/develop/v1.6.0/installer/installer.sh
chmod +x ./installer.shin the directory where you have run installer Knitfab.
To renew and update certifications, follow admin-guide.
Bug Fix on Helm Chart: External RDB mode works.
knitfab-install-settings/values/knit-db-postgres.yaml has had an external parameter, but it did not work.
With this release, the parameter is effective. You can use PostgreSQL server that you provisioned for Knitfab.
This may help deploy Knitfab on cloud platforms and use managed RDBs.
For details of configurations, see the file generated by ./installer.sh --prepare (before that, backup your install configurations).
Other Changes
- Golang used to build is upgraded to 1.24.0.
- Kubernetes Client library "k8s.io/client-go" is upgraded to v0.32.0.
- This release is tested with Kubernetes v1.32.
- This may affect users who use al older version of Kubernetes.
License
Knitfab v1.6.0 is released under BSL 1.1, as written in the LICENSE file.
CHANGE DATE for v1.6.x is 2029-2-26.
Previous releases, v1.5.x or brefore, are not changed in their CHANGE DATE.
Upgrade Path
This release has breaking change, so upgrade both Knitfab system and CLI.
Knitfab System
Download the installer, and run
installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- Feature/#246/web list dashboard by @takaoka-youta-opst in #248
- make knitd shutting down on certs updated by @takaoka-youta-opst in #250
- Add a new install mode: "external database" by @takaoka-youta-opst in #253
- update version string: v1.6.0-beta by @takaoka-youta-opst in #254
- Feature/#245/lineage graph viewer by @takaoka-youta-opst in #255
- Feature/#249/upgrade go by @takaoka-youta-opst in #257
- update dev-cluster: k8s -> v1.32 by @takaoka-youta-opst in #259
- Fix/#258/show all update at by @takaoka-youta-opst in #260
- Feature/#261/prepare to release v1.6.0 beta by @takaoka-youta-opst in #262
- Feature/#263/release v1.6.0 by @takaoka-youta-opst in #264
- Develop/v1.6.0 by @takaoka-youta-opst in #265
Full Changelog: v1.5.1...v1.6.0
Contributors
Assets 9
v1.6.0-beta
- Date: 2025-02-21
v1.6.0-beta introduces a new feature: Web Console!
Important Changes
Web Console
Knitfab frontend server, also known as WebAPI server, publishes Knitfab Web Console on its root (https://<DOMAIN or HOST>:<PORT>/).
You can access Web Console with your web browser (we tested with Google Chrome).
The Web Console has:
- "Data" tab: list and filter Data in your Knitfab
- "Plan" tab: list and filter Plans in your Knitfab
- "Run" tab: list and filter Runs in your Knitfab
Each item in the lists -- Data, Plan and Run --, has a button that shows a graph.
For Data and Run, it opens Lineage Graph (similar to knit data lineage) related to the Data or Run.
For Plan, it opens Plan Graph (similar to knit plan graph) related to the Plan.
Note
For Knitfab admins who have installed Knitfab with installer-generated TLS certificates:
For your users to access the Web Console, you need to renew & replace TLS certificates.
Follow the next section.
And, these TLS certificates are not authorized with any publicly trusted CA.
Your users may see a browser warning page the first time they access it.
Installer: Renew TLS certificates
With this release, our Installer now supports renewing TLS certificates.
Upgrade the Installer with
curl -LO https://raw.githubusercontent.com/opst/knitfab/refs/heads/develop/v1.6.0/installer/installer.sh
chmod +x ./installer.shin the directory where you have run installer Knitfab.
To renew and update certifications, follow admin-guide.
Bug Fix on Helm Chart: External RDB mode works.
knitfab-install-settings/values/knit-db-postgres.yaml has had an external parameter, but it did not work.
With this release, the parameter is effective. You can use PostgreSQL server that you provisioned for Knitfab.
This may help deploy Knitfab on cloud platforms and use managed RDBs.
For details of configurations, see the file generated by ./installer.sh --prepare (before that, backup your install configurations).
Other Changes
- Golang used to build is upgraded to 1.24.0.
- Kubernetes Client library "k8s.io/client-go" is upgraded to v0.32.0.
- This release is tested with Kubernetes v1.32.
- This may affect users who use al older version of Kubernetes.
How to Try
Knitfab System
Download the installer from branch develop/v1.6.0, and run
BRANCH=develop/v1.6.0 CHART_VERSION=v1.6.0-beta installer.sh --install
in the directory where you have installed Knitfab.
As mentioned above, you may need to renew & replace TLS certificates to access the Web Console.
CLI knit
Download from assets of this release.
What's Changed
- Feature/#246/web list dashboard by @takaoka-youta-opst in #248
- make knitd shutting down on certs updated by @takaoka-youta-opst in #250
- Add a new install mode: "external database" by @takaoka-youta-opst in #253
- update version string: v1.6.0-beta by @takaoka-youta-opst in #254
- Feature/#245/lineage graph viewer by @takaoka-youta-opst in #255
- Feature/#249/upgrade go by @takaoka-youta-opst in #257
- update dev-cluster: k8s -> v1.32 by @takaoka-youta-opst in #259
- Fix/#258/show all update at by @takaoka-youta-opst in #260
- Feature/#261/prepare to release v1.6.0 beta by @takaoka-youta-opst in #262
Full Changelog: v1.5.1...v1.6.0-beta
Contributors
Assets 9
v1.5.1
- Date: 2024-12-23
The v1.5.1 introduces the following updates:
- Security Update
- Bug Fixes
Important Changes
Security Update
Knitfab depends on echo,
and echo releases v4.13.3 addressing https://pkg.go.dev/vuln/GO-2024-3321 / https://www.cve.org/CVERecord?id=CVE-2024-45337 and https://pkg.go.dev/vuln/GO-2024-3333 / https://www.cve.org/CVERecord?id=CVE-2024-45338.
We upgraded echo.
Bug Fixes
knit plan template contained typo
Plan Definition file template generated from knit plan template contained typo.
The key resources was mistakenly written as resouce (missing the r).
This issue has been fixed.
knit plan find: There are cases in which --image flag did not work
The --image flag of the knit plan find command, which filters results based on the image of Plans, did not work correctly in some cases.
When an image with a port (e.g., example.com:5000/image:1.0) was provided, both the command and the Knitfab server (knitd) incorrectly parsed the image name as example.com with the tag 5000/image:1.0. As a result, no plans matching the specified image could be found.
This issue has been fixed.
Note: knit plan apply was unaffected, and registered Plans retained the correct image names and tags.
Upgrade Path
This release has breaking change, so upgrade both Knitfab system and CLI.
Knitfab System
Download the installer, and run
installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- set version: v1.5.1-beta by @takaoka-youta-opst in #228
- update echo => v4.13.2 by @takaoka-youta-opst in #229
- fix parsing algorithms for container image name by @takaoka-youta-opst in #230
- fix typo in
knit plan templateby @takaoka-youta-opst in #231 - Update ansible by @takaoka-youta-opst in #233
- Reconstruct README: emphasize documents by @takaoka-youta-opst in #236
- upgrade echo to approach CVE-2024-45338 by @takaoka-youta-opst in #238
- Feature/#234/preapre release v1.5.1 by @takaoka-youta-opst in #239
- Update release note: refer #237 by @takaoka-youta-opst in #241
- prepare to release v1.5.1 by @takaoka-youta-opst in #243
- Release v1.5.1 by @takaoka-youta-opst in #244
Full Changelog: v1.5.0...v1.5.1
Contributors
Assets 9
v1.5.1-beta
- Date: 2024-12-20
The v1.5.1-beta introduces the following updates:
- Security Update
- Bug Fixes
Important Changes
Security Update
Knitfab depends on echo,
and echo releases v4.13.3 addressing https://pkg.go.dev/vuln/GO-2024-3321 / https://www.cve.org/CVERecord?id=CVE-2024-45337 and https://pkg.go.dev/vuln/GO-2024-3333 / https://www.cve.org/CVERecord?id=CVE-2024-45338.
We upgraded echo.
Bug Fixes
knit plan template contained typo
Plan Definition file template generated from knit plan template contained typo.
The key resources was mistakenly written as resouce (missing the r).
This issue has been fixed.
knit plan find: There are cases in which --image flag did not work
The --image flag of the knit plan find command, which filters results based on the image of Plans, did not work correctly in some cases.
When an image with a port (e.g., example.com:5000/image:1.0) was provided, both the command and the Knitfab server (knitd) incorrectly parsed the image name as example.com with the tag 5000/image:1.0. As a result, no plans matching the specified image could be found.
This issue has been fixed.
Note: knit plan apply was unaffected, and registered Plans retained the correct image names and tags.
How to try
This release containing breaking changes, so please be sure to update both the Knitfab system AND CLI to try.
Knitfab System
Download the installer from branch develop/v1.5.0, and run
BRANCH=develop/v1.5.1 CHART_VERSION=v1.5.1-beta installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- set version: v1.5.1-beta by @takaoka-youta-opst in #228
- update echo => v4.13.2 by @takaoka-youta-opst in #229
- fix parsing algorithms for container image name by @takaoka-youta-opst in #230
- fix typo in
knit plan templateby @takaoka-youta-opst in #231 - Update ansible by @takaoka-youta-opst in #233
- Reconstruct README: emphasize documents by @takaoka-youta-opst in #236
- upgrade echo to approach CVE-2024-45338 by @takaoka-youta-opst in #238
- Feature/#234/preapre release v1.5.1 by @takaoka-youta-opst in #239
Full Changelog: v1.5.0...v1.5.1-beta
Contributors
Assets 9
v1.5.0
- Date: 2024-11-20
The v1.5.0 release includes
knit plan showandknit plan finddisplay direct upstreams/downstreams of each Plan- New command
knit plan graph - Fix ambiguity of Data's upstream (breaking change)
- Fix case inconsistency in JSON (breaking change)
Important Changes
knit plan showand knit plan find display direct upstreams/downstreams of each Plan
These commands show upstream/downstream neighboring Plans of a Plan. As shown below:
{
"planId": ...
...
"inputs": [
{
"path": "/in/1",
"tags": [ ... ],
"upstreams": [ // NEW!
{
"plan": {
"planId": "...",
"image": "...",
"entrypoint": [ ... ],
"args": [ ... ],
"annotations": [ ... ]
},
"mountpoint": { // this upstream is a normal output.
"path": "/out/1",
"tags": [ ... ]
}
},
{
"plan": {
"planId": "...",
"image": "...",
"entrypoint": [ ... ],
"args": [ ... ],
"annotations": [ ... ]
},
"log": { // this upstream is a log.
"tags": [ ... ]
}
},
]
}
],
"outputs": [
{
"path": "/out/1",
"tags": [ ... ],
"downstreams": [ // NEW!
{
"plan": {
"planId": "...",
"image": "...",
"entrypoint": [ ... ],
"args": [ ... ],
"annotations": [ ... ]
},
"mountpoint": {
"path": "/in/2",
"tags": [ ... ]
}
},
]
}
],
...
}Upstreams of a Plan Input are Plan Outputs or a Logs with matching Tags. If an Output (or Log) which has all Tags of the Input of another Plan, the Output is considered as Upstream of the Input. In other words, when Output or Log of a Plan A generates Data which can be assigned to an Input of Plan B, the Output/Log is upstream of the Input.
knit plan graph
The new command knit plan graph generates Plan Graph, which is an overview of your Plans, in dot format.
knit plan graph traverse Plans upstream and/or downstream Plans recursively and visualize the "pipeline" made by Plans.
See knit plan graph --help for more details.
Fix ambiguity of Data's upstream (breaking change)
Before this change, "upstream" of Data had not distinguished normal outputs and logs.
If the upstream of Data is a log, knit data find tells you the upstream has "path" (as /log).
However, the expression is identical to a that of normal output with the path /log, so with only knit data find, we could not know the upstream of Data is whether output or log.
By now, knit data find tells explicitly the upstream of Data is output or is log.
To do that, breaking changes are introduced. In JSON format of Data, "path" and "tags" are moved to a new field "mountpoint" (for normal outputs) or "log" (for log) .
{
"knitId": "...",
"tags": [ ... ],
"upstream": {
"mountpoint": { // NEW!
"path": "/upload",
"tags": []
},
"run": {
"runId": " ... ",
"status": "done",
"updatedAt": "2024-11-18T04:25:32.076+00:00",
"plan": { ... }
}
},
"downstreams": [
{
"mountpoint": { // NEW!
"path": "/in/dataset",
"tags": [
"mode:training",
"project:first-knitfab",
"type:dataset"
]
},
"run": {
"runId": "b7ed ... ",
"status": "running",
"updatedAt": "2024-11-18T04:44:06.008+00:00",
"plan": { ... }
}
}
],
"nomination": [ ... ]
}{
"knitId": "de17825d-16f3-4a5d-b3cb-16ef95379c0c",
"tags": [
"knit#id:de17825d-16f3-4a5d-b3cb-16ef95379c0c",
"knit#timestamp:2024-11-18T05:12:32.075+00:00",
"project:first-knitfab",
"type:log"
],
"upstream": {
"log": { // NEW!
"tags": [
"project:first-knitfab",
"type:log"
]
},
"run": {
"runId": "b7ed106b-cb49-4671-8979-4e85e249f15c",
"status": "done",
"updatedAt": "2024-11-18T05:12:32.075+00:00",
"exit": {
"code": 0,
"message": ""
},
"plan": {
"planId": "5770077f-e7a2-4b0f-8e8e-4d73d4b14144",
"image": "localhost:30503/knitfab-first-train:v1.0",
"entrypoint": [
"python",
"-u",
"train.py"
],
"args": [
"--dataset",
"/in/dataset",
"--save-to",
"/out/model"
]
}
}
},
"downstreams": [],
"nomination": []
}Data with an "upstream" containing "mountpoint" represents a normal output, while one containing "log" represents a log.
Fix case inconsistency in JSON (breaking change)
Plan had log.Tag field. Differently than other fields, the first letter of the filed name is capitalized.
This inconcistency is not intended, so it has been fixed to log.tag.
{
"planId": " ... ",
"image": " ... ",
// ...
"log": {
"tags": [ // renamed from "Tags".
"project:first-knitfab",
"type:log"
],
"downstreams": []
},
// ...
}License
Knitfab v1.5.0 is released under BSL 1.1, as written in the LICENSE file.
CHANGE DATE for v1.5.x is 2028-11-20.
Previous releases, v1.4.x or brefore, are not changed in their CHANGE DATE.
Upgrade Path
This release has breaking change, so upgrade both Knitfab system and CLI.
Knitfab System
Download the installer, and run
installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- update VERSION => v1.5.0-beta by @takaoka-youta-opst in #206
knit plan template: memory footprint slim down by @takaoka-youta-opst in #207- remove unused codes by @takaoka-youta-opst in #208
- Feature/#201/sort packages by @takaoka-youta-opst in #209
- show upstream/downstream of input/output of Plan by @takaoka-youta-opst in #211
- Feature/#212/enrich upstream downstream by @takaoka-youta-opst in #213
- update Data response type by @takaoka-youta-opst in #215
- Feature/#24/workflow overview by @takaoka-youta-opst in #216
- fix test by @takaoka-youta-opst in #219
- Feature/#217/prepare v1.5.0 beta by @takaoka-youta-opst in #220
- Feature/#221/prepare release v1.5.0 by @takaoka-youta-opst in #222
- Reelase v1.5.0 by @takaoka-youta-opst in #223
Full Changelog: v1.4.0...v1.5.0
Contributors
Assets 9
v1.5.0-beta
- Date: 2024-11-18
The v1.5.0-beta release includes
knit plan showandknit plan finddisplay direct upstreams/downstreams of each Plan- New command
knit plan graph - Fix ambiguity of Data's upstream (breaking change)
- Fix case inconsistency in JSON (breaking change)
Important Changes
knit plan showand knit plan find display direct upstreams/downstreams of each Plan
These commands show upstream/downstream neighboring Plans of a Plan. As shown below:
{
"planId": ...
...
"inputs": [
{
"path": "/in/1",
"tags": [ ... ],
"upstreams": [ // NEW!
{
"plan": {
"planId": "...",
"image": "...",
"entrypoint": [ ... ],
"args": [ ... ],
"annotations": [ ... ]
},
"mountpoint": { // this upstream is a normal output.
"path": "/out/1",
"tags": [ ... ]
}
},
{
"plan": {
"planId": "...",
"image": "...",
"entrypoint": [ ... ],
"args": [ ... ],
"annotations": [ ... ]
},
"log": { // this upstream is a log.
"tags": [ ... ]
}
},
]
}
],
"outputs": [
{
"path": "/out/1",
"tags": [ ... ],
"downstreams": [ // NEW!
{
"plan": {
"planId": "...",
"image": "...",
"entrypoint": [ ... ],
"args": [ ... ],
"annotations": [ ... ]
},
"mountpoint": {
"path": "/in/2",
"tags": [ ... ]
}
},
]
}
],
...
}Upstreams of a Plan Input are Plan Outputs or a Logs with matching Tags. If an Output (or Log) which has all Tags of the Input of another Plan, the Output is considered as Upstream of the Input. In other words, when Output or Log of a Plan A generates Data which can be assigned to an Input of Plan B, the Output/Log is upstream of the Input.
knit plan graph
The new command knit plan graph generates Plan Graph, which is an overview of your Plans, in dot format.
knit plan graph traverse Plans upstream and/or downstream Plans recursively and visualize the "pipeline" made by Plans.
See knit plan graph --help for more details.
Fix ambiguity of Data's upstream (breaking change)
Before this change, "upstream" of Data had not distinguished normal outputs and logs.
If the upstream of Data is a log, knit data find tells you the upstream has "path" (as /log).
However, the expression is identical to a that of normal output with the path /log, so with only knit data find, we could not know the upstream of Data is whether output or log.
By now, knit data find tells explicitly the upstream of Data is output or is log.
To do that, breaking changes are introduced. In JSON format of Data, "path" and "tags" are moved to a new field "mountpoint" (for normal outputs) or "log" (for log) .
{
"knitId": "...",
"tags": [ ... ],
"upstream": {
"mountpoint": { // NEW!
"path": "/upload",
"tags": []
},
"run": {
"runId": " ... ",
"status": "done",
"updatedAt": "2024-11-18T04:25:32.076+00:00",
"plan": { ... }
}
},
"downstreams": [
{
"mountpoint": { // NEW!
"path": "/in/dataset",
"tags": [
"mode:training",
"project:first-knitfab",
"type:dataset"
]
},
"run": {
"runId": "b7ed ... ",
"status": "running",
"updatedAt": "2024-11-18T04:44:06.008+00:00",
"plan": { ... }
}
}
],
"nomination": [ ... ]
}{
"knitId": "de17825d-16f3-4a5d-b3cb-16ef95379c0c",
"tags": [
"knit#id:de17825d-16f3-4a5d-b3cb-16ef95379c0c",
"knit#timestamp:2024-11-18T05:12:32.075+00:00",
"project:first-knitfab",
"type:log"
],
"upstream": {
"log": { // NEW!
"tags": [
"project:first-knitfab",
"type:log"
]
},
"run": {
"runId": "b7ed106b-cb49-4671-8979-4e85e249f15c",
"status": "done",
"updatedAt": "2024-11-18T05:12:32.075+00:00",
"exit": {
"code": 0,
"message": ""
},
"plan": {
"planId": "5770077f-e7a2-4b0f-8e8e-4d73d4b14144",
"image": "localhost:30503/knitfab-first-train:v1.0",
"entrypoint": [
"python",
"-u",
"train.py"
],
"args": [
"--dataset",
"/in/dataset",
"--save-to",
"/out/model"
]
}
}
},
"downstreams": [],
"nomination": []
}Data with an "upstream" containing "mountpoint" represents a normal output, while one containing "log" represents a log.
Fix case inconsistency in JSON (breaking change)
Plan had log.Tag field. Differently than other fields, the first letter of the filed name is capitalized.
This inconcistency is not intended, so it has been fixed to log.tag.
{
"planId": " ... ",
"image": " ... ",
// ...
"log": {
"tags": [ // renamed from "Tags".
"project:first-knitfab",
"type:log"
],
"downstreams": []
},
// ...
}How to try
This release containing breaking changes, so please be sure to update both the Knitfab system AND CLI to try.
Knitfab System
Download the installer from branch develop/v1.5.0, and run
BRANCH=develop/v1.5.0 CHART_VERSION=v1.5.0-beta installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- update VERSION => v1.5.0-beta by @takaoka-youta-opst in #206
knit plan template: memory footprint slim down by @takaoka-youta-opst in #207- remove unused codes by @takaoka-youta-opst in #208
- Feature/#201/sort packages by @takaoka-youta-opst in #209
- show upstream/downstream of input/output of Plan by @takaoka-youta-opst in #211
- Feature/#212/enrich upstream downstream by @takaoka-youta-opst in #213
- update Data response type by @takaoka-youta-opst in #215
- Feature/#24/workflow overview by @takaoka-youta-opst in #216
- fix test by @takaoka-youta-opst in #219
- Feature/#217/prepare v1.5.0 beta by @takaoka-youta-opst in #220
Full Changelog: v1.4.0...v1.5.0-beta
Contributors
Assets 9
v1.4.0
- Date: 2024-10-25
The v1.4.0 release includes
- New fields on Plan
- Dynamic Environmental Variable Injection to Run via Lifecycle Hook
Important Changes
New Fields on Plan
We added new fields to Plan.
Annotation
Now, Plan can be "annotated".
image: "example.com/train:v1.0"
annotation:
- "key=value"
- "description=annotation can have arbitrary text content"
- "creator=takaoka.youta@opst.co.jp"
inputs:
- ...
outputs:
- ...
...annotation is key=value formatted metadata of Plans itself.
They are not Tags, so Knitfab does not consider annotations when making new Runs.
You can use annotations to record arbitrary text content, for example, "What is the Plan?" or "Who is creator of the Plan?".
Annotations are mutable. To update them, use the knit plan annotate command.
Entrypoint and Args
image: "example.com/train:v1.0"
entrypoint: ["python", "main.py"]
args: ["--data", "/in/1/data", "--output", "/out/1/model"]
inputs:
- path: /in/1/data
tags:
- "format:image/png"
- "mode:train"
- "type:dataset"
- "project:example"
outputs:
- path: /out/1/model
tags:
- "framework:pytorch"
- "type:model"
- "project:example"
...With this change, Plan can have entrypoint and args which override default behavior of the image.
entrypoint overrides ENTRYPOINT directive in Dockerfile, and args overrides CMD directive.
These are optional, and by default, image will run as defined in its Dockerfile.
Service Account
image: "example.com/train:v1.0"
inputs:
- ...
outputs:
- ...
service_account: service_account_name
...You can assign ServiceAccount of Kubernetes on Plans.
When a new Run based a Plan with service_account is created, the Pod of the Run is started with the ServiceAccount specified on the Plan.
Pods are started without ServiceAccount by default, which is sufficient for most Runs are self-contained within a Pod.
When your Run needs access to the Kubernetes API or cloud platforms managed services, this feature may be useful.
service_account is mutable. To update this, use knit plan serviceaccount command.
Dynamic Environmental Variable Injection to Run via Lifecycle Hook
Before this change, Knitfab ignores response bodies from Lifecycle Hooks.
With this change, Knitfab uses the response body of "Before starting" Lifecycle hook, if able.
If the response has Content-Type: application/json and contains the following structure,
Knitfab use knitfabExtension.env as environmental variables for the starting Pod.
{
// ...
"knitfabExtension": {
"env": { [key: string]: string }
}
// ...
}(the type notation is in TypeScript syntax)
If the response is not Content-Type: application/json or does not contain the structure,
it is ignored as it was.
You can write Lifecycle Hooks which determine envvars by the Run.
knit data tag: behavior change
new flag --remove-key
Now Tags of data can be removed by the key.
If you do below, all tags with key example-key are removed from Data identified with SOME-KNIT-ID.
knit data tag --remove-key example-key SOME-KNIT-ID
Unlike --remove, --remove-key does not consider the Value of Tag.
It is useful to change Tags with a very long note.
knit data --remove ... --add ...: remove and then add now
With this release, execution order of --add, --remove or --remove-key are changed.
Now, --removes and --remove-keys are effected first, and then --adds are effected.
The new behavior (remove first, then add) allows "update" Tags in atomic operation.
Example: update long description,
knit data tag --remove-key description --add "description:very long note of the Data" SOME-KNIT-ID
Example: update Tag referred by Plans in atomic operation.
knit data tag --remove version:latest --add version:3 SOME-KNIT-ID
License
Knitfab v1.4.0 is released under BSL 1.1, as written in the LICENSE file.
CHANGE DATE for v1.4.x is 2028-10-25.
Previous releases, v1.3.x or brefore, are not changed in their CHANGE DATE.
Upgrade Path
Knitfab System
Download the installer, and run
installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- backmerge: v1.3.1 -> v1.4.0 by @takaoka-youta-opst in #178
- set version v1.4.0-beta by @takaoka-youta-opst in #180
- Feature/#172/inject envvar for worker by @takaoka-youta-opst in #181
- Feature/#171/annotation on plan by @takaoka-youta-opst in #183
- Make Plan Annotation mutable by @takaoka-youta-opst in #185
- Plan: serviceaccount is mutable by @takaoka-youta-opst in #187
- Feature/#188/remove tag and annotation by key by @takaoka-youta-opst in #190
- Feature/#21/plan with args by @takaoka-youta-opst in #191
knit plan template:argsis defaulted by imagesCMDby @takaoka-youta-opst in #193- Feature/#195/prepare v1.4.0 beta by @takaoka-youta-opst in #196
- Feature/#199/prepare release v1.4.0 by @takaoka-youta-opst in #203
- release v1.4.0 by @takaoka-youta-opst in #204
Full Changelog: v1.3.1...v1.4.0
Contributors
Assets 9
v1.4.0-beta
- Date: 2024-10-23
The v1.4.0-beta release includes
- New fields on Plan
- Dynamic Environmental Variable Injection to Run via Lifecycle Hook
Important Changes
New Fields on Plan
We added new fields to Plan.
Annotation
Now, Plan can be "annotated".
image: "example.com/train:v1.0"
annotation:
- "key=value"
- "description=annotation can have arbitrary text content"
- "creator=takaoka.youta@opst.co.jp"
inputs:
- ...
outputs:
- ...
...annotation is key=value formatted metadata of Plans itself.
They are not Tags, so Knitfab does not consider annotations when making new Runs.
You can use annotations to record arbitrary text content, for example, "What is the Plan?" or "Who is creator of the Plan?".
Annotations are mutable. To update them, use the knit plan annotate command.
Entrypoint and Args
image: "example.com/train:v1.0"
entrypoint: ["python", "main.py"]
args: ["--data", "/in/1/data", "--output", "/out/1/model"]
inputs:
- path: /in/1/data
tags:
- "format:image/png"
- "mode:train"
- "type:dataset"
- "project:example"
outputs:
- path: /out/1/model
tags:
- "framework:pytorch"
- "type:model"
- "project:example"
...With this change, Plan can have entrypoint and args which override default behavior of the image.
entrypoint overrides ENTRYPOINT directive in Dockerfile, and args overrides CMD directive.
These are optional, and by default, image will run as defined in its Dockerfile.
Service Account
image: "example.com/train:v1.0"
inputs:
- ...
outputs:
- ...
service_account: service_account_name
...You can assign ServiceAccount of Kubernetes on Plans.
When a new Run based a Plan with service_account is created, the Pod of the Run is started with the ServiceAccount specified on the Plan.
Pods are started without ServiceAccount by default, which is sufficient for most Runs are self-contained within a Pod.
When your Run needs access to the Kubernetes API or cloud platforms managed services, this feature may be useful.
service_account is mutable. To update this, use knit plan serviceaccount command.
Dynamic Environmental Variable Injection to Run via Lifecycle Hook
Before this change, Knitfab ignores response bodies from Lifecycle Hooks.
With this change, Knitfab uses the response body of "Before starting" Lifecycle hook, if able.
If the response has Content-Type: application/json and contains the following structure,
Knitfab use knitfabExtension.env as environmental variables for the starting Pod.
{
// ...
"knitfabExtension": {
"env": { [key: string]: string }
}
// ...
}(the type notation is in TypeScript syntax)
If the response is not Content-Type: application/json or does not contain the structure,
it is ignored as it was.
You can write Lifecycle Hooks which determine envvars by the Run.
knit data tag: behavior change
new flag --remove-key
Now Tags of data can be removed by the key.
If you do below, all tags with key example-key are removed from Data identified with SOME-KNIT-ID.
knit data tag --remove-key example-key SOME-KNIT-ID
Unlike --remove, --remove-key does not consider the Value of Tag.
It is useful to change Tags with a very long note.
knit data --remove ... --add ...: remove and then add now
With this release, execution order of --add, --remove or --remove-key are changed.
Now, --removes and --remove-keys are effected first, and then --adds are effected.
The new behavior (remove first, then add) allows "update" Tags in atomic operation.
Example: update long description,
knit data tag --remove-key description --add "description:very long note of the Data" SOME-KNIT-ID
Example: update Tag referred by Plans in atomic operation.
knit data tag --remove version:latest --add version:3 SOME-KNIT-ID
How to try
Knitfab System
Download the installer from branch develop/v1.4.0, and run
BRANCH=develop/v1.4.0 CHART_VERSION=v1.4.0-beta installer.sh --install
in the directory where you have installed Knitfab.
CLI knit
Download from assets of this release.
What's Changed
- backmerge: v1.3.1 -> v1.4.0 by @takaoka-youta-opst in #178
- set version v1.4.0-beta by @takaoka-youta-opst in #180
- Feature/#172/inject envvar for worker by @takaoka-youta-opst in #181
- Feature/#171/annotation on plan by @takaoka-youta-opst in #183
- Make Plan Annotation mutable by @takaoka-youta-opst in #185
- Plan: serviceaccount is mutable by @takaoka-youta-opst in #187
- Feature/#188/remove tag and annotation by key by @takaoka-youta-opst in #190
- Feature/#21/plan with args by @takaoka-youta-opst in #191
knit plan template:argsis defaulted by imagesCMDby @takaoka-youta-opst in #193- Feature/#195/prepare v1.4.0 beta by @takaoka-youta-opst in #196
Full Changelog: v1.3.1...v1.4.0-beta