ArtifactHub.io Integration #323

joeybrown-sf · 2025-01-16T15:28:33Z

buildpack-bot · 2025-01-16T15:28:56Z

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

text/0000-artifact-hub-integration.md

jkutner · 2025-01-30T21:58:14Z

text/0000-artifact-hub-integration.md

+
+[unresolved-questions]: #unresolved-questions
+
+Is there a Buildpacks team that owns the Registry Service now? Are they willing to take on the burden of owning this? If


Yea, it's owned by Platform Team

@jkutner I am curious about how the Tekton integration ended up published on artifacthub.io

Is this something Javier was doing?

jkutner · 2025-01-30T21:59:21Z

text/0000-artifact-hub-integration.md

+
+[drawbacks]: #drawbacks
+
+We would need to consider changes to the ArtifactHub type schemas when modifying our spec. It is possible that the


hmm, are you sure we need to include them? or could they just be a supplement. or at least a spec extension?

what does this schema look like?

Upon further review, I do not think we would need to consider changes to the Artifact Hub schema.

Here is the Artifact Hub schema. If it does not suffice, there are also custom annotations we can provide. Here is an example of custom annotations.

Co-authored-by: Joe Kutner <jpkutner@gmail.com> Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf · 2025-04-29T20:50:25Z

I have made substantial changes to this RFC after discussing with an ArtifactHub.io maintainer and doing some more research. I am now proposing to decommission registry.buildpacks.io.

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

jkutner · 2025-05-02T13:39:40Z

text/0000-artifact-hub-integration.md

+
+This should happen in phases so that buildpack authors and `pack` users have minimal disruption to established
+workflows. CNB is committed to backwards compatibility, but there are sometimes good reasons to break this backwards
+compatibility. This RFC proposes a slow phase-out and eventual decommission of the buildpacks registry. More details


I don't think that shutting down the old buildpacks registry necessarily means breaking backwards compat. If ArtifactHub matches the features we provide today, then changing our "backend" means we're still backwards compat.

It would be a bit different because the publishing process for the CNB registry is an explicit operation that happens against the cnb index repository. Future publishing would be implicit in that publishers modify a git repo they own and then the Artifact Hub scanner picks it up.

jkutner · 2025-05-02T13:40:49Z

text/0000-artifact-hub-integration.md

+eventually deprecates the buildpack registry.
+
+This RFC proposes that in 12 months, the buildpacks registry index becomes read-only. Buildpack publishers
+will have 12 months to change their publish process so that buildpacks are no longer published to the registry index,


hmm, will they actually need to change their publish process, or could our tooling (i.e. pack) paper over the changes so that they go forward not worrying about the backend change?

I suppose they would have to change their expectations about the website and slack notifications.

The publishing process would be different.

Instead of using pack to explicitly update the cnb registry, publishers would add a new directory in their Artifact Hub registry source.

For instance, today if publisher wanted to publish verion 1.30.1 of their buildpack, they would do the following:

Push the image to their registry

Update the CNB registry index (manually or via pack)

CNB registry is updated when the indexer eventually scans the registry index.

Meanwhile, if this RFC is adopted, publishers would do the following:

Push the image to their registry

Create a new directory in their Artifact Hub registry source (a git repo they have configured in the Artifact Hub admin console).

Artifact Hub is updated when it's scanner checks the Artifact Hub registry source.

CNB registry is updated when the indexer eventually scans Artifact Hub.

ok, so we're saying existing users of Buildpack Registry will need to take a one-time action (step 2)?

yes that is correct. They will need to claim ownership of their artifacts in the ArtifactHub.io management console.

Do you have any idea about how long the delay is for steps 3 & 4? Just wondering if we're talking a couple of minutes or a couple of hours or longer?

text/0000-artifact-hub-integration.md

jkutner · 2025-05-02T13:48:26Z

text/0000-artifact-hub-integration.md

+
+This is a disruptive change for buildpack publishers. The ownership and publishing model will be different.
+
+This is a destructive change that will affect `pack`. Any versions of `pack` that depend on the buildpack registry will


this is the biggest risk IMO. we should think about how we might get ahead of this (like releasing something in pack ASAP that helps future proof)

I don't know what the API for artifact hub is like or if this is in any way feasible, but what about running a proxy for requests from old versions of pack? If it could proxy and transparently map those requests to the new API, then old versions of pack would in theory keep working.

Not sure if there are any other platform implementations that integrated with the buildpacks registry, but that would be a way to keep other non-pack implementations working too.

That's a really good thought @dmikusa I will look into this.

jkutner · 2025-05-02T13:50:50Z

My biggest bit of feedback is that I think we should tighten up the timeline for cutting over to the ArtifactHub. But I'm less certain about the shutdown of the old registry (either 12 or 18 months)

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

dmikusa · 2025-05-21T11:56:00Z

text/0000-artifact-hub-integration.md

+This is a destructive change that will affect `pack`. Any versions of `pack` that depend on the buildpack registry will
+break as the various requirements of this RFC are implemented.
+
+ArtifactHub.io will be on the hot path for some `pack` operations.


What is the uptime like for this service? Does the CNCF have any guarantees about service availability?

What happens if there is an issue with the service? Do we have someone to page? Or a way to open a ticket and get things addressed?

Great questions. I will try to get some answers.

There is not a published uptime report, nor are there guarantees about service availability.

There is a #artifact-hub channel in the CNCF Slack and github issues that can be used to communicate with maintainers.

Do note that Artifact Hub does not host the packages.

dmikusa · 2025-05-21T11:59:59Z

text/0000-artifact-hub-integration.md

+
+ArtifactHub.io will be on the hot path for UI discoverability.
+
+If ArtifactHub.io goes away in the future, CNB will have to adapt. CNB discoverability and tooling could be left in a


Does the CNCF have a health report for ArtifactHub? Does it have funding guaranteed for a period of time? Are there other projects from the CNCF where ArtifactHub is on the critical path?

Maybe we want to consider adding a layer of indirection, that we control, so that pack isn't talking to ArtifactHub directly. That way if it goes away, pack's talking to our layer of indirection, and we can swap out a backend easier.

There is a CLO report here here that is at 99%. I will note, it's in the top 7 of the projects listed here.

Artifacthub is the primary distribution mechanism for helm charts

edmorley

Thank you for putting this together! I'm very excited about the prospect of us having fewer services/components to maintain :-)

edmorley · 2025-05-27T12:24:29Z

text/0000-artifact-hub-integration.md

+After month 18, the registry service will be terminated. At this point, ArtifactHub.io will be the official source for
+buildpacks.
+
+#### Processes - CNB Registry is terminated


Just to clarify, by "CNB Registry is terminated" do you mean everything, including the indexer and API/UI components at https://registry.buildpacks.io/ ? Or just the publishing/index management parts? (I can't tell whether the dashboard will still exist and use ArtifactHub as a data source, or ...?)

I am proposing terminating the entire instance.

edmorley · 2025-05-27T12:27:18Z

text/0000-artifact-hub-integration.md

+> We need to ensure the Artifact
+> Hub [package metadata](https://github.com/artifacthub/hub/blob/master/docs/metadata/artifacthub-pkg.yml) suffices
+> for Component Buildpacks and Builders. If not, we need to ensure we create
+> adequate [custom annotations](https://artifacthub.io/docs/topics/annotations/keptn/).


Does this spec need to cover what custom annotations we should use for the buildpack and builder types on ArtifactHub.io?

eg Should Buildpack API version be a field? Supported Targets? Distro flavour? Or for builders, things like run/build image, Lifecycle version?

If the entirety of https://registry.buildpacks.io/ is going away (including the indexer/API/database parts), then I presume we'll need to make sure that the ArtifactHub types support all fields we need for lookup and searching (including fields we currently exact from the Docker Image in the current indexer).

For example, one of the features we're missing in the current CNB registry is the ability to eg filter by CNBs that support the latest Buildpack API versions. To support this use-case we'd presumably need Buildpack API as a custom annotation etc.

Lastly, if we have custom annotations that contain mirror metadata from the buildpack/builder for convenience/searchability - do we need some way to validate these to make sure they are correct? ie: To prevent a buildpack declaring support for one Buildpack API version or target in its buildpack.toml but a different value in its ArtifactHub entry? If we were still using https://github.com/buildpacks/github-actions I'd suggest that as the place to do so - but I guess that action will be sunset too since ArtifactHub will pull directly from people's GitHub repos? Do we need a lint action or similar? Maybe the ArtifactHub maintainers have some suggestions or existing tooling to help with this?

edmorley · 2025-05-27T12:28:56Z

text/0000-artifact-hub-integration.md

+is a very important tool, but there is room for improvement. There are many features that ArtifactHub.io provides that
+the buildpack registry does not provide. Some features include rich descriptions, usage information, activity,
+dependency information, email subscriptions to packages, webhooks, related packages, and, furthermore, it is a service
+that the buildpacks team does not need to maintain.


Another motivation we could list for sunsetting the old CNB registry is that its architecture has a number of issues, eg the constant rescanning of images resulting in hitting Docker Hub rate limits, xref:
https://cloud-native.slack.com/archives/C032YE21V1T/p1746521791976579

I will add this.

edmorley · 2025-05-27T12:34:05Z

text/0000-artifact-hub-integration.md

+    end
+```
+
+### Requirement 7 - Terminate the buildpacks registry service


Do we need to mention the need to update https://github.com/buildpacks/github-actions with deprecation messages and eventually sunsetting its publishing functionality etc?

That is a good idea!

edmorley · 2025-05-27T12:38:49Z

text/0000-artifact-hub-integration.md

+_The goal here is to create a process that will register all the current (and new) buildpacks from the registry index to
+ArtifactHub.io._


I wonder if we should intentionally skip mirroring some CNBs, such as those that are stale/using EOL Buildpack APIs or were clearly only for testing/demo purposes? I suspect we have quite a lot of inactive/unusable CNBs in the current registry - it seems like now might be a good time to drop some of that clutter?

For example, anything older than Buildpack API 0.7 won't work with recent lifecycle versions:
https://github.com/buildpacks/lifecycle?tab=readme-ov-file#supported-apis

(People would still be free to publish any API 0.6 and older CNBs themselves, but we could skip doing it for them... I kinda prefer the idea of not pre-emptively adding a bunch of stuff that will likely never be used, and instead waiting to hear for a need for it)

Also, I wonder if we should first ask the big publishers (eg Heroku, Paketo, GCP, ...) if they want to publish their CNBs first, before we mass reserve all of their namespaces and make them claim ownership back?

This is a good point. I will add this.

joeybrown-sf added 2 commits January 16, 2025 09:29

start in on artifact hub

6c1d17f

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

Finish up artifact hub RFC

512ee84

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf force-pushed the artifact-hub branch from d2683a2 to 512ee84 Compare January 16, 2025 15:29

jabrown85 added type/rfc status/needs-steward labels Jan 16, 2025

add note about our own ArtifactHub instance

96b7c04

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

jkutner self-assigned this Jan 20, 2025

jkutner added roadmap/registry status/steward-assigned and removed status/needs-steward labels Jan 20, 2025

jkutner reviewed Jan 30, 2025

View reviewed changes

text/0000-artifact-hub-integration.md Outdated Show resolved Hide resolved

jkutner reviewed Jan 30, 2025

View reviewed changes

text/0000-artifact-hub-integration.md Outdated Show resolved Hide resolved

jkutner reviewed Jan 30, 2025

View reviewed changes

Update text/0000-artifact-hub-integration.md

a3e6517

Co-authored-by: Joe Kutner <jpkutner@gmail.com> Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf mentioned this pull request Apr 10, 2025

Support Cloud Native Buildpack Artifacts artifacthub/hub#4352

Open

joeybrown-sf added 3 commits April 29, 2025 15:51

update based on feedback from cnb maintainers and artifacthub maintainer

6ac44a1

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

small formatting/text changes

4b12726

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

add spec changes

534e818

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf force-pushed the artifact-hub branch from bfd41d4 to 534e818 Compare April 29, 2025 20:52

add reference to platform team

012cb85

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf requested review from jjbustamante and jkutner April 29, 2025 21:11

joeybrown-sf added 4 commits April 29, 2025 16:17

small wording change

dc140b6

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

add explicit drawback for older pack versions

643383c

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

provide more context to the summary

160280a

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

added link to artifacthub.io api openapi docs

4528948

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

jkutner reviewed May 2, 2025

View reviewed changes

text/0000-artifact-hub-integration.md Outdated Show resolved Hide resolved

jkutner reviewed May 2, 2025

View reviewed changes

text/0000-artifact-hub-integration.md Show resolved Hide resolved

jkutner reviewed May 2, 2025

View reviewed changes

text/0000-artifact-hub-integration.md Outdated Show resolved Hide resolved

jkutner reviewed May 2, 2025

View reviewed changes

update based on feedback

dd0eea9

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf requested a review from jkutner May 15, 2025 15:43

added diagrams

5b8dfed

Signed-off-by: Joey Brown <brown.joseph@salesforce.com>

joeybrown-sf force-pushed the artifact-hub branch from ef063e7 to 5b8dfed Compare May 20, 2025 20:55

dmikusa reviewed May 21, 2025

View reviewed changes

edmorley reviewed May 27, 2025

View reviewed changes


		[unresolved-questions]: #unresolved-questions

		Is there a Buildpacks team that owns the Registry Service now? Are they willing to take on the burden of owning this? If


		[drawbacks]: #drawbacks

		We would need to consider changes to the ArtifactHub type schemas when modifying our spec. It is possible that the


		This is a disruptive change for buildpack publishers. The ownership and publishing model will be different.

		This is a destructive change that will affect `pack`. Any versions of `pack` that depend on the buildpack registry will


		ArtifactHub.io will be on the hot path for UI discoverability.

		If ArtifactHub.io goes away in the future, CNB will have to adapt. CNB discoverability and tooling could be left in a

		_The goal here is to create a process that will register all the current (and new) buildpacks from the registry index to
		ArtifactHub.io._

ArtifactHub.io Integration #323

Are you sure you want to change the base?

ArtifactHub.io Integration #323

Uh oh!

Conversation

joeybrown-sf commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

buildpack-bot commented Jan 16, 2025

Issues

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joeybrown-sf commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkutner commented May 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edmorley left a comment

Choose a reason for hiding this comment

Uh oh!

edmorley May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edmorley May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edmorley May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edmorley May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

joeybrown-sf commented Jan 16, 2025 •

edited

Loading

edmorley May 27, 2025 •

edited

Loading

edmorley May 27, 2025 •

edited

Loading

edmorley May 27, 2025 •

edited

Loading

edmorley May 27, 2025 •

edited

Loading