Propose following every Ubuntu LTS

[loewenstein-sap]

Summary

This is to propose to

1. Release Ubuntu Noble based stacks and builders asap
2. Follow every Ubuntu LTS release, i.e. do the same once 26.04, 28.04, etc are released

Use Cases

Having an up-to-date root file system.

Checklist

• I have viewed, signed, and submitted the Contributor License Agreement.
• I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
• I have added an integration test, if necessary.
• I have reviewed the styleguide for guidance on my code quality.
• I'm happy with the commit history on this PR (I have rebased/squashed as needed).

[mhdawson]

LGTM

[dmikusa]

I worry about this creating more work than we as a team can handle.

1. Do we have enough people willing to take on the work of producing stacks for every release, instead of every other release? That's effectively twice as much work. It would be nice to hear from @paketo-buildpacks/stacks-maintainers and @paketo-buildpacks/builders-maintainers on how they feel about this.

2. Do we know that this is a problem for users? Have we seen users requesting the latest stack when it's not available? It will also create twice as much work for users. That's twice as often that they need to review applications and make sure they run on the new stack. Do we have any user feedback that supports a need for more frequent stacks?

3.) I'd also be curious if you could elaborate on the security improvements of releasing twice as often. The way I read the RFC, there is an implication that our present release cycle may have security drawbacks. I think it would be good to more clearly state these drawbacks. My understanding is that the present stack release cycle we follow keeps us in the Ubuntu patch cycle, but that what we miss out on are low/minor patches (sometimes Ubuntu defers these to the next release) and new/updated libraries (not security related). This is what we have documented on the website anyway, https://paketo.io/docs/concepts/stacks/#when-are-paketo-stacks-updated.

---

In regards to 2.), I've not observed that personally. The question I usually get is do you have X stack, where X is some other non-Ubuntu distribution of Linux. Debian, UBI, Alpine, or some other base that promises to have marginally smaller images. I'd be curious to hear from users on what they want/expect/need in this regard.

[loewenstein-sap]

I worry about this creating more work than we as a team can handle.

I am a bit surprised to read this, because I recall this has been a topic in a past WG meeting (quite a while ago though) and I thought the only thing missing was materializing the decision from then in an RFC.

1. Do we have enough people willing to take on the work of producing stacks for every release, instead of every other release? That's effectively twice as much work. It would be nice to hear from @paketo-buildpacks/stacks-maintainers and @paketo-buildpacks/builders-maintainers on how they feel about this.
2. Do we know that this is a problem for users? Have we seen users requesting the latest stack when it's not available? It will also create twice as much work for users. That's twice as often that they need to review applications and make sure they run on the new stack. Do we have any user feedback that supports a need for more frequent stacks?

3.) I'd also be curious if you could elaborate on the security improvements of releasing twice as often. The way I read the RFC, there is an implication that our present release cycle may have security drawbacks. I think it would be good to more clearly state these drawbacks. My understanding is that the present stack release cycle we follow keeps us in the Ubuntu patch cycle, but that what we miss out on are low/minor patches (sometimes Ubuntu defers these to the next release) and new/updated libraries (not security related). This is what we have documented on the website anyway, https://paketo.io/docs/concepts/stacks/#when-are-paketo-stacks-updated.

In regards to 2.), I've not observed that personally. The question I usually get is do you have X stack, where X is some other non-Ubuntu distribution of Linux. Debian, UBI, Alpine, or some other base that promises to have marginally smaller images. I'd be curious to hear from users on what they want/expect/need in this regard.

Anyway, regarding your points.
Regarding 1. This is - unfortunately - a good point and I am curious to hear from the @paketo-buildpacks/stacks-maintainers and @paketo-buildpacks/builders-maintainers what they think. FWIW I would see the potential to counter that with reducing efforts by changing the stacks and builder setup for 26.04 onwards. We could have single 26.04 stack producing multiple build and/or run images. That way we still have different lists of packages to maintain, but they don't change frequently. The releases would be reduced from currently four to one. Given that pack meanwhile has support for merging layers, we should also be able to extend that to a single builder. Either extensions could be used to change build and/or run images or we could align on the largest build image and the user should pick the run image (instead of picking the builder as they do now).

Regarding 2. & 3. I suppose users usually don't care much about the stack, that's why they use buildpacks and not Dockerfiles in the first place. However, we've seen in the past that the number of known CVEs on Ubuntu LTS releases grows over time. This might not be an immediately worrying concern, as Ubuntu indeed patches critical vulnerabilities throughout the five years an LTS is maintained. But there are CVEs in the security advisory that have been rated low priority while they show a CVSS score >7.
I suppose with security awareness increasing users don't want to see those vulnerabilities in their security scans and having to explain why they are not critical.

Using docker scout cves --format markdown on 22.04 an 24.04 might be a minor difference, but it is a difference.

🔍 Vulnerabilities of ubuntu:22.04

📦 Image Reference ubuntu:22.04

digest	sha256:981912c48e9a89e903c89b228be977e23eeba83d42e2c8e0593a781a2b251cba
vulnerabilities
size	31 MB
packages	143

pam 1.4.0-11ubuntu2.4 (deb) pkg:deb/ubuntu/pam@1.4.0-11ubuntu2.4?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.4 CVSS Vector CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:N EPSS Score 0.09% EPSS Percentile 41st percentile Description A flaw was found in pam_access, where certain rules in its configuration file are mistakenly treated as hostnames. This vulnerability allows attackers to trick the system by pretending to be a trusted hostname, gaining unauthorized access. This issue poses a risk for systems that rely on this feature to control who can access certain services or terminals. Affected range >=0 Fixed version Not Fixed CVSS Score 4.7 CVSS Vector CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N EPSS Score 0.05% EPSS Percentile 23rd percentile Description A vulnerability was found in PAM. The secret information is stored in memory, where the attacker can trigger the victim program to execute by sending characters to its standard input (stdin). As this occurs, the attacker can train the branch predictor to execute an ROP chain speculatively. This flaw could result in leaked passwords, such as those found in /etc/shadow while performing authentications.

krb5 1.19.2-2ubuntu0.4 (deb) pkg:deb/ubuntu/krb5@1.19.2-2ubuntu0.4?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 11th percentile Description Kerberos 5 (aka krb5) 1.21.2 contains a memory leak vulnerability in /krb5/src/kdc/ndr.c. Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 11th percentile Description Kerberos 5 (aka krb5) 1.21.2 contains a memory leak vulnerability in /krb5/src/lib/gssapi/krb5/k5sealv3.c. Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 11th percentile Description Kerberos 5 (aka krb5) 1.21.2 contains a memory leak in /krb5/src/lib/rpc/pmap_rmt.c.

gcc-12 12.3.0-1ubuntu1~22.04 (deb) pkg:deb/ubuntu/gcc-12@12.3.0-1ubuntu1~22.04?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 4.8 CVSS Vector CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N EPSS Score 0.06% EPSS Percentile 26th percentile Description DISPUTEDA failure in the -fstack-protector feature in GCC-based toolchains that target AArch64 allows an attacker to exploit an existing buffer overflow in dynamically-sized local variables in your application without this being detected. This stack-protector failure only applies to C99-style dynamically-sized local variables or those created using alloca(). The stack-protector operates as intended for statically-sized local variables. The default behavior when the stack-protector detects an overflow is to terminate your application, resulting in controlled loss of availability. An attacker who can exploit a buffer overflow without triggering the stack-protector might be able to change program flow control to cause an uncontrolled loss of availability or to go further and affect confidentiality or integrity. NOTE: The GCC project argues that this is a missed hardening bug and not a vulnerability by itself. Affected range >=0 Fixed version Not Fixed CVSS Score 5.5 CVSS Vector CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H EPSS Score 0.09% EPSS Percentile 39th percentile Description libiberty/rust-demangle.c in GNU GCC 11.2 allows stack consumption in demangle_const, as demonstrated by nm-new.

ncurses 6.3-2ubuntu0.1 (deb) pkg:deb/ubuntu/ncurses@6.3-2ubuntu0.1?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 6.5 CVSS Vector CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H EPSS Score 0.06% EPSS Percentile 28th percentile Description NCurse v6.4-20230418 was discovered to contain a segmentation fault via the component _nc_wrap_entry(). Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 11th percentile Description ncurses 6.4-20230610 has a NULL pointer dereference in tgetstr in tinfo/lib_termcap.c.

openssl 3.0.2-0ubuntu1.18 (deb) pkg:deb/ubuntu/openssl@3.0.2-0ubuntu1.18?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 17th percentile Description Validating the order of the public keys in the Diffie-Hellman Key Agreement Protocol, when an approved safe prime is used, allows remote attackers (from the client side) to trigger unnecessarily expensive server-side DHE modular-exponentiation calculations. The client may cause asymmetric resource consumption. The basic attack scenario is that the client must claim that it can only communicate with DHE, and the server must be configured to allow DHE and validate the order of the public key.

libzstd 1.4.8+dfsg-3build1 (deb) pkg:deb/ubuntu/libzstd@1.4.8%2Bdfsg-3build1?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.5 CVSS Vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H EPSS Score 0.22% EPSS Percentile 61st percentile Description A vulnerability was found in zstd v1.4.10, where an attacker can supply empty string as an argument to the command line tool to cause buffer overrun.

glibc 2.35-0ubuntu3.8 (deb) pkg:deb/ubuntu/glibc@2.35-0ubuntu3.8?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.5 CVSS Vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H EPSS Score 0.20% EPSS Percentile 59th percentile Description sha256crypt and sha512crypt through 0.6 allow attackers to cause a denial of service (CPU consumption) because the algorithm's runtime is proportional to the square of the length of the password.

pcre2 10.39-3ubuntu0.1 (deb) pkg:deb/ubuntu/pcre2@10.39-3ubuntu0.1?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.5 CVSS Vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H EPSS Score 0.09% EPSS Percentile 38th percentile Description Integer overflow vulnerability in pcre2test before 10.41 allows attackers to cause a denial of service or other unspecified impacts via negative input.

libgcrypt20 1.9.4-3ubuntu3 (deb) pkg:deb/ubuntu/libgcrypt20@1.9.4-3ubuntu3?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 17th percentile Description A timing-based side-channel flaw was found in libgcrypt's RSA implementation. This issue may allow a remote attacker to initiate a Bleichenbacher-style attack, which can lead to the decryption of RSA ciphertexts.

gnupg2 2.2.27-3ubuntu2.1 (deb) pkg:deb/ubuntu/gnupg2@2.2.27-3ubuntu2.1?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 3.3 CVSS Vector CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:L EPSS Score 0.05% EPSS Percentile 19th percentile Description GnuPG can be made to spin on a relatively small input by (for example) crafting a public key with thousands of signatures attached, compressed down to just a few KB.

systemd 249.11-0ubuntu3.12 (deb) pkg:deb/ubuntu/systemd@249.11-0ubuntu3.12?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 5.9 CVSS Vector CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:H/A:N EPSS Score 0.07% EPSS Percentile 33rd percentile Description A vulnerability was found in systemd-resolved. This issue may allow systemd-resolved to accept records of DNSSEC-signed domains even when they have no signature, allowing man-in-the-middles (or the upstream DNS resolver) to manipulate records.

coreutils 8.32-4.1ubuntu1.2 (deb) pkg:deb/ubuntu/coreutils@8.32-4.1ubuntu1.2?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 6.5 CVSS Vector CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:C/C:N/I:H/A:N EPSS Score 0.04% EPSS Percentile 5th percentile Description chroot in GNU coreutils, when used with --userspec, allows local users to escape to the parent session via a crafted TIOCSTI ioctl call, which pushes characters to the terminal's input buffer.

shadow 1:4.8.1-2ubuntu2.2 (deb) pkg:deb/ubuntu/shadow@1:4.8.1-2ubuntu2.2?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 3.3 CVSS Vector CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N EPSS Score 0.05% EPSS Percentile 22nd percentile Description In Shadow 4.13, it is possible to inject control characters into fields provided to the SUID program chfn (change finger). Although it is not possible to exploit this directly (e.g., adding a new user fails because \n is in the block list), it is possible to misrepresent the /etc/passwd file when viewed. Use of \r manipulations and Unicode characters to work around blocking of the : character make it possible to give the impression that a new user has been added. In other words, an adversary may be able to convince a system administrator to take the system offline (an indirect, social-engineered denial of service) by demonstrating that "cat /etc/passwd" shows a rogue user account.

pcre3 2:8.39-13ubuntu0.22.04.1 (deb) pkg:deb/ubuntu/pcre3@2:8.39-13ubuntu0.22.04.1?os_distro=jammy&os_name=ubuntu&os_version=22.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.5 CVSS Vector CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H EPSS Score 0.37% EPSS Percentile 73rd percentile Description In PCRE 8.41, the OP_KETRMAX feature in the match function in pcre_exec.c allows stack exhaustion (uncontrolled recursion) when processing a crafted regular expression.

🔍 Vulnerabilities of ubuntu:24.04

📦 Image Reference ubuntu:24.04

digest	sha256:cdc507f6026a62440bc601815bba185960e93f8b971112722cebe3cae1e125a0
vulnerabilities
size	29 MB
packages	130

pam 1.5.3-5ubuntu5.1 (deb) pkg:deb/ubuntu/pam@1.5.3-5ubuntu5.1?os_distro=noble&os_name=ubuntu&os_version=24.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.4 CVSS Vector CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:N EPSS Score 0.09% EPSS Percentile 41st percentile Description A flaw was found in pam_access, where certain rules in its configuration file are mistakenly treated as hostnames. This vulnerability allows attackers to trick the system by pretending to be a trusted hostname, gaining unauthorized access. This issue poses a risk for systems that rely on this feature to control who can access certain services or terminals. Affected range >=0 Fixed version Not Fixed CVSS Score 4.7 CVSS Vector CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N EPSS Score 0.05% EPSS Percentile 23rd percentile Description A vulnerability was found in PAM. The secret information is stored in memory, where the attacker can trigger the victim program to execute by sending characters to its standard input (stdin). As this occurs, the attacker can train the branch predictor to execute an ROP chain speculatively. This flaw could result in leaked passwords, such as those found in /etc/shadow while performing authentications.

libgcrypt20 1.10.3-2build1 (deb) pkg:deb/ubuntu/libgcrypt20@1.10.3-2build1?os_distro=noble&os_name=ubuntu&os_version=24.04 Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 17th percentile Description A timing-based side-channel flaw was found in libgcrypt's RSA implementation. This issue may allow a remote attacker to initiate a Bleichenbacher-style attack, which can lead to the decryption of RSA ciphertexts.

openssl 3.0.13-0ubuntu3.4 (deb) pkg:deb/ubuntu/openssl@3.0.13-0ubuntu3.4?os_distro=noble&os_name=ubuntu&os_version=24.04 Affected range >=0 Fixed version Not Fixed EPSS Score 0.04% EPSS Percentile 17th percentile Description Validating the order of the public keys in the Diffie-Hellman Key Agreement Protocol, when an approved safe prime is used, allows remote attackers (from the client side) to trigger unnecessarily expensive server-side DHE modular-exponentiation calculations. The client may cause asymmetric resource consumption. The basic attack scenario is that the client must claim that it can only communicate with DHE, and the server must be configured to allow DHE and validate the order of the public key.

glibc 2.39-0ubuntu8.3 (deb) pkg:deb/ubuntu/glibc@2.39-0ubuntu8.3?os_distro=noble&os_name=ubuntu&os_version=24.04 Affected range >=0 Fixed version Not Fixed CVSS Score 7.5 CVSS Vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H EPSS Score 0.20% EPSS Percentile 59th percentile Description sha256crypt and sha512crypt through 0.6 allow attackers to cause a denial of service (CPU consumption) because the algorithm's runtime is proportional to the square of the length of the password.

coreutils 9.4-3ubuntu6 (deb) pkg:deb/ubuntu/coreutils@9.4-3ubuntu6?os_distro=noble&os_name=ubuntu&os_version=24.04 Affected range >=0 Fixed version Not Fixed CVSS Score 6.5 CVSS Vector CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:C/C:N/I:H/A:N EPSS Score 0.04% EPSS Percentile 5th percentile Description chroot in GNU coreutils, when used with --userspec, allows local users to escape to the parent session via a crafted TIOCSTI ioctl call, which pushes characters to the terminal's input buffer.

gnupg2 2.4.4-2ubuntu17 (deb) pkg:deb/ubuntu/gnupg2@2.4.4-2ubuntu17?os_distro=noble&os_name=ubuntu&os_version=24.04 Affected range >=0 Fixed version Not Fixed CVSS Score 3.3 CVSS Vector CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:L EPSS Score 0.05% EPSS Percentile 19th percentile Description GnuPG can be made to spin on a relatively small input by (for example) crafting a public key with thousands of signatures attached, compressed down to just a few KB.

[loewenstein-sap]

@dmikusa I'd say that #313 would make a decision to follow every LTS easier. Would you agree?

[dmikusa]

Yes @loewenstein-sap - That would alleviate my concerns for 1.) and if it's not materially increasing the amount of work the project needs to do, then I'm less inclined to be picky about 2.). I do agree that there is a subset of users where the non-critical unpatched vulnerabilities in our Ubuntu buildpacks are a bit annoying. If it's not a lot of work, and we can make things better for them, then 👍

[dmikusa]

Do we really want to wait until every last buildpack has been verified before we release the builders with buildpacks? If one buildpack team is not responsive, this can delay the whole process with no limit.

I could see some other possibilities like:

• Buildpack teams have three months to verify and apply any fixes required to make their buildpack compatible. After three months, we release the builders with all the buildpacks that have been verified, omitting any that have not been marked as verified. As buildpack teams confirm compatibility the builder will be updated to include those buildpacks as well.

• The builder team will release the builders with buildpacks in beta mode when we have confirmed that 50% of the buildpacks are compatible. After that, as buildpack teams confirm compatibility the builder will be updated to include those buildpacks as well. The builder will exit beta when 100% of the buildpacks have been verified.

Thoughts?

[pacostas]

The builder team will release the builders with buildpacks in beta mode when we have confirmed that 50% of the buildpacks are compatible.

50% of the buildpacks from what is included on the previous builder, or 50% of the buildpacks from all the buildpacks under the paketo org?

The builder will exit beta when 100% of the buildpacks have been verified.

I think it will never exit. Can we omit the beta concept and just publish the builder once we have reach the 50% ?

[dmikusa]

After going through Noble, my vote would be to release the builder as soon as we have a single composite that is ready to go. It's frustrating working on a composite buildpack, and having it done, but it can't go into the builder because other buildpack teams haven't verified yet.

You can introspect a builder to see what's in it. We document what's in it (README). It should be fairly clear what users should get when they use a builder. I would vote for just releasing what we have when we have it.

[loewenstein-sap]

I'll pick this up again once Paketo Noble is released.

[pacostas]

First is necessary to open an RFC about the new stack

[pacostas]

Is also necessary for an RFC about the builder repo

[pacostas]

Suggested change

	4. Once the buildpackless builders are available, the individual buildpack teams will begin to evaluate and test their buildpacks on the new builders.
	4. Once the buildpackless builders are available, the individual buildpack teams will begin to evaluate and test their buildpacks on the new buildpackless builder.

[pacostas]

Suggested change

	5. Once all buildpacks have been confirmed to work on the new builders, the Builders team releases the buildpackfull builders.
	5. Once all buildpacks have been confirmed to work on the new buildpackless builder, the Builders team releases the buildpackfull builders.

[pacostas]

The builder also needs to be added on the samples repo for each composite buildpack that the builder works with, example P.R. paketo-buildpacks/samples#1165

[pacostas]

Then the builder needs to be added to the reference page https://paketo.io/docs/reference/builders-reference/

example PR paketo-buildpacks/paketo-website#907

[pacostas]

Final step is the builder needs to be added on the pack CLI on the suggested builders https://github.com/buildpacks/pack/blob/main/internal/builder/trusted_builder.go

Examle PR https://github.com/buildpacks/pack/pull/2383/files

[pacostas]

The builder team will release the builders with buildpacks in beta mode when we have confirmed that 50% of the buildpacks are compatible.

50% of the buildpacks from what is included on the previous builder, or 50% of the buildpacks from all the buildpacks under the paketo org?

The builder will exit beta when 100% of the buildpacks have been verified.

I think it will never exit. Can we omit the beta concept and just publish the builder once we have reach the 50% ?

[pacostas]

I think this RFC is good time to shape it, as we have the noble builder as an example.

[pacostas]

@dmikusa

I worry about this creating more work than we as a team can handle.

Yes, good point. I've been through that for the past 4 months for the ubi9(buildres/stacks) and for the noble(builders/stacks), and I believe is zero overhead (approximately no more than a day of work for everything). In the past was quite a lot of work, as we had quite a few builders and stacks , in terms of repositories, to create/maintain, but based on the latest refactor, practically is copy pasting the two repos (base imagse/stacks and builders) and pretty match replacing the noble keyword or a similar keyword.