Update 0003-cluster-wide-machineconfigs.yaml.template

[willhaines]

🛈 If accepted, this would require changes to the blog post content.

The librhsm library included in ubi8, ubi8-minimal, and possibly others can become "confused" by both entitlement.pem and entitlement-key.pem including the same content. This library is used by dnf and microdnf.

From rhsm-context.c, it will search the entitlement directory in an indeterminate order for the first *.pem file containing a section bounded by -----BEGIN ENTITLEMENT DATA----- and -----END ENTITLEMENT DATA-----. Then, in rhsm-entitlement-certificate.c it will assume that the key is in an adjacent file with -key appended to the basename.

In a cluster configured per the original blog, if librhsm first sees entitlement.pem, everything will be okay. If, however, librhsm sees entitlement-key.pem first, it will fail with the following error message, looking for entitlement-key-key.pem

error: cannot update repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Curl error (58): Problem with the local SSL certificate for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml [unable to set private key file: '/etc/pki/entitlement-host/entitlement-key-key.pem' type PEM]

[kpouget]

Hello @willhaines, thanks for the PR,

I'm confused by this problem you're raising, as we're using this template almost verbatim in our nightly CI testing, see there:
https://github.com/openshift-psap/ci-artifacts/blob/master/roles/entitlement_deploy/files/mc_pem.yml

but we never had any issue once valid keys are properly deployed ...

how would you suggest handling the .pem file vs the .key.pem one?

[willhaines]

Hi @kpouget,

Apologies for not being more of an expert here, but my assumption is that the order of files returned by g_dir_read_name is not randomized, but likely to be the same each time it is executed in the same environment, but may be different in another environment.

In our case, the solution directly from the blog was working several months ago, but now after many updates to both OKD and the UBI images, we see the error above every time when building a Dockerfile that installs packages on top of a UBI image.

On our registered RHEL systems, the {id}.pem and {id}-key.pem files contain the following sections, which is the arrangement that has also fixed UBI builds on our OKD cluster. I don't know why the RedHat subscription website doesn't provide files in this format, but I propose separating the key section into the -key.pem file, rather than duplicating the entire entitlement.pem file.

entitlement.pem

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN ENTITLEMENT DATA-----
-----END ENTITLEMENT DATA-----
-----BEGIN RSA SIGNATURE-----
-----END RSA SIGNATURE-----

entitlement-key.pem

-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

In the blog, this could be achieved with the following commands:

sed -n '/BEGIN RSA PRIVATE KEY/q;p' {ID}.pem > entitlement.pem
sed -n '/BEGIN RSA PRIVATE KEY/,$p' {ID}.pem > entitlement-key.pem

FWIW, I am also able to replicate this behavior with podman on Fedora 34 and the following Dockerfile

FROM registry.access.redhat.com/ubi8-minimal:8.3-298

RUN microdnf update

If you're running licensed OpenShift, maybe differing kernels and/or glib versions account for the lack of problems in your CI testing. Unfortunately, I cannot think of a good test case to force librhsm to read the "wrong" file first to recreate the problem in an environment that does not naturally suffer from it.

[kpouget]

thanks for the additional information, at least now I understand the difference between entitlement.pem and entitlement-key.pem :)

I tried to replicate the issue with this command, which should be equivalent to what you described, but it works as expected:

$ KEY=....
$ cat $KEY | grep -- ---
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN ENTITLEMENT DATA-----
-----END ENTITLEMENT DATA-----
-----BEGIN RSA SIGNATURE-----
-----END RSA SIGNATURE-----
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

$ podman run --rm -it \
    -v $KEY:/etc/pki/entitlement/entitlement.pem \
    -v $KEY:/etc/pki/entitlement/entitlement-key.pem 
    registry.access.redhat.com/ubi8-minimal:8.3-298 
            microdnf install kernel-devel

HOWEVER, this doesn't work:

$ NAME=key
$ podman run --rm -it -v $KEY:/etc/pki/entitlement/$NAME.pem registry.access.redhat.com/ubi8-minimal:8.3-298 bash -x -c "cp /etc/pki/entitlement/$NAME.pem /etc/pki/entitlement/$NAME-key.pem; microdnf install kernel-devel"

+ cp /etc/pki/entitlement/key.pem /etc/pki/entitlement/key-key.pem
+ microdnf install kernel-devel
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
error: cannot update repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Curl error (58): Problem with the local SSL certificate for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml [unable to set private key file: '/etc/pki/entitlement/key-key-key.pem' type PEM]

• NAME=entite --> doesn't work
• NAME=entitlement --> works

I think we've got something to fix here, thanks for point it out ;-)