`docker save` returns incorrect digests for digest-referenced images when not using containerd storage

[ligfx]

Reported by pvannierop in #13.

When Docker is not configured to use the containerd storage backend, docker save returns an image with a different digest than that returned by containerd. This is fine when referencing images by tag, but breaks referencing images by digest. Kubernetes expects to get exactly that digest back from the registry and will refuse to load the image.

#13 (comment) :

@ligfx It fails with the following log message:

2025/02/19 11:10:06 Attempting to fix registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366
2025/02/19 11:10:06 {"status":"Pulling from ingress-nginx/kube-webhook-certgen","id":"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"}
2025/02/19 11:10:07 {"status":"Digest: sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"}
2025/02/19 11:10:07 {"status":"Status: Image is up to date for registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"}
2025/02/19 11:10:07 Starting buildx session ysy4k1mdadawaea5vuzikdupt
2025/02/19 11:10:07 500 Internal Server Error - HEAD /v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366?ns=registry.k8s.io - (907.797261ms) - %!w(errdefs.errInvalidParameter={0xc0000100f0}): ""
2025/02/19 11:10:07 Exporting Docker image registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/2388d21e8e2b74e055216652e6af384768e97513ad568e10c27bacec389b7f0a
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/2b4246367e170c13f8ce2475016a74c01b0c6ee588912f444946f40c71f16056
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/32ae37dc07be998c2ea491ff1a9826b2873cd0cf6b5c40ccfc65990bb649b7ad
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/42a78a06a8c67134438304ccb397563d8ab0b31db5de333ed97016709966a521
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/501fdb3a56cefd89995b16690b95252aa2d6509cd74c18cdc616eaa69f88a64f
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/521aaba35edf491f606ffd13eef5a214627d2a7357d411ae3402ea86820ddb7e
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/577c8ee06f39e2bc276615f1058fa40081255ce5e2f072df4875e27868de5660
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/684c5ea3b61b299cd4e713c10bfd8989341da91f6175e2e6e502869c0781fb66
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/6a60dedb9757e95f14c86efb1b24dc8b5a60e07f9c1d2716d0b0210d00a97ab5
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/838d01085bb96f1e827d59a5a984b702495e25f4e7b6f264048439a20828a06e
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/94067264fb25c488ca72a39d6d6762a4a6db51c078a3c37d0d782543d20037f5
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/99ed329935af1035f7ee620c22669ef02ce242457d78f28ac201f333bf1816f6
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/9b0858b7501b6c742d7854ffabd7b1165145237ecd33b45c792e4a58f5bf8ceb
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/9ed498e122b248a801130d052c25418381ee7bf215cdf7990965bae0dc37dcc2
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/ac805962e47900b616b2f4b4584a34ac7b07d64ac1fd2c077478cf65311addcc
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/c048279a7d9f8e94b4c022b699ad8e8a0cb08b717b014ce4af15afaf375a6ac2
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/cae4b515950a6771778ed2f0d8b8ae1b7d886799f328ddba9f27119ab5d67d87
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/cc083b1fabafa0fc46f3a25645adbcd728ad243c8e61173af18cf7401f2c2c7d
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/d9ea5ecfae01dcc576aae67db283093658fa99840680ee27b99d8b8e53cfa6c5
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/e4fc30233532112cafc47218c6e4399da38a6306591dd36eb18c4a61a7a3e9cc
2025/02/19 11:10:07 Skipping registry.k8s.io/ingress-nginx/kube-webhook-certgen blobs/sha256/f87ed9f178549998653445efc08158c36f64ed6dc3ce55d834a76f28eae14f13
2025/02/19 11:10:08 Ignoring registry.k8s.io/ingress-nginx/kube-webhook-certgen/sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 index.json
2025/02/19 11:10:08 Ignoring registry.k8s.io/ingress-nginx/kube-webhook-certgen/sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 manifest.json
2025/02/19 11:10:08 Ignoring registry.k8s.io/ingress-nginx/kube-webhook-certgen/sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 oci-layout
2025/02/19 11:10:08 Attempting to fix registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366
2025/02/19 11:10:08 {"status":"Pulling from ingress-nginx/kube-webhook-certgen","id":"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"}
2025/02/19 11:10:08 {"status":"Digest: sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"}
2025/02/19 11:10:08 {"status":"Status: Image is up to date for registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"}
2025/02/19 11:10:08 Starting buildx session rjlzkaodrcxaak6hw8hw89lyn
2025/02/19 11:10:08 500 Internal Server Error - GET /v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366?ns=registry.k8s.io - (1.006627217s) - %!w(errdefs.errInvalidParameter={0xc000306228}): ""

#13 (comment) :

@ligfx Sure! The contents of index.json are:

$ cat index.json
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:501fdb3a56cefd89995b16690b95252aa2d6509cd74c18cdc616eaa69f88a64f","size":2061}]}

And manifest.json:

[{"Config":"blobs/sha256/684c5ea3b61b299cd4e713c10bfd8989341da91f6175e2e6e502869c0781fb66","RepoTags":null,"Layers":["blobs/sha256/32ae37dc07be998c2ea491ff1a9826b2873cd0cf6b5c40ccfc65990bb649b7ad","blobs/sha256/577c8ee06f39e2bc276615f1058fa40081255ce5e2f072df4875e27868de5660","blobs/sha256/9ed498e122b248a801130d052c25418381ee7bf215cdf7990965bae0dc37dcc2","blobs/sha256/4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368","blobs/sha256/af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc","blobs/sha256/ac805962e47900b616b2f4b4584a34ac7b07d64ac1fd2c077478cf65311addcc","blobs/sha256/bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b","blobs/sha256/2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1","blobs/sha256/1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849","blobs/sha256/c048279a7d9f8e94b4c022b699ad8e8a0cb08b717b014ce4af15afaf375a6ac2","blobs/sha256/2388d21e8e2b74e055216652e6af384768e97513ad568e10c27bacec389b7f0a","blobs/sha256/f87ed9f178549998653445efc08158c36f64ed6dc3ce55d834a76f28eae14f13"],"LayerSources":{"sha256:1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":10240,"digest":"sha256:1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849"},"sha256:2388d21e8e2b74e055216652e6af384768e97513ad568e10c27bacec389b7f0a":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":225280,"digest":"sha256:2388d21e8e2b74e055216652e6af384768e97513ad568e10c27bacec389b7f0a"},"sha256:2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":1536,"digest":"sha256:2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1"},"sha256:32ae37dc07be998c2ea491ff1a9826b2873cd0cf6b5c40ccfc65990bb649b7ad":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":327680,"digest":"sha256:32ae37dc07be998c2ea491ff1a9826b2873cd0cf6b5c40ccfc65990bb649b7ad"},"sha256:4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":1536,"digest":"sha256:4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368"},"sha256:577c8ee06f39e2bc276615f1058fa40081255ce5e2f072df4875e27868de5660":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":51200,"digest":"sha256:577c8ee06f39e2bc276615f1058fa40081255ce5e2f072df4875e27868de5660"},"sha256:9ed498e122b248a801130d052c25418381ee7bf215cdf7990965bae0dc37dcc2":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":3379200,"digest":"sha256:9ed498e122b248a801130d052c25418381ee7bf215cdf7990965bae0dc37dcc2"},"sha256:ac805962e47900b616b2f4b4584a34ac7b07d64ac1fd2c077478cf65311addcc":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":2560,"digest":"sha256:ac805962e47900b616b2f4b4584a34ac7b07d64ac1fd2c077478cf65311addcc"},"sha256:af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":2560,"digest":"sha256:af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc"},"sha256:bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":2560,"digest":"sha256:bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b"},"sha256:c048279a7d9f8e94b4c022b699ad8e8a0cb08b717b014ce4af15afaf375a6ac2":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":3072,"digest":"sha256:c048279a7d9f8e94b4c022b699ad8e8a0cb08b717b014ce4af15afaf375a6ac2"},"sha256:f87ed9f178549998653445efc08158c36f64ed6dc3ce55d834a76f28eae14f13":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":52745728,"digest":"sha256:f87ed9f178549998653445efc08158c36f64ed6dc3ce55d834a76f28eae14f13"}}}]

[ligfx]

In v0.8, k3d-registry-dockerd will now detect this situation, log a warning message suggesting the user configure Docker to use the containerd image store, and return HTTP 404 to Kubernetes. This is also called out explicitly in README.md.

I'm not sure if there's anything more that can be done but I'll leave this issue open for reference.