Add dm-verity proposal #1

dallasd1 · 2025-11-21T23:15:43Z

No description provided.

FeynmanZhou · 2025-11-24T18:40:18Z

specs/proposals/dm-verity.md

+
+Signing container images at the manifest level alone is not sufficient to ensure continuous runtime integrity. While manifest signatures protect against image tampering during distribution (image pull), they do not enable enforcement at runtime. Container layers must also have kernel-verifiable signatures to ensure their integrity. With the increasing adoption of immutable infrastructure and zero-trust security models, securing these artifacts with continuous kernel enforcement is critical. By adding per-layer container image signing, Notation can extend its capabilities to enable kernel-enforced container integrity.
+
+The challenge is that OCI registries and container image tools currently support distributing signatures but there is no good tooling for creating per-layer signatures. Existing Notation signatures use JWS or COSE formats, which cannot be verified by the Linux kernel. Kernel-level dm-verity enforcement requires each layer's dm-verity root hash to be accompanied by a PKCS#7 signature that the kernel can check at mount time.


It's recomnneded to define the PKCS#7 signature envelop format in https://github.com/notaryproject/specifications/tree/main/specs

Got it, I'll create a new PR under that repo and refer to it in this document

FeynmanZhou · 2025-11-24T18:41:00Z

specs/proposals/dm-verity.md

+
+The following describes how per-layer dm-verity signing can enhance container security across different attack scenarios:
+
+### Scenario 1: Runtime Layer Tampering (Current Implementation)


This scenario is supposed to be documented and clarified in https://github.com/notaryproject/specifications/blob/main/threatmodels/notation-threatmodel.md

Will create a new PR under that repo and refer to it in this document

FeynmanZhou · 2025-11-24T18:43:07Z

specs/proposals/dm-verity.md

+
+### Extended Notation CLI
+
+Extend `notation sign` with a new `--dm-verity` flag to enable automated per-layer signing:


If a image contains more than 10 layers, does it require the user to sign each layer one by one? Do we consider supporting recursively sign all layers?

This new argument would take the recursive approach where the user runs a single command and then Notation signs all layers defined in its image manifest

FeynmanZhou

Thanks @dallasd1 !

yizha1

Thanks @dallasd1 Overall LGTM. Great to see Notation extending coverage to ensure runtime integrity.

yizha1 · 2025-11-26T22:27:25Z

specs/proposals/dm-verity.md

+2. Per-layer signing capability
+3. Deterministic EROFS image and Merkle tree generation
+4. OCI registry distribution for a new artifact containing the signed layer root hashes via ORAS Attached Artifacts
+5. A new AKV plugin for the PKCS#7 format


We may not need a new AKV plugin as the existing notation AKV plugin returns the raw signature (see the SIGNATURE_GENERATOR.RAW defined in https://github.com/notaryproject/specifications/blob/main/specs/plugin-extensibility.md#plugin-metadata) , and it is the notation CLI to add the signature envelope based on the option "COSE" or "JWS“. In this scenario, the same raw signature is returned by AKV, notation pack PKCS#7 envelope by default for layer signing.

If it is true, we can also support another Azure Trusted signing plugin, as it is also raw signature produced, see https://github.com/Azure/trustedsigning-notation-plugin, but trusted signing generating shot-lived certificate, validity is less than 3 days, I am not sure whether this meet the certificate requirements for dm-verity

Removed the AKV plugin requirement for now and can come back to this point in the next review iteration

yizha1 · 2025-11-26T22:30:02Z

specs/proposals/dm-verity.md

+```bash
+notation sign --dm-verity \
+  --id myKeyId \
+  --signature-format pkcs7 \


Using --signature-format pkcs7 also enforce the manifest signature to be the pkcs7, maybe it is only the layer signing should be using pkcs7, and if users don't specific, the default format for layer signing should be pkcs7 as well as it is the only format.

Added a blurb in the Extended Notation CLI section explaining this and removed the --signature-format argument from the command.

yizha1 · 2025-11-26T22:37:55Z

specs/proposals/dm-verity.md

+**Verification command:**
+
+```bash
+notation verify myregistry.azurecr.io/myapp@sha256:def456...


As it dm-verity is at runtime, so the notation verify command doesn't verify layers, the behavior is the same as existing one, just output addtional information for dm-verity

That's correct. It should be able to be done by the verify API inspecting either the artifact type of the referrer or the annotation within the referrer (io.cncf.notary.dm-verity.signature=true).

Expanding verify to work with the dm-verity layer hashes also may not be strictly required since the kernel enforce the signatures exist

yizha1 · 2025-11-26T22:39:53Z

specs/proposals/dm-verity.md

+
+If an attacker attempts to modify a layer like in the first scenario, the root hash verification fails immediately and the kernel refuses to mount the tampered layer. If an attacker drops an unsigned binary into a running container and tries to execute it, IPE blocks execution because it was not loaded from a signed dm-verity volume.
+
+## Proposal


How about: Multi-arch image signing? For example, when users sign the index manifest of multi-arch image. Currently Notation doesn't support recursively signing multi-arch image, which means users sign the index first, then users need to sign individual arch manifest one by one.

Added a blurb in the Extended Notation CLI section. The goal would be to deviate from the default manifest signing as little as possible. So, for dm-verity signing, the same would happen meaning each individual arch manifest would have to be signed by the user. The index and manifest referrers don't contain root hashes so no need to sign those with the dm-verity arg.

yizha1 · 2025-11-26T22:45:21Z

specs/proposals/dm-verity.md

+If an attacker attempts to modify a layer like in the first scenario, the root hash verification fails immediately and the kernel refuses to mount the tampered layer. If an attacker drops an unsigned binary into a running container and tries to execute it, IPE blocks execution because it was not loaded from a signed dm-verity volume.
+
+## Proposal
+


Another thing to consider is Notation supports signing OCI image layout, which is a directory structure on filesystem. This allows notation to sign images on local filesystem without registries, after signing, users can use oras to publish images including all signatures to remote registries. This way is more secure as images and sigantures are within the same trust boundary during signing.

Added a blurb in the Extended Notation CLI section to explain this is also a requirement. However, I haven't implemented this in the minimal prototype.

yizha1 · 2025-11-26T23:03:22Z

specs/proposals/dm-verity.md

+1. **Pull the image manifest** from the registry.
+2. **Iterate through all layers** in the manifest. For each layer:
+    - Pull the layer blob
+    - Generate an EROFS image by decompressing the tar layer and using `mkfs.erofs` (deterministic, read-only filesystem image)


Is there a limitation for layer size?

There is no hardcoded size limit so it would likely depend on the size of the directory the operations take place in, which would be the /tmp directory. There is a timeout set to 5 minutes.

yizha1 · 2025-11-26T23:06:08Z

specs/proposals/dm-verity.md

+2. **Iterate through all layers** in the manifest. For each layer:
+    - Pull the layer blob
+    - Generate an EROFS image by decompressing the tar layer and using `mkfs.erofs` (deterministic, read-only filesystem image)
+    - Compute the dm-verity Merkle tree root hash from the EROFS image


Do we have an estimated time for signing, say, a 1 MB layer

It's around 5s regardless of layer size. Added section Performance Metrics to capture this.

This consistent 5s time is likely due to the other processes like decompressing the tar, writing files, starting mkfs.erofs, etc. taking longer than the data processing like generating the Merkle tree and calculating the root hash of the actual layer

yizha1 · 2025-11-26T23:07:26Z

specs/proposals/dm-verity.md

+    - Digest and size of the PKCS#7 signature file
+4. **Create a signature manifest** containing:
+    - All per-layer signature envelopes
+5. **Attach the signature manifest** to the image manifest as an attached artifact (referrer) in the registry.


What will happen if the dm-verity signature manifest is removed or modified by maliciuos users?

Good question - if the dm-verity manifest is completely removed, the containerd snapshotter (pending upstream work) will look for root hashes from the dm-verity referrer artifact. When the snapshotter doesn't find any, container layer mounting will fail and prevent the container workload from running.

If the dm-verity referrer artifact is modified, if the root hashes were modified, the snapshotter will calculate the root hashes at runtime and compare them with the values in the referrer artifact. If they match, the layer can get mounted. If they don't match the layer will not be mounted like in the first scenario.

If the signatures were tampered with but the root hashes remain the same, the snapshotter will see the root hash calculated at runtime matches the value in the manifest and mount each container image layer. In this case the root hash signatures are tampered with, so when a binary from the container image tries to execute, the kernel detects that the layer's signature is invalid and IPE prevents the binary from executing even though the layers mounted successfully.

yizha1 · 2025-11-26T23:08:36Z

specs/proposals/dm-verity.md

+This work is ongoing in the containerd project and is described in milestone 1 of the [RFC for Code Integrity](https://github.com/containerd/containerd/issues/12081). The EROFS containerd snapshotter implements the following:
+
+1. **Container Start**: When a container is scheduled, fetch the image manifest
+2. **Signature Discovery**: Fetch the signature referrer artifact if one is attached to the image manifest


I’d like to understand the performance impact of runtime verification.

This is a little trickier to calculate but I'm finding ~10ms for the end-to-end for image+referrer pull to pod exit for a single layer "hello world" image.

The problem is that the erofs snapshotter PR is under review but has not been merged into containerd yet (milestone1.2 in this RFC). To test these Notation changes, I am using a prototype snapshotter (tardev) that already has the required changes. This prototype tardev snapshotter will inherently have a different execution time than the erofs snapshotter.

Additionally, the containerd changes that utilize the signing portion of these changes (milestone1.3 in this RFC) have not been created yet. The performance impact we may want to capture is the erofs snapshotter with vs. without signing validation.

Signed-off-by: Dallas Delaney <dadelan@microsoft.com>

Add perf metrics Comment on OCI format and multi-arch scenarios Signed-off-by: Dallas Delaney <dadelan@microsoft.com>

FeynmanZhou reviewed Nov 24, 2025

View reviewed changes

yizha1 reviewed Nov 26, 2025

View reviewed changes

dallasd1 added 4 commits December 29, 2025 10:36

add dm-verity proposal

f38089f

Signed-off-by: Dallas Delaney <dadelan@microsoft.com>

update attack scenarios

6de803f

Signed-off-by: Dallas Delaney <dadelan@microsoft.com>

update verify command documentation

f53439a

Signed-off-by: Dallas Delaney <dadelan@microsoft.com>

Remove AKV requirement

b4efd40

Add perf metrics Comment on OCI format and multi-arch scenarios Signed-off-by: Dallas Delaney <dadelan@microsoft.com>

dallasd1 force-pushed the dm-verity-proposal branch from 53be9dd to b4efd40 Compare December 29, 2025 18:38


		Signing container images at the manifest level alone is not sufficient to ensure continuous runtime integrity. While manifest signatures protect against image tampering during distribution (image pull), they do not enable enforcement at runtime. Container layers must also have kernel-verifiable signatures to ensure their integrity. With the increasing adoption of immutable infrastructure and zero-trust security models, securing these artifacts with continuous kernel enforcement is critical. By adding per-layer container image signing, Notation can extend its capabilities to enable kernel-enforced container integrity.

		The challenge is that OCI registries and container image tools currently support distributing signatures but there is no good tooling for creating per-layer signatures. Existing Notation signatures use JWS or COSE formats, which cannot be verified by the Linux kernel. Kernel-level dm-verity enforcement requires each layer's dm-verity root hash to be accompanied by a PKCS#7 signature that the kernel can check at mount time.


		The following describes how per-layer dm-verity signing can enhance container security across different attack scenarios:

		### Scenario 1: Runtime Layer Tampering (Current Implementation)


		### Extended Notation CLI

		Extend `notation sign` with a new `--dm-verity` flag to enable automated per-layer signing:


		If an attacker attempts to modify a layer like in the first scenario, the root hash verification fails immediately and the kernel refuses to mount the tampered layer. If an attacker drops an unsigned binary into a running container and tries to execute it, IPE blocks execution because it was not loaded from a signed dm-verity volume.

		## Proposal

Add dm-verity proposal #1

Are you sure you want to change the base?

Add dm-verity proposal #1

Conversation

dallasd1 commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FeynmanZhou left a comment

Choose a reason for hiding this comment

Uh oh!

yizha1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dallasd1 Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dallasd1 Dec 9, 2025 •

edited

Loading