add support libkrun by saku3 · Pull Request #3234 · youki-dev/youki

saku3 · 2025-09-02T23:39:34Z

Description

Add support libkrun

See here for more details.: https://gist.github.com/saku3/7594fe4d2bd1a8d0edbb8b1d0e2e255c

How to

Build

./scripts/build.sh -o . -r -f libkrun

Run

Assuming libkrun and libkrunfw are installed.

 LD_LIBRARY_PATH=/usr/local/lib64 ./youki run -b tutorial container

Impact Scope

libcontainer
- Modified so that ContainerBuilder::build invokes pre_exec.
- This change is not expected to affect normal container startup behavior.
Executor
- Uses a feature flag; the libkrun-specific exec runs only when launching libkrun-based containers, so there’s no impact on others.
- Since the executor needs to keep libkrun-related state (lib, ctx_id), we adjusted DefaultExecutor to reuse a single executor instance (calling get_executor repeatedly would re-initialize state).

sequenceDiagram
    participant youki
    participant Main Process
    participant Intermediate Process
    participant Init Process
    participant Executor
    participant libkrun
    participant kvm

    Note over youki: load config.json
    rect rgb(235,250,235) 
        youki->>Executor: pre_exec()
        opt feature "libkrun"
            Executor->>libkrun: load libkrun.so
            Executor->>libkrun: krun_create_ctx()
            Note over Executor: modify config.json
            Note over Executor: setup .krun_config.json
        end
    end
    youki->>Main Process: container_main_process()
    Main Process->>Intermediate Process: clone3
    Intermediate Process->>Init Process: clone3
    Note over Init Process: pivot_root
    Init Process->>Executor: exec()
    rect rgb(235,250,235)
        opt feature "libkrun"
            Executor->>libkrun: krun_set_vm_config()
            Executor->>libkrun: krun_set_root()
            Executor->>libkrun: krun_set_log_level(1)
            Executor->>libkrun: krun_start_enter()
            libkrun->>kvm: start
        end
    end

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)
Performance improvement
Test updates
CI/CD related changes
Other (please describe):

Testing

Added new unit tests
Added new integration tests
Ran existing test suite
Tested manually (please provide steps)

Related Issues

Part of # #3196

Additional Context

saku3 · 2025-09-13T02:56:28Z

@utam0k

This PR is ready for review. Please take a look when you have a moment.

Copilot

Pull Request Overview

This PR adds support for libkrun to youki, enabling containers to run inside lightweight KVM-based virtual machines. The implementation adds a new executor that interfaces with the libkrun library to provide VM-based container isolation.

Introduces a new libkrun executor with VM configuration and KVM device management
Modifies the executor architecture to support pre-execution hooks for VM setup
Adds feature flag support for optional libkrun functionality

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
crates/youki/src/workload/mod.rs	Adds conditional module import for libkrun support
crates/youki/src/workload/libkrun.rs	New module implementing the libkrun executor with VM management
crates/youki/src/workload/executor.rs	Updates DefaultExecutor to include libkrun support and pre-execution hooks
crates/youki/Cargo.toml	Adds libkrun feature flag and required dependencies
crates/libcontainer/src/workload/mod.rs	Adds pre_exec trait method to Executor interface
crates/libcontainer/src/workload/default.rs	Implements default pre_exec method
crates/libcontainer/src/error.rs	Adds ExecutorError to LibcontainerError enum
crates/libcontainer/src/container/init_builder.rs	Integrates pre_exec call into container building process

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-13T12:26:44Z

crates/youki/src/workload/libkrun.rs

+use nix::sys::stat::{major, minor, stat, Mode};
+
+const EXECUTOR_NAME: &str = "libkrun";
+const LIBKRUN_NAME: &str = "libkrun.so.1";


The hardcoded library name 'libkrun.so.1' may not be portable across different systems. Consider making this configurable or using a more flexible library loading approach that can handle different versions.

+1. How about using the annotation and default value?

I modified it to use annotations and default value.

Copilot · 2025-09-13T12:26:44Z

crates/youki/src/workload/libkrun.rs

+        krun.set_root(ctx_id, &root)?;
+
+        let res = krun.start_enter(ctx_id)?;
+        std::process::exit(res)


Direct call to std::process::exit() makes the code difficult to test and can cause issues in library contexts. Consider returning the exit code and letting the caller handle the exit.

Copilot · 2025-09-13T12:26:44Z

crates/youki/src/workload/executor.rs

+    fn pre_exec(&self, spec: Spec) -> Result<Spec, ExecutorError> {
+        #[cfg(feature = "libkrun")]
+        {
+            let res = self.libkrun.pre_exec(spec.clone());


Cloning the entire spec can be expensive for large specifications. Consider using references or only cloning when necessary based on the libkrun executor's needs.

Suggested change

let res = self.libkrun.pre_exec(spec.clone());

let res = self.libkrun.pre_exec(&spec);

fix not using clone

Copilot · 2025-09-13T12:26:45Z

crates/youki/Cargo.toml

 tracing-subscriber = { version = "0.3.20", features = ["json", "env-filter"] }
 tracing-journald = "0.3.1"
+libloading = { version = "0.8",  optional = true }
+thiserror = { version ="2.0.16", optional = true }


Missing space after the equals sign. Should be 'version = "2.0.16"' for consistency with other dependencies.

utam0k · 2025-09-14T07:27:01Z

crates/youki/src/workload/mod.rs

@@ -1,4 +1,6 @@
 pub mod executor;
+#[cfg(feature = "libkrun")]


Do we really need the feature flag? The libkrun feature has already been protected by the annotation to prevent accidental running. And I don't think this feature will increase the binary and memory footprint size, right?(We should measure that.)

Do we need the additional dependency to run or build youki?

I added libloading to call (dynamically load) the libkrun library. With the current code, the resulting binary sizes were:

youki (release):6.2 MB

youki + libkrun: 6.4 MB

(youki + WasmEdge): 111 MB

If the differences above aren’t an issue, I’ll remove the features flag.

utam0k

I ran out of time today, but I've left a few comments on things that caught my attention.

utam0k · 2026-01-04T12:04:33Z

crates/youki/src/workload/libkrun.rs

+}
+
+// Add /dev/kvm to `linux.devices` so libkrun can access KVM device.
+pub fn modify_spec_device(spec: &mut Spec) -> Result<(), KrunError> {


Let's make the name a bit more precise. It's too abstract. What this function does is exactly as described in the explanation just above it.

utam0k · 2026-01-04T12:04:44Z

crates/youki/src/workload/libkrun.rs

+
+// Add an allow rule for /dev/kvm to linux.resources.devices
+// if resources.devices is None or empty, it's effectively permissive, so skip.
+pub fn modify_spec_resource_device(spec: &mut Spec) -> Result<(), KrunError> {


Let's make the name a bit more precise. It's too abstract. What this function does is exactly as described in the explanation just above it.

utam0k · 2026-01-04T12:18:44Z

crates/libcontainer/src/workload/mod.rs


 pub trait Executor: CloneBoxExecutor {
+    /// Pre Executes the workload
+    fn pre_exec(&self, spec: Spec) -> Result<Spec, ExecutorError> {


I've reconsidered, and including exec could be misleading.

Also, we need to clarify for the library users:

a: When is it called?

b: Where are the methods called from?

'a' is already expressed in the method name. What about 'b'?
My proposal is to separate the trait based on whether it's within a container or on the host. For example:

trait HostExecutor { fn modify_spec(&self, spec: Spec) -> Result<Spec, ExecutorError> { Ok(spec) } } trait ContainerExecutor { fn exec(&self, spec: &Spec) -> Result<(), ExecutorError>; fn validate(&self, spec: &Spec) -> Result<(), ExecutorValidationError>; } trait Executor: HostExecutor + ContainerExecutor + CloneBoxExecutor {}

Thank you for the review.
If we follow the approach above, do we need to split the existing executors that implement Executor (e.g., DefaultExecutor, WasmedgeExecutor, etc.) into HostExecutor and ContainerExecutor implementations?

We need it. But, I'd like to get other @youki-dev/youki's opinion.

I split it into HostExecutor and ContainerExecutor.
As a result, I updated the other executors as well (e.g., DefaultExecutor, WasmedgeExecutor, etc.). I confirmed that the build succeeds.

utam0k · 2026-01-04T12:23:38Z

crates/youki/src/workload/libkrun.rs

+    let (kvm_major, kvm_minor) = match stat_dev_numbers("/dev/kvm") {
+        Ok(v) => v,
+        Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()),
+        Err(e) => return Err(KrunError::Other(format!("stat `/dev/kvm`: {e}"))),
+    };


How about adding a validation to return as early as possible?

I updated it so that validate_kvm function is executed inside modify_spec to check whether KVM is available.

utam0k · 2026-01-04T12:29:09Z

crates/youki/src/workload/libkrun.rs

+// Add an allow rule for /dev/kvm to linux.resources.devices
+// if resources.devices is None or empty, it's effectively permissive, so skip.
+pub fn modify_spec_resource_device(spec: &mut Spec) -> Result<(), KrunError> {
+    let mut linux = match spec.linux() {


Please use mutating getters. Everything else is the same.

I updated the code to use mutating getters.

utam0k · 2026-01-05T06:12:46Z

Could you please write the user documentation for this feature?

Signed-off-by: Yusuke Sakurai <yusuke.sakurai@3-shake.com>

saku3 · 2026-01-31T06:51:51Z

crates/youki/src/workload/libkrun.rs

+    let (kvm_major, kvm_minor) = match stat_dev_numbers("/dev/kvm") {
+        Ok(v) => v,
+        Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()),
+        Err(e) => return Err(KrunError::Other(format!("stat `/dev/kvm`: {e}"))),
+    };


I updated it so that validate_kvm function is executed inside modify_spec to check whether KVM is available.

saku3 · 2026-01-31T07:01:24Z

crates/youki/src/workload/libkrun.rs

+// see: https://github.com/containers/crun/blob/92977c0fc843e4649fe4611a97ba12b06cb5073f/src/libcrun/handlers/krun.c#L514C1-L515C72
+pub fn write_krun_config(rootfs: &Path, json_spec: &str) -> Result<(), KrunError> {
+    let mut root =
+        Root::open(rootfs).map_err(|e| KrunError::Other(format!("failed to open rootfs {e}")))?;


I changed this implementation to use libpathrs.

saku3 · 2026-01-31T07:23:49Z

@utam0k

Could you please write the user documentation for this feature?

I added documentation and believe I’ve addressed the items raised in the review.

Sorry for the trouble, but I’d appreciate it if you could take another look.

utam0k · 2026-02-05T11:19:49Z

docs/src/user/libkrun.md

@@ -0,0 +1,97 @@
+# Libkrun


Technically, it seems to be in lowercase. Also, isn't it more of a VM than libkrun?
https://github.com/containers/libkrun

Suggested change

# Libkrun

# VM Support(libkrun)

utam0k · 2026-02-05T11:21:48Z

docs/src/user/libkrun.md

@@ -0,0 +1,97 @@
+# Libkrun
+
+youki can launch microVM-based containers by dynamically loading the libkrun shared library. Here, we describe how to launch a container with youki using libkrun.


What's microVM-based containers? It'd be better to put one big picture (using Excalidraw?)

utam0k · 2026-02-05T11:22:09Z

docs/src/user/libkrun.md

+
+## Prepare KVM on the host
+
+Libkrun uses KVM to start a microVM.


Suggested change

Libkrun uses KVM to start a microVM.

libkrun uses KVM to start a microVM.

utam0k · 2026-02-05T11:23:26Z

docs/src/user/libkrun.md

+LD_LIBRARY_PATH=/usr/local/lib64 youki run -b tutorial container
+```
+
+### Keys


What are keys?

utam0k · 2026-02-05T11:25:47Z

docs/src/user/libkrun.md

+
+## Install libkrun and libkrunfw
+
+To use libkrun with youki, you need to install libkrun and libkrunfw in advance.


Suggested change

To use libkrun with youki, you need to install libkrun and libkrunfw in advance.

To use youki with libkrun, you need to install libkrun and libkrunfw in advance.

utam0k · 2026-02-05T11:26:39Z

docs/src/user/libkrun.md

+```bash
+ls /usr/local/lib64 
+libkrun.so  libkrun.so.1  libkrun.so.1.17.0  libkrunfw.so  libkrunfw.so.5  libkrunfw.so.5.1.0  pkgconfig
+```


Should the versions be specified?

utam0k · 2026-02-05T11:35:02Z

crates/libcontainer/src/workload/mod.rs

 }

-pub trait Executor: CloneBoxExecutor {
+pub trait HostExecutor {


Could you write the description and why these traits are separated?

utam0k · 2026-02-05T11:46:54Z

crates/youki/src/workload/libkrun.rs

+}
+
+impl Krun {
+    fn load(libkrun_path: String) -> Result<Self, ExecutorError> {


Please use P: AsRef<OsStr>

utam0k · 2026-02-05T12:12:13Z

docs/src/user/libkrun.md

Does this PR support exec?

utam0k · 2026-02-05T12:26:30Z

crates/youki/src/workload/libkrun.rs

+
+#[derive(Clone)]
+pub struct LibkrunExecutor {
+    lib: Rc<OnceCell<Rc<Krun>>>,


Do we really Rc and OnceCell? I'd like to avoid them if possible.

saku3 marked this pull request as draft September 2, 2025 23:39

saku3 force-pushed the feat-libkrun branch 2 times, most recently from 95f3f1b to c2489eb Compare September 7, 2025 08:25

saku3 force-pushed the feat-libkrun branch 2 times, most recently from 98b9acc to 5644242 Compare September 13, 2025 02:27

saku3 marked this pull request as ready for review September 13, 2025 02:53

utam0k added the kind/feature label Sep 13, 2025

utam0k requested a review from Copilot September 13, 2025 12:25

Copilot AI reviewed Sep 13, 2025

View reviewed changes

utam0k added this to the v0.6.0 milestone Sep 14, 2025

utam0k reviewed Sep 14, 2025

View reviewed changes

saku3 force-pushed the feat-libkrun branch 3 times, most recently from 78eea37 to b91b3f2 Compare September 15, 2025 09:42

saku3 force-pushed the feat-libkrun branch from 92a3aaa to c085066 Compare November 24, 2025 03:49

saku3 force-pushed the feat-libkrun branch from 30a047a to 2776932 Compare December 26, 2025 23:25

utam0k reviewed Jan 4, 2026

View reviewed changes

saku3 marked this pull request as draft January 25, 2026 01:20

saku3 force-pushed the feat-libkrun branch from cc5af1d to f2baa5f Compare January 31, 2026 03:39

add support libkrun

74258e9

Signed-off-by: Yusuke Sakurai <yusuke.sakurai@3-shake.com>

saku3 force-pushed the feat-libkrun branch from f2baa5f to 74258e9 Compare January 31, 2026 06:09

saku3 commented Jan 31, 2026

View reviewed changes

saku3 marked this pull request as ready for review January 31, 2026 07:23

utam0k reviewed Feb 5, 2026

View reviewed changes

	let res = self.libkrun.pre_exec(spec.clone());
	let res = self.libkrun.pre_exec(&spec);

		@@ -0,0 +1,97 @@
		# Libkrun

		youki can launch microVM-based containers by dynamically loading the libkrun shared library. Here, we describe how to launch a container with youki using libkrun.


		## Prepare KVM on the host

		Libkrun uses KVM to start a microVM.


		## Install libkrun and libkrunfw

		To use libkrun with youki, you need to install libkrun and libkrunfw in advance.

	To use libkrun with youki, you need to install libkrun and libkrunfw in advance.
	To use youki with libkrun, you need to install libkrun and libkrunfw in advance.

Conversation

saku3 commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to

Build

Run

Impact Scope

Type of Change

Testing

Related Issues

Additional Context

Uh oh!

saku3 commented Sep 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

utam0k left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

utam0k commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saku3 commented Jan 31, 2026

saku3 commented Sep 2, 2025 •

edited

Loading