Use mio to replace Epoll #2

uran0sH · 2025-07-13T15:49:58Z

Epoll is linux-specific. So we use mio, which is a cross-platform event notification, to replace Epoll.
This is a draft version.

uran0sH · 2025-07-14T17:16:51Z

cc @stefano-garzarella @germag @osteffenrh

vhost-user-backend/src/backend.rs

vhost-user-backend/src/event_loop.rs

uran0sH · 2025-07-16T01:14:20Z

vhost-user-backend/src/event_loop.rs

 pub struct VringEpollHandler<T: VhostUserBackend> {
-    epoll: Epoll,
+    poller: Mutex<Poll>,
+    fd_map: DashMap<RawFd, Vec<(Interest, u64)>>,


Because mio couldn't deregister a single Interest and Token， we use this structure to record which Interest and Token we need. Maybe are there any better ways?

uran0sH · 2025-07-23T07:28:44Z

I found a deadlock problem. https://github.com/uran0sH/vhost/pull/2/files#diff-e1c9dc7802234bc118de8314ec9cacc98066eea325a6f14ff5887ece3466d5a3R260 Here will hold the lock to listen for events's arrival. https://github.com/uran0sH/vhost/pull/2/files#diff-e1c9dc7802234bc118de8314ec9cacc98066eea325a6f14ff5887ece3466d5a3R193 A lock is also needed here to register events. So this is what happens

      thread A                                                                                     thread B
poller.lock().unwrap().register(exit_event_fd)
poller.lock().unwrap().poll()
         |
         | Nerver get the exit event besides exit, so                          poller.lock().unwrap().registry().register()
         | this will block and hold the lock

solve this problem by using the following method, but is there any other better method? @stefano-garzarella @germag @osteffenrh

        'poll: loop {
            self.poller
                .lock()
                .unwrap()
                .poll(&mut events, Some(Duration::from_secs(1)))
                .map_err(VringEpollError::EpollWait)?;

            thread::yield_now();
        }

stefano-garzarella · 2025-07-23T12:44:21Z

@uran0sH looking at the documentation, it seems that Registry has a try_clone() method: https://docs.rs/mio/latest/mio/struct.Registry.html#method.try_clone

So, when you create self.poller, you can clone the registry. In this way in register_event you can use the clone, instead of holding the mutex of the event loop.

WDYT?

uran0sH · 2025-07-25T02:00:19Z

@uran0sH looking at the documentation, it seems that Registry has a try_clone() method: https://docs.rs/mio/latest/mio/struct.Registry.html#method.try_clone

So, when you create self.poller, you can clone the registry. In this way in register_event you can use the clone, instead of holding the mutex of the event loop.

WDYT?

I try it, it can work. And there seems to be no unsafe behavior

vhost-user-backend/src/event_loop.rs

stefano-garzarella · 2025-08-06T10:22:16Z

vhost-user-backend/src/event_loop.rs

-
+pub type VringEpollResult<T> = std::result::Result<T, VringPollError>;
+
+// According https://github.com/tokio-rs/mio/blob/master/src/interest.rs#L27


Please use a permalink, like https://github.com/tokio-rs/mio/blob/cd972977e7a8234e62c7ba8bd4d4ec1da208e576/src/interest.rs#L27

stefano-garzarella · 2025-08-06T10:24:21Z

vhost-user-backend/src/event_loop.rs

-    epoll: Epoll,
+    poller: Mutex<Poll>,
+    registry: Registry,
+    // Record which fd and interest are registered


Please add "why" we need to register them.

stefano-garzarella · 2025-08-06T10:24:30Z

vhost-user-backend/src/event_loop.rs

        thread_id: usize,
    ) -> VringEpollResult<Self> {
-        let epoll = Epoll::new().map_err(VringEpollError::EpollCreateFd)?;
+        // let epoll = Epoll::new().map_err(VringEpollError::EpollCreateFd)?;


stefano-garzarella · 2025-08-06T10:34:35Z

vhost-user-backend/src/event_loop.rs

+                .iter()
+                .position(|(interest, d)| *interest == ev_type && data == *d)
+            {
+                interests.remove(pos);


Should we also call self.registry.deregister() and call again self.registry.register() with the remaining interestes ?

stefano-garzarella · 2025-08-06T10:38:58Z

vhost-user-backend/src/event_loop.rs

+        self.registry
+            .register(&mut SourceFd(&fd), Token(data as usize), ev_type)
+            .map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;


From https://docs.rs/mio/latest/mio/struct.Registry.html#method.register:

Callers must ensure that if a source being registered with a Poll instance was previously registered with that Poll instance, then a call to deregister has already occurred. Consecutive calls to register is unspecified behavior.

So maybe we can't do that, without calling deregister

Maybe we need to call reregister, see https://docs.rs/mio/latest/mio/struct.Registry.html#method.reregister

Re-registering an event source allows changing the details of the registration. Specifically, it allows updating the associated token and interests specified in previous register and reregister calls.

germag · 2025-08-06T14:58:36Z

Today, talking with @stefano-garzarella we agreed that we don't need to use reregister(). We only call register once, otherwise epoll_ctl(..., EPOLL_CTL_ADD, ...) will fail with EEXIST (we also test that test_vring_epoll_handler()).
So, we can also assume when we unregister an FD we do that for all its interests.

So, we don't really need the map, since we don't need to track the registered interest, but since the documentation for mio::Registry::register() says:

Consecutive calls to register is unspecified behavior.

We need to keep track of the already register FD's, so a set should be enough. Something that should make mio::Registry::register() to be marked as unsafe (and so maybe VringEpollHandler::register_listener()) IMO.

I'll look into alternatives (for a more sane API), since on epoll registering twice returns an error instead of UB, and kevent will modify the already registered one.

uran0sH · 2025-08-10T12:18:34Z

Today, talking with @stefano-garzarella we agreed that we don't need to use reregister(). We only call register once, otherwise epoll_ctl(..., EPOLL_CTL_ADD, ...) will fail with EEXIST (we also test that test_vring_epoll_handler()). So, we can also assume when we unregister an FD we do that for all its interests.

So, we don't really need the map, since we don't need to track the registered interest, but since the documentation for mio::Registry::register() says:
Consecutive calls to register is unspecified behavior.
We need to keep track of the already register FD's, so a set should be enough. Something that should make mio::Registry::register() to be marked as unsafe (and so maybe VringEpollHandler::register_listener()) IMO.

I'll look into alternatives (for a more sane API), since on epoll registering twice returns an error instead of UB, and kevent will modify the already registered one.

OK. If I understand correctly, normally we will only register each fd once, if it is registered twice we should throw an exception to the user to tell it that it is registered twice. So when calling deregister we can deregister the entire fd

Bumps [rust-vmm-ci](https://github.com/rust-vmm/rust-vmm-ci) from `af54708` to `0b1cb86`. - [Commits](rust-vmm/rust-vmm-ci@af54708...0b1cb86) --- updated-dependencies: - dependency-name: rust-vmm-ci dependency-version: 0b1cb86353cc093f2e17d7e9f6820de80a6c274d dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

germag

Hi,

I think is in a good shape, just apply my suggestions and send a PR, we are approaching the end of the gsoc, so we need to have the public PR

germag · 2025-08-19T09:42:00Z

vhost-user-backend/src/event_loop.rs

+impl From<VringPollEvent> for Interest {
+    fn from(value: VringPollEvent) -> Self {
+        let mut interest = None;
+        if value == VringPollEvent::READABLE {
+            interest = interest
+                .map(|interest| Interest::READABLE | interest)
+                .or(Some(Interest::READABLE));
+        }
+        if value == VringPollEvent::WRITABLE {
+            interest = interest
+                .map(|interest| Interest::WRITABLE | interest)
+                .or(Some(Interest::WRITABLE));
+        }
+        return interest.expect("Interest is invalid");
+    }
+}


Where do we use this?, I mean, why we need to convert from mio::Interest to VringPollEvent?, if we don't do it, better removing it, remember we want to hide mio::Interest

We need this one. This one is to convert VringPollEvent to Interest. It will be used here: https://github.com/uran0sH/vhost/pull/2/files#diff-e1c9dc7802234bc118de8314ec9cacc98066eea325a6f14ff5887ece3466d5a3R242. User can use like this: https://github.com/uran0sH/vhost/pull/2/files#diff-e1c9dc7802234bc118de8314ec9cacc98066eea325a6f14ff5887ece3466d5a3R358

Forget what I said, I see that mio::Interest is not public anymore, implementing impl From<VringPollEvent> for Interest is Ok

germag · 2025-08-19T09:43:15Z

vhost-user-backend/src/event_loop.rs

+impl From<Interest> for VringPollEvent {
+    fn from(value: Interest) -> Self {
+        let mut event = VringPollEvent::empty();
+        if value.is_readable() {
+            event |= VringPollEvent::READABLE;
+        }
+        if value.is_writable() {
+            event |= VringPollEvent::WRITABLE;
+        }
+        return event;
+    }
+}


doesn't this expose the mio::Interest type?, since VringPollEvent is public, this impl is also public. Maybe just a private function is enough

This one is unecessary. I will delete it.

germag · 2025-08-19T09:46:53Z

vhost-user-backend/src/event_loop.rs


+bitflags! {
+    #[derive(Debug, PartialEq, PartialOrd)]
+    pub struct VringPollEvent: u32 {


VringPollEvent is too verbose, is a type that will be part of the public API of VringEpollHandler, so maybe something like:

EventType

Interest (if you choose this one, please also change the parameters' name ev_type in register_listener() and unregister_listener())

are better

Epoll is linux-specific. So we use mio, which is a cross-platform event notification, to replace Epoll. Signed-off-by: Wenyu Huang <huangwenyuu@outlook.com>

uran0sH · 2025-08-19T13:46:00Z

See rust-vmm#316

uran0sH force-pushed the mio branch 3 times, most recently from b5d2117 to d91521f Compare July 14, 2025 17:15

stefano-garzarella reviewed Jul 15, 2025

View reviewed changes

vhost-user-backend/src/backend.rs Outdated Show resolved Hide resolved

uran0sH force-pushed the mio branch from d91521f to 7fb878d Compare July 15, 2025 16:55

uran0sH commented Jul 15, 2025

View reviewed changes

vhost-user-backend/src/event_loop.rs Outdated Show resolved Hide resolved

uran0sH commented Jul 16, 2025

View reviewed changes

uran0sH force-pushed the mio branch from 7fb878d to cb460e6 Compare July 22, 2025 17:42

uran0sH force-pushed the mio branch 2 times, most recently from 17621b6 to add1eeb Compare July 29, 2025 17:22

uran0sH commented Jul 30, 2025

View reviewed changes

vhost-user-backend/src/event_loop.rs Show resolved Hide resolved

germag reviewed Jul 30, 2025

View reviewed changes

vhost-user-backend/src/event_loop.rs Show resolved Hide resolved

uran0sH force-pushed the mio branch 3 times, most recently from d529720 to ba2d4ee Compare August 4, 2025 16:04

stefano-garzarella reviewed Aug 6, 2025

View reviewed changes

uran0sH force-pushed the mio branch from ba2d4ee to ddd3e12 Compare August 12, 2025 14:35

uran0sH force-pushed the mio branch 2 times, most recently from bd38446 to 478864f Compare August 18, 2025 16:39

germag reviewed Aug 19, 2025

View reviewed changes

Use mio to replace Epoll

edbc0a1

Epoll is linux-specific. So we use mio, which is a cross-platform event notification, to replace Epoll. Signed-off-by: Wenyu Huang <huangwenyuu@outlook.com>

uran0sH force-pushed the mio branch from 478864f to edbc0a1 Compare August 19, 2025 13:44

uran0sH closed this Aug 19, 2025


		pub type VringEpollResult<T> = std::result::Result<T, VringPollError>;

		// According https://github.com/tokio-rs/mio/blob/master/src/interest.rs#L27

Use mio to replace Epoll #2

Use mio to replace Epoll #2

Uh oh!

Conversation

uran0sH commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uran0sH commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uran0sH Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uran0sH commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stefano-garzarella commented Jul 23, 2025

Uh oh!

uran0sH commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

germag commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uran0sH commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

germag left a comment

Choose a reason for hiding this comment

Uh oh!

germag Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uran0sH Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

germag Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

germag Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uran0sH commented Aug 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

uran0sH commented Jul 13, 2025 •

edited

Loading

uran0sH commented Jul 14, 2025 •

edited

Loading

uran0sH Jul 16, 2025 •

edited

Loading

uran0sH commented Jul 23, 2025 •

edited

Loading

germag commented Aug 6, 2025 •

edited

Loading

uran0sH commented Aug 10, 2025 •

edited

Loading

germag Aug 19, 2025 •

edited

Loading

uran0sH Aug 19, 2025 •

edited

Loading

germag Aug 19, 2025 •

edited

Loading

germag Aug 19, 2025 •

edited

Loading