[wasi-sockets] Fresh UDP sockets occasionally return `ErrorCode::WouldBlock` on first send

[dicej]

I'm (still) working to add WASIp2 support to mio and have noticed that several of the UDP tests usually pass but sometimes fail. When they fail, it's always due to the host returning error-code::would-block from outgoing-datagram-stream.send (which wasi-libc translates to returning EWOULDBLOCK from send(2)). This can happen even if the socket was just created, has not previously been used to send anything, and for which outgoing-datagram-stream.check-send has returned a non-zero number. It can also happen when poll.poll has just returned a write-ready event for the pollable representing the outgoing-datagram-stream.

I've simplified the test case somewhat here:

// udp.rs
use std::{error::Error, io::ErrorKind, net::UdpSocket};

fn main() -> Result<(), Box<dyn Error>> {
    for _ in 0..50 {
        let socket1 = UdpSocket::bind("127.0.0.1:0")?;
        socket1.set_nonblocking(true)?;
        let socket2 = UdpSocket::bind("127.0.0.1:0")?;
        socket2.set_nonblocking(true)?;

        let address1 = socket1.local_addr()?;
        let address2 = socket2.local_addr()?;

        socket1.connect(address2)?;
        socket2.connect(address1)?;

        let data = b"foobar";
        socket1.send(data)?;

        let mut buffer = [0u8; 20];
        match socket2.recv(&mut buffer) {
            Ok(n) => {
                assert_eq!(data, &buffer[..n]);
            }
            Err(error) => {
                assert!(matches!(error.kind(), ErrorKind::WouldBlock));
            }
        }
    }

    println!("success!");

    Ok(())
}

If you run e.g. rustc --target=wasm32-wasip2 udp.rs && while wasmtime run -Sudp,inherit-network udp.wasm; do :; done, sooner or later you'll see Error: Os { code: 6, kind: WouldBlock, message: "Resource temporarily unavailable" }. However, if you run e.g. rustc udp.rs && while ./udp; do :; done on Linux, it will succeed indefinitely.

When I dug into this, I found that wasmtime-wasi uses tokio::net::UdpSocket::try_send[_to] to send datagrams, which eventually consults this function from here to determine whether to even bother trying to send. Indeed, when I add debug logging to that function and patch Wasmtime to use it, I see that the "not ready" case corresponds to when the host returns would-block to the guest, which indicates that tokio is applying backpressure rather than the OS.

I imagine that in most real-world code, returning spurious would-block errors occasionally wouldn't be a problem; the application can just poll and try again. However, it makes the mio tests flaky, so I'd like to brainstorm solutions here.

One rather heavy-handed approach could be to bypass Tokio for sending and receiving datagrams, using tokio::net::UdpSocket::{into,from}_std to temporarily convert between the Tokio version and the std version. I imagine that could have other implications, though.

[dicej]

@badeend your reward for being so helpful with the last issue is that I'm going to tag you on this one also 🥇

[alexcrichton]

My understanding of what's happening here is that this is a race within Tokio. Tokio guards all I/O with ensuring that readiness has actually been signaled before actually doing the I/O, and it looks like all I/O readiness starts out as empty. On most iterations of the test by the time the send is executed the readiness has already been updated to indicate that the socket is writable -- there's a background thread doing epoll_wait which is woken up immediately after the epoll_ctl adding the UDP socket to the set. In some iterations, though, this background thread isn't fast enough so the UDP socket's own internal readiness still says "not ready" so the I/O is blocked.

The call-stack is:

• UdpSocket::try_send calls try_io
• try_io sees readiness is empty and returns without actually doing anything

One fix is to wait for writable-ness in the test itself which would resolve this. Whether that's acceptable, I'm not entirely sure. Short of that I'm not sure how best to fix this. Reaching inside and performing operations manually runs the risk of bypassing Tokio's tracking of the readiness state, which can maybe still work but would require some finesse.

[dicej]

One fix is to wait for writable-ness in the test itself which would resolve this.

Where would that happen? This mio test calls send on a fresh socket without polling it for writability first, which limits our options. I suppose we could kick it off in udp-socket.start-bind and only return success from udp-socket.finish-bind once it's ready.

[badeend]

Every I/O operation is allowed to return would-block at any time, even if the socket was just created or a previous poll indicated readiness. So strictly speaking, the mio test is somewhat naive. That said, since it works as expected on other platforms, this is likely an edge case that mio just do happens to surface first, but they probably won't be the last. Test suites in specific are often less defensive than production code.

If this is easily fixable on our end I think we should.

---

after the epoll_ctl adding the UDP socket to the set

Do you know when does this happens? Is it at creation time? If so, maybe we can update our UdpSocket::new method to await readiness immediately E.g.

let socket = with_ambient_tokio_runtime(|| {
    tokio::net::UdpSocket::try_from(unsafe {
        std::net::UdpSocket::from_raw_socketlike(fd.into_raw_socketlike())
    })
})?;

socket.readable().await?;
socket.writable().await?;

IIUC, that should usually not actually block, correct?