fix: time delay for message buffer flushing with iroh QUIC streams

[samherring99]

We currently have an adaptive delay set up in shared/src/protocol_handler.rs to account for the message buffer flushing fully to return inference responses to the gateway.

Apparently, the correct way to handle this is to have the receiving peer close the connection after it reads the message, but I couldn't get this to work. Pasting the line of code below, and the messages from iroh team.

Code (lines 65-75 in protocl_handler.rs):

// adaptive delay to ensure data is flushed before connection is dropped
// without this, the connection might close before the peer reads all bytes
// base 50ms + 10ms per MB of data
let size_mb = response_bytes.len() as f64 / (1024.0 * 1024.0);
let delay_ms = 50 + (size_mb * 10.0) as u64;
debug!(
    "Waiting {}ms for {} bytes to flush",
    delay_ms,
    response_bytes.len()
);
tokio::time::sleep(tokio::time::Duration::from_millis(delay_ms)).await;

iroh team responses:

Let me see if I understood you correctly:
endpoint A: has an open bi-directional stream and is sending bytes on it.
A: sent last byte and calls SendStream::finish().
A: calls Connection::close()
B: does not receive all bytes and/or the FIN.
The way you need to think about this is that the last peer to receive data should be the one closing the connection:
A: sends bytes & calls SendStream::finish()
A: waits for the peer to close the connection by calling Connection::closed() (needs awaiting)
B: reads until it gets the FIN (RecvStream::read_to_end or RecvStream::received_reset are common ways, both need awaiting)
B: closes the connection: Connection::close(0u8.into(), b"bye")
And that should ensure you never lose data.

and

Anytime you call conn.close() that'll interrupt all work on the connection (e.g. any data that still needs to be sent). send.finish() will only tell the connection to queue a stream frame with FIN set (to indicate the stream ends), but it won't wait for the stream to be fully sent.

I recently wrote a thread on bluesky about specifically this issue where you don't want the other side to call close, because you want to keep the connection open potentially. The solution is to send back another message that basically indicates "ok, I'm done with receiving, you can close if you want to", and you can implement that with just the stream finish mechanism. There are more details in the bluesky thread:
https://bsky.app/profile/matheus23.com/post/3mcysyj5tvc23

Will make this change in an upcoming PR after #487 is merged