Build fails on WASM as reqwest::blocking is not available

[coolbluewater]

On wasm targets only async calls are available.
As a result, reqwest::blocking::Client is not available.
reqwest::Client (the async version of the client) should be used instead.

To support this there are two choices:

1. Expose blocking as well as non-blocking APIs for the llm crate, with only the latter available on WASM.
2. Breaking change: make all the API's non-blocking, thus exposing a single async surface.

Personally I'm in favor of the latter as there isn't a good reason to have these calls block.

[graniet]

Thank you for your issue @coolbluewater . I have just merged a contributor’s pull request (@ernestas-poskus ), which should fix your problem!

[coolbluewater]

Wow, nice! I will give it a try, closing this issue in the meantime.

[coolbluewater]

Hmm, still seems broken:

cargo build --target wasm32-unknown-unknown --features=ollama

results in the same errors as before.

[ernestas-poskus]

Check this issue it might have some clues seanmonstar/reqwest#288

[coolbluewater]

Yes, that's the underlying problem. FWIW the ollama-rs crate does compile and run fine in webassembly builds (as the name implies it supports ollama only, but that's not relevant here.) Not sure what http client they are using but it's worth checking out.

[reneleonhardt]

It uses the latest reqwest::Client, maybe async is working? 😉
https://github.com/pepperoni21/ollama-rs/blob/HEAD/ollama-rs/src/lib.rs#L86

[vemonet]

Not sure why ollama-rs is mentioned but the issue seems to come straight from the llm code:

    fn mark_for_summary(&mut self) {
        tokio::task::block_in_place(|| {
            tokio::runtime::Handle::current().block_on(async {
                let mut guard = self.inner.write().await;
                guard.mark_for_summary();
            })
        })
    }

    fn replace_with_summary(&mut self, summary: String) {
        tokio::task::block_in_place(|| {
            tokio::runtime::Handle::current().block_on(async {
                let mut guard = self.inner.write().await;
                guard.replace_with_summary(summary);
            })
        })
    }

tokio::spawn(async move {

From my experience reqwest::Client is not blocking by default, here is an example of use inside an app that is compiled to wasm and works all fine in browser or nodejs: https://github.com/vemonet/nanopub-rs/blob/main/lib/src/network.rs#L26

Here is a simple explanation of my understanding of the issue with runtimes (like tokio) in wasm (this was written for a lib that had 3 components: rust core, python and JS bindings, but fits well the current question):

• If possible you should completely avoid using the tokio runtime in the core and wasm libs, apart for dev testing, and make everything async/await when needed (if one of the function you use is async, e.g. send a non-blocking HTTP reqwest, then the function calling it needs to be async too, etc). The browser has its own runtime, and tokio conflicts with it. It is not that hard and probably a good practice on the long run to avoid using runtimes, and properly work with async/await + it gives you more control over the systen and it avoids entering dependency hell when using a library like tokio-with-wasm
  • You should only use tokio to await in the CLI and maybe the python lib, because these 2 don't come with a runtime to do await so you need to bring tokio in
  • In the python lib you could even avoid tokio by letting the user do await themselves, but python users are used to have non-async function so that's usually nice to give them this option
  • It can be a bit tricky if your dependencies already uses tokio, but the most sustainable would be to remove them in this case
  • Overall using async/await is a better API choice, gives better control to the dev using your lib

So ideally, the core of llm should be tokio free to be used in wasm. Just do async/await all the way through!

We could still have it as a feature for when we need tokio (e.g. in the CLI you'll need it, so you can do the await on the top function called from the CLI)