-
Notifications
You must be signed in to change notification settings - Fork 23
Description
Using the fetch API, I am triggering a generation request and I am making sure that it has streaming enabled and usage enabled. And when I provide an abort controller to be able to cancel the stream, I expect the usage to still come at the moment of cancellation. But this is not what I get. Instead the connection just terminates and I get no usage chunks, or any data of any kind.
How am I suppose to be able to support stopping of generation and usage? Is there some other api end point I am supposed to call to get the usage of the generation? I need the usage because this is how I can calculate how much to bill the user.
The main core of the request is here showing that I am requesting for streaming, usage, and providing an abort controller.
const requestBody = {
model,
messages,
stream: true,
stream_options: {
include_usage: true,
}
};
// Create AbortController for OpenRouter request cancellation
const openRouterAbortController = new AbortController();
const fetchOptions = {
method: "POST",
headers,
body: JSON.stringify(requestBody),
signal: openRouterAbortController.signal,
};
let response;
try {
response = await fetch(openRouterUrl, fetchOptions);
} catch (fetchError) {
...
}The expected result would be to call the abort controller and the stream of chunks ending by including a final finish reason chunk, usage chunk, and that [DONE] message.