Skip to content

[Bug]: How to make stream cancellation provide usage chunk #149

@SpicyMelonYT

Description

@SpicyMelonYT

Using the fetch API, I am triggering a generation request and I am making sure that it has streaming enabled and usage enabled. And when I provide an abort controller to be able to cancel the stream, I expect the usage to still come at the moment of cancellation. But this is not what I get. Instead the connection just terminates and I get no usage chunks, or any data of any kind.

How am I suppose to be able to support stopping of generation and usage? Is there some other api end point I am supposed to call to get the usage of the generation? I need the usage because this is how I can calculate how much to bill the user.

The main core of the request is here showing that I am requesting for streaming, usage, and providing an abort controller.

const requestBody = {
  model,
  messages,
  stream: true,
  stream_options: {
    include_usage: true,
  }
};

// Create AbortController for OpenRouter request cancellation
const openRouterAbortController = new AbortController();

const fetchOptions = {
  method: "POST",
  headers,
  body: JSON.stringify(requestBody),
  signal: openRouterAbortController.signal,
};

let response;
try {
  response = await fetch(openRouterUrl, fetchOptions);
} catch (fetchError) {
  ...
}

The expected result would be to call the abort controller and the stream of chunks ending by including a final finish reason chunk, usage chunk, and that [DONE] message.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions