Skip to content

Prevent High Frequency Image Downloads in Response API Integration Tests #5322

@iamemilio

Description

@iamemilio

🤔 What is the technical debt you think should be addressed?

Problem

There are a number of tests in the Responses API test suite that make API calls to sites like Wikipedia to download images for use with vision model tests. This especially impacts the Ollama suite, which can be a good starting point for debugging. There are two problems with this:

  1. The images at the link may change, resulting in a different outcome in our tests.
  2. We are bombarding sites like Wikipedia with API requests whenever our test suite runs, and we get rate limited. Its also just rude of us.

Solution Options

Download once and cache for the duration of the tests

If we are ok with downloading the images and risking the links breaking or the content changing, this is a suitable option, and there may be a supported path to do this already in CI.

Store the images as test fixtures

This is a tried and true method, and its hard to go wrong with it. The only downside is that those images will be stored in this repo, but only in the tests fixture section, so it should not bloat production installs.

💡 What is the benefit of addressing this technical debt?

This will Reduce CI flakes, and make us a better citizen of the internet in general.

Other thoughts

Good First Issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions