-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
🤔 What is the technical debt you think should be addressed?
Problem
There are a number of tests in the Responses API test suite that make API calls to sites like Wikipedia to download images for use with vision model tests. This especially impacts the Ollama suite, which can be a good starting point for debugging. There are two problems with this:
- The images at the link may change, resulting in a different outcome in our tests.
- We are bombarding sites like Wikipedia with API requests whenever our test suite runs, and we get rate limited. Its also just rude of us.
Solution Options
Download once and cache for the duration of the tests
If we are ok with downloading the images and risking the links breaking or the content changing, this is a suitable option, and there may be a supported path to do this already in CI.
Store the images as test fixtures
This is a tried and true method, and its hard to go wrong with it. The only downside is that those images will be stored in this repo, but only in the tests fixture section, so it should not bloat production installs.
💡 What is the benefit of addressing this technical debt?
This will Reduce CI flakes, and make us a better citizen of the internet in general.
Other thoughts
Good First Issue