Skip to content

Move request formatting to backend #478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
sjmonson wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Move request formatting to backend #478

sjmonson wants to merge 17 commits into from
+2,250 −2,048

Conversation

@sjmonson
Copy link
Collaborator

@sjmonson sjmonson commented Nov 21, 2025 •
edited
Loading

Summary

Splits the RequestFormatter into multiple separate components across data loading, data preprocessing, and the backend. Necessary step for next attempt at multi-turn.

Details

  • Removed the RequestFormatter preprocessor
  • Added "Encode" preprocessors for audio, video, and image
  • Added a "Finalizer" stage after preprocessors to convert from column ordered data to request object
  • Added most of previous formatter code to backend

Test Plan

  • Run all know benchmark well-lit pathways

Related Issues

  • Resolves #
  • "I certify that all code in this PR is my own, except as noted below."

Use of AI

  • Includes AI-assisted code completion
  • Includes code generated by an AI application
  • Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

@sjmonson sjmonson force-pushed the feat/split_formatter branch from 1e80237 to 687f702 Compare November 21, 2025 23:11
@sjmonson sjmonson force-pushed the feat/split_formatter branch from 8dbc7f5 to cef2e60 Compare December 9, 2025 18:23
@sjmonson sjmonson mentioned this pull request Dec 15, 2025
Open
26 tasks
@sjmonson sjmonson force-pushed the feat/split_formatter branch 2 times, most recently from c98234b to 28ece45 Compare January 26, 2026 16:43
@sjmonson
Copy link
Collaborator Author

sjmonson commented Jan 26, 2026 •
edited
Loading

A note on --request-type: Request type only really makes sense in the context of the openai_http backend. For upcoming backends (such as #178 and #548) I imagine request_type will be completely ignored. Ideally we would not give openai_http any special treatment and require request_type to either be specified as a --backend-kwarg (basically how its implemented now without the special --request-type shorthand) or as entirely separate backend for each request-type. For now I have kept --request-type to avoid user confusion.

@sjmonson sjmonson marked this pull request as ready for review January 27, 2026 20:18
jaredoconnell
jaredoconnell reviewed Jan 27, 2026
View reviewed changes
@sjmonson sjmonson force-pushed the feat/split_formatter branch from 01c2dce to dbbd18a Compare January 28, 2026 18:20
jaredoconnell
jaredoconnell reviewed Jan 28, 2026
View reviewed changes
Copy link
Collaborator

@jaredoconnell jaredoconnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far. I tested the default paths with a simple local benchmark, and that didn't break anything.

I have a few minor comments.

I'll give a final approving review once the tests are passing.

@sjmonson sjmonson requested a review from markurtz January 29, 2026 15:15
@sjmonson sjmonson changed the title Split up RequestFormatter Move request formatting to backend Jan 29, 2026
sjmonson added 14 commits January 29, 2026 14:25
@sjmonson
Split formatter across multiple stages
66bdbf8 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Rework entrypoint to match new preprocess and finalizer flow
dce6450 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Migrate remaining code off of requests.arguments
d140157 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Fix naming for finalizer registry
0f313d7 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Move OpenAI backend to subdir
5b04866 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Move request formatting to separate file
c3be4cf 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Move request_type handling to backend
a6cae96 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Ignore args on backend failure
ab89db5 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Pass all columns to backend
8056d0d 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Move openai backend to openai/http.py
dc54184 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Ensure model name is always set
ea085b6 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Rename request-type to request-format
941f7bd 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Unify request_formatter and response_handler
3e110e7 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Move legacy API name handling to backend
c8f1edb 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson sjmonson force-pushed the feat/split_formatter branch 2 times, most recently from ac0c1e0 to 1ecb2a1 Compare January 29, 2026 19:35
@sjmonson
Add parser for json lists
2291611 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Improve various builtin docs
9b38c53 
Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson
Update tests
3f05211 
Fix schema tests

Drop unused backend mock

Move openai tests to match new structure

Rewrite OpenAI backend tests

Drop HuggingFace test that relies on network

Fix failing backend tests

Remove remaining references to request_type in tests

Unify request_formatter and response_handler tests

More unit test fixes

Signed-off-by: Samuel Monson <smonson@redhat.com>
@sjmonson sjmonson force-pushed the feat/split_formatter branch from 1ecb2a1 to 3f05211 Compare January 29, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@markurtz markurtz Awaiting requested review from markurtz

@jaredoconnell jaredoconnell Awaiting requested review from jaredoconnell

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sjmonson @jaredoconnell