Skip to content

Add streaming-in-data implementations #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
armaan-25 wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add streaming-in-data implementations #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b97f8e1
Add streaming brute-force model selector
armaan-25 Mar 21, 2026
7df8255
Add convergence stop policy for streaming selection
armaan-25 Mar 22, 2026
a194f86
Add fake-data streaming script and tests
armaan-25 Mar 22, 2026
2b5558f
Merge main into streaming-in-data and resolve README conflicts
armaan-25 Mar 24, 2026
81be75d
Add streaming random search model selector
armaan-25 Mar 24, 2026
e32eafe
Update src/agentopt/model_selection/streaming_brute_force.py
armaan-25 Mar 25, 2026
ec7b010
Update src/agentopt/model_selection/streaming_brute_force.py
armaan-25 Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ If you do not need the strict best model combination and want **more evaluation
| `"auto"` (default) | General use | Automatically finds the best combination (wired to `arm_elimination` — strong best-arm identification with lower evaluation cost than `brute_force`) |
| `"brute_force"` | Small search spaces | Evaluates all combinations |
| `"random"` | Quick exploration | Samples a random fraction |
| `"streaming_random"` | Streaming / online updates | Samples once, then incrementally updates the sampled combos on new batches |
| `"hill_climbing"` | Topology-aware search | Greedy search using model quality/speed rankings |
| `"arm_elimination"` | Best-arm identification | Bandit; eliminates statistically dominated combinations |
| `"epsilon_lucb"` | Extra cost savings when ε-optimal is enough | Bandit; stops when an epsilon-optimal best arm is identified |
Expand Down Expand Up @@ -261,6 +262,23 @@ selector = ModelSelector(
)
```

**Streaming incoming data** — use `StreamingBruteForceModelSelector` to update rankings incrementally as labeled batches arrive:

```python
from agentopt import StreamingBruteForceModelSelector

selector = StreamingBruteForceModelSelector(
agent_fn=agent_maker,
models=models,
eval_fn=eval_fn,
dataset=warm_start_dataset, # required seed dataset
)
Comment on lines +265 to +275
Copy link

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StreamingBruteForceModelSelector is used here with an agent_fn argument, but selectors in this repo (and the docs in docs/api/selectors.md) use an agent class interface (__init__(models), run(input_data)). As implemented, the selector will also fail to initialize because BaseModelSelector doesn’t accept agent_fn. Consider updating the README example to the supported agent interface (or, if you intentionally add an agent_fn API, document it consistently and implement it in the base selector).

Copilot uses AI. Check for mistakes.

for batch in stream_of_labeled_batches:
selector.update(batch) # batch: Sequence[(input_data, expected_answer)]
print("Current best:", selector.best_combo())
```

## Documentation

Full documentation at **[agentoptimizer.github.io/agentopt](https://agentoptimizer.github.io/agentopt/)** — including detailed guides on the [Results API](https://agentoptimizer.github.io/agentopt/api/results/), [response caching](https://agentoptimizer.github.io/agentopt/concepts/caching/), and [custom model pricing](https://agentoptimizer.github.io/agentopt/api/selectors/).
Expand Down
5 changes: 5 additions & 0 deletions docs/api/selectors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ Returns a [`SelectionResults`](results.md) object.
members: false
show_bases: false

::: agentopt.model_selection.streaming_random_search.StreamingRandomSearchModelSelector
options:
members: false
show_bases: false

Copy link

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StreamingBruteForceModelSelector is exported publicly (in agentopt.model_selection.__init__ and agentopt.__init__) but isn’t listed on this selectors API page. Either add a corresponding ::: agentopt.model_selection.streaming_brute_force.StreamingBruteForceModelSelector entry here, or avoid exporting it as part of the public API until it’s documented.

Suggested change
::: agentopt.model_selection.streaming_brute_force.StreamingBruteForceModelSelector
options:
members: false
show_bases: false

Copilot uses AI. Check for mistakes.
::: agentopt.model_selection.hill_climbing.HillClimbingModelSelector
options:
members: false
Expand Down
35 changes: 34 additions & 1 deletion docs/concepts/algorithms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# Selection Algorithms

AgentOpt provides 8 selection algorithms. Choose based on your search space size and evaluation budget.
AgentOpt provides 9 selection algorithms. Choose based on your search space size and evaluation budget.

## At a Glance

| Algorithm | Strategy | Evaluations | Best For |
|:----------|:---------|:------------|:---------|
| [Brute Force](#brute-force) | Exhaustive | All | Small spaces (< 50 combos) |
| [Random Search](#random-search) | Sampling | Configurable fraction | Quick baselines |
| [Streaming Random Search](#streaming-random-search) | Streaming sampling | Incremental | Online / incoming batches |
| [Hill Climbing](#hill-climbing) | Greedy + restarts | Guided neighbors | Medium spaces |
Comment on lines +3 to 12
Copy link

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guide now says “AgentOpt provides 9 selection algorithms”, but this PR introduces two new selectors (StreamingRandomSearchModelSelector and StreamingBruteForceModelSelector). Either document both streaming algorithms here (and update the count/table accordingly), or explicitly clarify that streaming brute-force is considered part of “Brute Force” rather than a separate algorithm (and add a note/section so users can discover it).

Copilot uses AI. Check for mistakes.
| [Arm Elimination](#arm-elimination) | Progressive pruning | Adaptive | Statistical early stopping |
| [Epsilon LUCB](#epsilon-lucb) | ε-optimal LUCB | Adaptive | Cost savings when ε-optimal is enough |
Expand Down Expand Up @@ -80,6 +81,38 @@ selector = RandomSearchModelSelector(

---

## Streaming Random Search

Samples a random subset once, then incrementally updates those sampled combinations as new batches arrive.

```python
from agentopt import StreamingRandomSearchModelSelector

selector = StreamingRandomSearchModelSelector(
agent=MyAgent,
models=models,
eval_fn=eval_fn,
dataset=warm_start_dataset, # seed batch
sample_fraction=0.25,
seed=42,
)

selector.select_best() # evaluate warm start once
selector.update(stream_batch_1) # keep updating online
Comment on lines +95 to +101
Copy link

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example uses selector.select_best() and then calls selector.update(...). Note that select_best() (via BaseModelSelector) stops the tracker on return, so subsequent streaming updates won’t record token usage / cache hits unless the selector restarts the tracker. Either adjust the example to warm-start via update(warm_start_dataset) instead, or update the streaming selector implementation to keep tracking active across select_best() + update().

Suggested change
dataset=warm_start_dataset, # seed batch
sample_fraction=0.25,
seed=42,
)
selector.select_best() # evaluate warm start once
selector.update(stream_batch_1) # keep updating online
sample_fraction=0.25,
seed=42,
)
# warm-start with an initial batch
selector.update(warm_start_dataset)
# then keep updating online as new batches arrive
selector.update(stream_batch_1)

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator

@Wenyueh Wenyueh Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@armaan-25

selector.update(stream_batch_2)
best_combo = selector.best_combo()
```

| Parameter | Default | Description |
|:----------|:--------|:------------|
| `sample_fraction` | `0.25` | Fraction of combinations sampled once and reused |
| `seed` | `None` | Random seed for reproducible sampling |

!!! success "When to use"
Streaming / online settings where data arrives over time and you want incremental updates instead of full reselection each round.

---

## Hill Climbing

Greedy local search with random restarts. Defines "neighbors" using model quality and speed rankings, so each step is an informed single-model swap.
Expand Down
6 changes: 6 additions & 0 deletions src/agentopt/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@
DatapointResult,
ModelResult,
RandomSearchModelSelector,
StreamingRandomSearchModelSelector,
SelectionResults,
StreamingBruteForceModelSelector,
ThresholdBanditSEModelSelector,
)

Expand All @@ -35,6 +37,7 @@
"auto": ArmEliminationModelSelector,
"brute_force": BruteForceModelSelector,
"random": RandomSearchModelSelector,
"streaming_random": StreamingRandomSearchModelSelector,
"hill_climbing": HillClimbingModelSelector,
"arm_elimination": ArmEliminationModelSelector,
"epsilon_lucb": EpsilonLUCBModelSelector,
Expand Down Expand Up @@ -64,6 +67,7 @@ def ModelSelector(
``"brute_force"``). Other options: ``"brute_force"``,
``"random"``, ``"hill_climbing"``, ``"arm_elimination"``,
``"epsilon_lucb"``, ``"threshold"``, ``"lm_proposal"``,
``"streaming_random"``,
``"bayesian"``.
**kwargs: Additional arguments passed to the selector
(e.g. ``epsilon``, ``threshold``, ``sample_fraction``).
Expand Down Expand Up @@ -95,10 +99,12 @@ def ModelSelector(
# Selectors
"BruteForceModelSelector",
"RandomSearchModelSelector",
"StreamingRandomSearchModelSelector",
"HillClimbingModelSelector",
"ArmEliminationModelSelector",
"EpsilonLUCBModelSelector",
"ThresholdBanditSEModelSelector",
"StreamingBruteForceModelSelector",
"LMProposalModelSelector",
"BayesianOptimizationModelSelector",
# Result types
Expand Down
4 changes: 4 additions & 0 deletions src/agentopt/model_selection/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
from .hill_climbing import HillClimbingModelSelector
from .lm_proposal import LMProposalModelSelector
from .random_search import RandomSearchModelSelector
from .streaming_brute_force import StreamingBruteForceModelSelector
from .streaming_random_search import StreamingRandomSearchModelSelector
from .threshold_successive_elimination import ThresholdBanditSEModelSelector

# Bayesian is optional (requires torch/botorch)
Expand All @@ -24,6 +26,8 @@
"EpsilonLUCBModelSelector",
"ThresholdBanditSEModelSelector",
"LMProposalModelSelector",
"StreamingBruteForceModelSelector",
"StreamingRandomSearchModelSelector",
"BayesianOptimizationModelSelector",
"DatapointResult",
"ModelResult",
Expand Down
Loading