Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,6 @@
<a href="https://github.com/convergence-ai/proxy-lite/issues/">
<img src="https://img.shields.io/github/issues/convergence-ai/proxy-lite" alt="open issues" />
</a>
<a href="https://github.com/convergence-ai/proxy-lite/blob/master/LICENSE">
<img src="https://img.shields.io/github/license/convergence-ai/proxy-lite.svg" alt="license" />
</a>
</p>

</div>
Expand Down Expand Up @@ -159,7 +156,7 @@ result = asyncio.run(
The `Runner` sets the solver and environment off in a loop, like in a traditional reinforcement learning setup.

<div align="center">
<img src="assets/loop.png" alt="Runner Loop" width="700" height="auto" style="margin-bottom: 20px;" />
<img src="assets/loop.png" alt="Runner Loop" width="800" height="auto" style="margin-bottom: 20px;" />
</div>


Expand All @@ -186,9 +183,9 @@ message_history = [
```
This would then build up the message history, alternating between the assistant (who takes the *action*) and the user (who provides the *observation*).

> **Context-Window Management:** When making calls to the model, all the last observations other than the current one are discarded in order to reduce the large number of image tokens required. Since the model responses include reflection on the observations and are all included in the message history, the model is still aware of the entire history when planning new actions.
> **Context-Window Management:** When making calls to the model, all the observations other than the current one are discarded in order to reduce the large number of image tokens required. Since the model responses include reflection on the observations and are all included in the message history, the model is still aware of the entire history when planning new actions.

The chat template will format this automatically. You should also pass the `Tools` that the model has access to, these will define the action space available to the model. You can do this with `transformers`:
You should also pass the `Tools` that the model has access to, these will define the action space available to the model. You can do this with `transformers`:

```python
from qwen_vl_utils import process_vision_info
Expand Down
Binary file modified assets/loop.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 7 additions & 3 deletions src/proxy_lite/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ def update_config_from_env(config: RunnerConfig) -> RunnerConfig:
config.solver.client.api_base = os.getenv("PROXY_LITE_API_BASE")
if os.getenv("PROXY_LITE_MODEL"):
config.solver.client.model_id = os.getenv("PROXY_LITE_MODEL")
if os.getenv("PROXY_LITE_VIEWPORT_WIDTH"):
config.environment.viewport_width = int(os.getenv("PROXY_LITE_VIEWPORT_WIDTH"))
if os.getenv("PROXY_LITE_VIEWPORT_HEIGHT"):
config.environment.viewport_height = int(os.getenv("PROXY_LITE_VIEWPORT_HEIGHT"))
return config


Expand All @@ -31,11 +35,11 @@ def do_command(args):
if args.model:
config.solver.client.model_id = args.model
if args.homepage:
config.homepage = args.homepage
config.environment.homepage = args.homepage
if args.viewport_width:
config.viewport_width = args.viewport_width
config.environment.viewport_width = args.viewport_width
if args.viewport_height:
config.viewport_height = args.viewport_height
config.environment.viewport_height = args.viewport_height
o = Runner(config=config)
result = asyncio.run(o.run(do_text))

Expand Down