Skip to content

Reduce database pressure#133

Draft
tylerpayne wants to merge 3 commits intomainfrom
reduce-database-pressure
Draft

Reduce database pressure#133
tylerpayne wants to merge 3 commits intomainfrom
reduce-database-pressure

Conversation

@tylerpayne
Copy link
Copy Markdown
Collaborator

@tylerpayne tylerpayne commented Oct 16, 2025

Leaving in draft while I work with Wenyue to test.

Reduce two high-pressure operations:

  1. Recording empty FetchMessages actions
  2. Returning all messages for an agent every time.

To solve 1, the protocol can return a boolean flag indicating whether an executed action should be persisted.

To solve 2, track how many messages each agent fetched last, and use that to indicate "empty" or not.

To test, run these three in order

magentic-marketplace run data/mexican_3_9
magentic-marketplace run data/mexican_3_9 --drop-empty-fetch-messages 
magentic-marketplace run data/mexican_3_9 --drop-all-fetch-messages

Then run

magentic-marketplace list --limit 3

And you should see the first has the least actions, then the second, then third has the most.


PR Checklist (do not remove):

  • I've added necessary new tests and they pass
  • My PR description explains how to test this contribution
  • I have linked this PR to an issue
  • I have requested reviews from two people

@tylerpayne tylerpayne requested a review from Wenyueh October 16, 2025 15:21
@willeppy
Copy link
Copy Markdown
Collaborator

My vote would be to just merge the change around not writing fetches since seems less disruptive (under a flag). Default behavior is same as now and then you can set flag to not write fetches

@Wenyueh
Copy link
Copy Markdown
Contributor

Wenyueh commented Oct 16, 2025

experiments on it work perfectly

Tyler Payne added 2 commits October 16, 2025 16:59
This lets the protocol avoid writing noisy actions to the database

Empty FetchMessages actions are not persisted to the databse
@tylerpayne tylerpayne force-pushed the reduce-database-pressure branch from 6406f8e to b426e05 Compare October 16, 2025 21:44
@tylerpayne
Copy link
Copy Markdown
Collaborator Author

@willeppy I know we talked about removing the NOT IN expression and only keeping the drop-empty-fetch logic. But without the NOT IN logic a fetch is never empty 😖

So I've added logic to the SimpleMarketplaceProtocol to track how many messages each agent fetched last, and using that to indicate if a FetchMessages action is "empty" or not. To reduce impact on our message passing logic, I still opted to send all messages to the agents and let the agents filter them out as they were before.

@tylerpayne tylerpayne marked this pull request as ready for review October 16, 2025 21:46
@tylerpayne tylerpayne requested a review from willeppy October 16, 2025 21:48
@tylerpayne tylerpayne marked this pull request as draft October 17, 2025 18:15
@willeppy
Copy link
Copy Markdown
Collaborator

@tylerpayne non-urgent but think would still be good to merge this soon to make dbs smaller, esp since we've already tagged an arxiv release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants