finance offline reproduction: playbook grows >100K tokens; are we supposed to use --use_bulletpoint_analyzer?

[meaten]

Hi, thanks for the project!

I’m reproducing the finance example (task_name=finer, mode=offline). The intermediate playbook/bulletpoints grow very quickly — >100K tokens before 300 examples — and then LLM calls start failing with rate limiting (likely due to prompt size with the entire playbook).

I also noticed curator doesn’t seem to support merging/deleting bulletpoints, so the playbook only grows.

Commands

Without Bulletpoint Analyzer (README):

python -m finance.run --task_name finer --mode offline --save_path results

With Bulletpoint Analyzer:

python -m finance.run --task_name finer --mode offline --save_path results --use_bulletpoint_analyzer

Questions

1. Is --use_bulletpoint_analyzer required to reproduce the reported finance results? (README doesn’t include it.)
2. Is this rapid playbook growth expected, or a bug/misconfig?
3. Curator ignores --playbook_token_budget. Are there recommended settings/strategies to bound playbook size (max bulletpoints, summarization, truncation, etc.)?