Skip to content

move "batch size estimation" biz-logic from SD to Exec#19011

Open
AskAlexSharov wants to merge 8 commits intomainfrom
alex/move_estimate_size_logic_34
Open

move "batch size estimation" biz-logic from SD to Exec#19011
AskAlexSharov wants to merge 8 commits intomainfrom
alex/move_estimate_size_logic_34

Conversation

@AskAlexSharov
Copy link
Collaborator

reason:

  • it creating chicken-egg problem. Can't print in logs size before commitment, then print after commitment - and understand how much data produced commitment. Because "print after" will show over-estimated size.
  • SD is now having less and less biz-logic. So, EstimateSize it's clearly abstraction-leak

Before this change i see:

buf=512M before commitment
buf=9G after commitment

but it's not because Commitment producing so much data
it's because we "double-estimate": using estimation logic which "predicting RAM usage after Commitment - before commitment", use same prediction "after Commitment" is incorrect - because "we just know how much RAM we using now".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants