feat(pglite/sync): Add commitGranularity parameter in the syncShapeToTable

[samwillis]

Adds three additional options to syncShapeToTable:

• commitGranularity: CommitGranularity
  
  The granularity of the commit operation, defaults to "up-to-date". Note that a commit will always be performed immediately on the up-to-date message.
  Options:

  • "up-to-date": Commit all messages when the up-to-date message is received.
  • "transaction": Commit all messages within transactions as they were applied to the source Postgres.
    Note: Remove transaction commit granularity until Electric has stabilised on LSN metadata
  • "operation": Commit each message in its own transaction.
  • number: Commit every N messages.

• commitThrottle: number
  
  The number of milliseconds to wait between commits, defaults to 0.

• onInitialSync: () => void
  
  A callback that is called when the initial sync is complete.

[github-actions]

Built bundles:

• Postgres WASM: https://github.com/electric-sql/pglite/actions/runs/12232609354/artifacts/2292659257
• Interim build files: https://github.com/electric-sql/pglite/actions/runs/12232609354/artifacts/2292659420
• PGlite: https://github.com/electric-sql/pglite/actions/runs/12232609354/artifacts/2292688234
• Demos: https://github.com/electric-sql/pglite/actions/runs/12232609354/artifacts/2292707729

[github-actions]

🚀 Deployed on https://6756b9f6857134c3e6bf330f--pglite.netlify.app

[msfstef]

Looks good, I like the API! Left some comments on implementation

[msfstef]

I don't like that this throttler might not run the last received commit - perhaps add a utility with a more full fledged throttler or use a utility like lodash's? Ideally the last commit call should always execute after the throttle period and not have to wait for one more if it was not run due to throttling.

edit: I realise that the behaviour I describe would introduce complexity in awaiting things, handling errors, etc - perhaps something like having a unified throttle timer mechanism that checks if there are messages to commit at the end of every throttle timeout and inserts a "pseudo-batch" of messages to trigger processing might work - let me know if the "last messages not being committed" issue is not really an issue and I'm missing something

[samwillis]

My thinking was that as we will always end with an up-to-date in the protocol, and that forces an un-throttled commit, that we may not need to have a timeout like a "normal" throttling utility?

[msfstef]

should onInitialSyncCalled be reset here? Or do we consider the rebuilding of data an internal thing and thus don't expose any "initial data" info?

[samwillis]

Yes, I consider it an internal thing.

The idea is that the initialSync callback is only ever called once, that way it can be used to turn on a write path, install triggers, etc. that may add overhead.

TBH, must-refetch adds a lot of pain for large datasets. Deleting everything and reinserting, will be slow, particularly if using Postgres triggers for conflict resolution (as I have in Linearlite). This needs more exploration.