[Feature]: Dirty memory size triggered CKPT

[githubzilla]

• Summary (Provide a general summary of the issue in the Title above):(required)

  • Leverage the statistics of dirty key ratio to trigger the CKPT
  • Base on the dirty key ratio and current memory foot print to calculate the actually dirty memory ratio to the total memory limit
  • If the dirty memory ratio is bigger than 20%, then trigger a checkpoint
  • Remove original checkpoint trigger when memory is full

• Detailed Description (Provide a detailed description of the change or addition you are proposing):(required)

• Software version (the version of the software you're running):(required)
  e.g. 0.0.1

• Possible Implementation(suggest an idea for implementing addition or change)(optional)

[githubzilla]

Yesterday, I spend some time to investigate several CI and nightly tests fails, so did not have updates for this work item.

• [Bug]: Write log fail and cc node group leader change at same time cause assert crash #404
• [Bug]: eloq-store crash when shutdown eloqstore#354
• [Bug]: Jepsen test failed due to eloqstore failed to read page eloqstore#357
• https://github.com/eloqdata/raft_host_manager/issues/12

[githubzilla]

• Ongoing tx_service PR

[githubzilla]

• eloq-test PR
• tx_service PR
• eloqkv PR

[githubzilla]

We have more work to do, since liuyl and I have new idea to control the frequency of the NotifyCKPT should be triggered

• Each ccshard will trigger CKPT individually, so we need to prevent the spam by check the request_ckpt_, but the mutex acquire too expensive, we change the type of request_ckpt_ as std::atomic, and using the release/acquire to guarantee the memory order.
• Before all DataSyncTask are finished on a core, the dirty memory footprint can be still high on that core, and the Ckpt will be triggered unnecessarily, we should avoid this issue.
  • Option 1: We keep track of the DataSyncTask for a core using a reference count, new Ckpt will be triggered only if the reference count is zero
  • Option 2: Define a minimum checkpoint interval to control the frequency