How to deal with ledger disk reaches 95%

[hangc0276]

For ledger disk usage reaches 95%, the minor and major compaction stopped issue, it is used to protect the ledger disk due to the garbage collection occupying more storage size. We have two parameters to control this behavior.

• isForceGCAllowWhenNoSpace and forceAllowCompaction Enable this two flags, bookie won't disable minor and major compaction when ledger disk usage reaches 90% and 95%. It will trigger major compaction when ledger disk usage reaches 90% and 95%. However, it has the risk of making the disk usage 100% and may introduce other unexpected issues. I prefer to enable these two flags, but we need the following changes: (Will discuss with @fantapsody @tuteng )
  • Change diskUsageThreshold=0.90, diskUsageWarnThreshold=0.85 and diskUsageLwmThreshold=0.85
• forceAllowCompaction If we only enable this flag and disable isForceGCAllowWhenNoSpace, the bookie will disable minor and major. But we can use the REST API command curl -XPUT http://<bookie-ip>:<port>/api/v1/bookie/gc -d '{"forceMajor": true}' to trigger major or minor compaction when the bookie runs into read-only mode. (This feature is only support since BookKeeper 4.15.0+, and Pulsar 2.11.0+ )

---

Knowledge

• Minor Compaction: If one entrylog file's remaining data size is lower than this threshold, the entrylog file will be compacted. The default value is 0.2. For example, the total entry log file size is 1GB, 900 MB of data has been expired, and the remaining data size 100 MB, which is lower than 0.2. This entrylog file will be compacted. The compacted process follows the following steps:

  • Bookie will read the remaining 100MB of data from this entrylog file and write to a new entrylog file
  • Delete the old entrylog file.

• Major Compaction: If one entrylog file's remaining data size is lower than this threshold, the entrylog file will be compacted. The default value is 0.5. The compaction process is the same as minor compaction. The more remaining data in the entry log file, the more extra disk space will be used during the compaction process.

• diskUsageWarnThreshold: When the ledger disk usage reaches this threshold, the bookie will suspend major compaction. The default value is 0.90

• diskUsageThreshold: When the ledger disk usage reaches this threshold, the bookie will run into read-only mode and suspend minor and major compaction. When the disk usage is lower than this threshold, resume minor compaction. The default value is 0.95

• diskUsageLwmThreshold: When the ledger disk usage is lower than this threshold, the bookie will recover to read-write mode and resume major and minor compaction. The default value is 0.95