Describe the feature
I recently encountered a problem. Our business has some tasks with relatively large traffic, which will write a large amount of data. At the same time, because of business characteristics, we expect the data to be stored for a long time, which will cause our disk usage to be relatively high. In extreme cases will cause disk exhaustion
So, I hope to have a new ability
- Added a new StorageMode inside Reader, which supports S3 remote storage + Memory mode, and supports tiered storage
- Support Binlog persistent storage
- The real-time consumption stream maintains the same speed as before
- The local disk will be used as the real-time data layer, S3 will be used as the persistent storage layer, and the non-latest data of the local disk will be asynchronously written to S3 through certain rules