Skip to content

[Feature]It is recommended that ptubes support the separation of storage and computation #6

@myksl

Description

@myksl

Describe the feature

I recently encountered a problem. Our business has some tasks with relatively large traffic, which will write a large amount of data. At the same time, because of business characteristics, we expect the data to be stored for a long time, which will cause our disk usage to be relatively high. In extreme cases will cause disk exhaustion
So, I hope to have a new ability

  1. Added a new StorageMode inside Reader, which supports S3 remote storage + Memory mode, and supports tiered storage
  2. Support Binlog persistent storage
  3. The real-time consumption stream maintains the same speed as before
  4. The local disk will be used as the real-time data layer, S3 will be used as the persistent storage layer, and the non-latest data of the local disk will be asynchronously written to S3 through certain rules

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions