Skip to content

[Feature]Hope ptubes can support storage layer partition #7

@myksl

Description

@myksl

Describe the feature

At present, the concept of sharding is a logical concept, which only exists in the Client for parallel consumption. As a result, each Reader needs to pull the full binlog, and there is a visible performance upper limit for pulling and storing (cannot scale linearly)

  1. From logical sharding to physical sharding, the concept of partition is introduced, each Reader does not need to pull all binlogs, reducing storage and processing pressure, and at the same time supporting parallel processing of data by Reader (each Reader processes a part of the partition), improving ptubes Scalability
  2. The data of each partition can be subscribed separately to reduce the data backtracking when the service is duplicated
  3. Each partition has a separate writing thread, effectively breaking through the performance bottleneck of single-threaded sequential writing

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions