[server] Partition Progress Log

[KaiSernLim]

Problem Statement

We have lag-based metrics, but we don't understand ingestion progress during bootstrap.

Solution

For a position, calculate how far in the topic it is by dividing the current position by the end position. Add ingestion progress percentage in the following 3 scenarios:

1. Topic subscription
2. OffsetRecord sync to disk
3. IngestionNotificationDispatcher reporting Push Statuses

INFO StoreIngestionTask Subscribed to: [store-name]_[version]-[partition] position: EARLIEST
INFO StoreIngestionTask Offset synced for replica: [store-name]_[version]-[partition] - localVtOffset: ApacheKafkaOffsetPosition{[offset]}
INFO IngestionNotificationDispatcher Reported END_OF_PUSH_RECEIVED to 3 notifiers for PartitionConsumptionState: PCS{...}

It is behind a new server config feature flag named positional.progress.logging.enabled.

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
  Behind positional.progress.logging.enabled.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

How was this PR tested?

• New unit tests added.
  testGetIngestionProgressPercentage() in TopicManagerTest.

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.

[sushantmane]

In general, these logs are useful to have. When adding them, we need to ensure they don’t increase metadata query load on the servers, and that the additional requests don’t reduce or weaken ingestion path resiliency to metadata failures.

[sushantmane]

Thanks!