Users/tanmayapanda/address review comments #458
+1,290
−381
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant changes to the ingestion client and uploader logic to simplify and clarify multi-source ingestion, enforce stricter type safety, and improve concurrency handling. The main focus is to restrict multi-source ingestion to only support blob sources, remove internal logic for uploading local sources within multi-ingest, and enhance the uploader's parallelism and load distribution.
Multi-source ingestion API and logic changes:
ingestAsync) to only acceptBlobSourceobjects, both in the interface and implementation, and updates all related documentation to clarify that local sources must be uploaded separately before ingestion. This prevents partial failure scenarios and enforces a clearer, safer workflow. [1] [2] [3] [4]uploadLocalSourcesAsync) fromQueuedIngestClient, making it the caller's responsibility to upload local files/streams before calling multi-ingest. [1] [2]Error handling and messaging improvements:
IngestClientExceptionto use themessageparameter for clarity and consistency. [1] [2]Uploader concurrency and load balancing enhancements:
ContainerUploaderBasefrom random to round-robin using an atomic counter, ensuring more even load distribution across containers during uploads and retries. [1] [2] [3]Java interop and API usability:
DefaultConfigurationCachefor easier creation from Java, supporting both minimal and full-parameter configurations.@JvmOverloadsconstructors toFileSourceandStreamSourcefor better Java interoperability and sensible defaults. [1] [2]These changes collectively make the ingestion API safer, simpler, and more robust, especially in multi-source scenarios.### Added
Changed
Fixed