-
Notifications
You must be signed in to change notification settings - Fork 44
Users/ramacg/refactor managed uploader #455
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/IngestV2
Are you sure you want to change the base?
Users/ramacg/refactor managed uploader #455
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR refactors the ManagedUploader by simplifying the selectContainers method interface and enhancing the configuration cache to support server-specified refresh intervals.
Key Changes:
- Removed
configurationCacheparameter fromselectContainersmethod, accessing it as a class property instead - Changed
configurationCachevisibility from private to public inContainerUploaderBase - Added logic to use server-specified refresh intervals from configuration responses
- Added comprehensive tests for container selection with different upload methods
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| ingest-v2/src/test/resources/config-response.json | New test fixture providing sample configuration with both storage and lake containers |
| ingest-v2/src/test/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ManagedUploaderTest.kt | New parameterized test validating container selection logic for DEFAULT, STORAGE, and LAKE upload methods |
| ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ManagedUploader.kt | Removed configurationCache parameter from selectContainers method signature |
| ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ContainerUploaderBase.kt | Made configurationCache property public and updated selectContainers signature and documentation |
| ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/ConfigurationCache.kt | Enhanced to calculate effective refresh interval using minimum of default and server-specified intervals |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/ConfigurationCache.kt
Outdated
Show resolved
Hide resolved
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ContainerUploaderBase.kt
Outdated
Show resolved
Hide resolved
ingest-v2/src/test/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ManagedUploaderTest.kt
Show resolved
Hide resolved
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ContainerUploaderBase.kt
Outdated
Show resolved
Hide resolved
…uploader/ContainerUploaderBase.kt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…uploader/ContainerUploaderBase.kt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| LocalTime.parse(configRefreshInterval).toSecondOfDay() * | ||
| 1000L, |
Copilot
AI
Jan 9, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The refresh interval parsing uses LocalTime.parse() which expects a time format (HH:mm:ss), but the actual format in the config is a TimeSpan/Duration string ("01:00:00"). While "01:00:00" may parse as LocalTime, this is semantically incorrect - you're parsing a duration as a time-of-day. Consider using Duration.parse() with the ISO-8601 duration format (e.g., "PT1H") or implement proper TimeSpan parsing that handles the format correctly.
| // If we get both lake and storage and user does not specify, lake is preferred. If user | ||
| // specifies, respect that. |
Copilot
AI
Jan 9, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test logic for DEFAULT upload method has an issue. According to the test comment on lines 40-41, when DEFAULT is used and both lake and storage are available, lake should be preferred. However, the config-response.json sets "preferredUploadMethod": "Lake", and the actual ManagedUploader implementation (lines 64-100 in ManagedUploader.kt) will honor this server preference. The test correctly expects lake, but the comment suggests this is a client-side default rather than respecting the server's preference. Consider clarifying whether this test is validating the server preference mechanism or the client-side default behavior.
| // If we get both lake and storage and user does not specify, lake is preferred. If user | |
| // specifies, respect that. | |
| // When the server configuration prefers Lake and the user does not specify (DEFAULT), | |
| // ManagedUploader should honor the server preference and use Lake. If the user explicitly | |
| // specifies a method (e.g., STORAGE), that explicit choice is respected. |
No description provided.