-
Notifications
You must be signed in to change notification settings - Fork 71
Open
Description
Summary
When building the configured catalog, the _get_sync_mode helper function unconditionally falls back to SyncMode.incremental, even for streams that only support full_refresh. This can cause connector failures for streams that don't support incremental sync.
Background
This issue was identified during code review of PR #899 (force_full_refresh fix). The current behavior predates that PR, so it's not a regression - but it's a valid edge case that should be addressed.
Requested by: @aaronsteers (AJ Steers)
Current Behavior
def _get_sync_mode(stream: AirbyteStream) -> SyncMode:
supported_modes = getattr(stream, "supported_sync_modes", None)
if force_full_refresh:
if supported_modes and SyncMode.full_refresh in supported_modes:
return SyncMode.full_refresh
return SyncMode.incremental # BUG: stream might not support incremental
return SyncMode.incremental # BUG: stream might only support full_refreshExpected Behavior
The selected sync mode should always be validated against supported_sync_modes when that attribute is present:
def _get_sync_mode(stream: AirbyteStream) -> SyncMode:
supported_modes = getattr(stream, "supported_sync_modes", None)
if force_full_refresh:
if supported_modes and SyncMode.full_refresh in supported_modes:
return SyncMode.full_refresh
if supported_modes and SyncMode.incremental in supported_modes:
return SyncMode.incremental
return SyncMode.full_refresh # fallback for unknown/empty
# Default: prefer incremental if supported, otherwise full_refresh
if supported_modes and SyncMode.incremental in supported_modes:
return SyncMode.incremental
if supported_modes and SyncMode.full_refresh in supported_modes:
return SyncMode.full_refresh
return SyncMode.incremental # preserve backwards compat for unknownAffected Code
airbyte/sources/base.py-_get_sync_modehelper function (around line 449)
References
Metadata
Metadata
Assignees
Labels
No labels