Dekaf: Collection of small updates #2109
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
MetadataAPI calls, as we found out that these were hanging indefinitely, and consumers were just opening more and more connections as a result. I'd like to have a more global timeout, but that's complicated by the fact that some requests are potentially long-running, such asFetchwith itsmax_wait_msparameter. So I went with a more targeted timeout to start with.dekaf::topology::Collection::fetch_spec()if they're not needed -- this reduces the query load on supabaseestuary.dev/collectionwhen there is no binding partition selector toname:prefix. Combined with assemble: Switchjournal_selectorover to usingname:prefixinstead ofestuary.dev/collection#2108 to includename:prefixin partition selectors themselves, this will cause all calls tofetch_partitionsto usename:prefix.Note: AFAIK, the only case where we won't have a binding partition selector is when a connection is using the old user-token auth which we're getting rid of ASAP. So the change in this PR is only applicable to sessions using user-token auth, of which there are almost none at this point.
This change is