Skip to content

draft for gbh#53

Open
mch2 wants to merge 7 commits intobharath-techie:native-engfrom
mch2:feature/delegation
Open

draft for gbh#53
mch2 wants to merge 7 commits intobharath-techie:native-engfrom
mch2:feature/delegation

Conversation

@mch2
Copy link

@mch2 mch2 commented Mar 24, 2026

  • Re-name SPI contract to ReaderManagerProvider to load in DataFormatAwareEngineFactory.
  • Deletes EngineBridge - replace with SearchExecEngine
  • add delegation framework api - sketch with just filter for now
  • removes non reader functionality from DataFormatAwareEngineFactory/Engine.

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

bharath-techie and others added 6 commits March 20, 2026 14:13
Signed-off-by: bharath-techie <bharath78910@gmail.com>
Signed-off-by: bharath-techie <bharath78910@gmail.com>
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
* Refactor CompositeEngine to use factory

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce SegmentCollector

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce SegmentCollector

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce SegmentCollector

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

---------

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
* Refactor CompositeEngine to use factory

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce SegmentCollector

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce SegmentCollector

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Introduce SegmentCollector

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* De-couple and simplify index file deleter

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* De-couple and simplify index file deleter

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* De-couple and simplify index file deleter, handle scorer and weight query lifecycle

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* De-couple and simplify index file deleter, handle scorer and weight query lifecycle

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* De-couple and simplify index file deleter, handle scorer and weight query lifecycle

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

---------

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
Signed-off-by: bharath-techie <bharath78910@gmail.com>
@mch2 mch2 requested a review from Bukhtawar as a code owner March 24, 2026 00:53
- Re-name SPI contract to ReaderManagerProvider to load in DataFormatAwareEngineFactory.
- Deletes EngineBridge - replace with SearchExecEngine
- add delegation framework api - sketch with just filter for now
- removes non reader functionality from DataFormatAwareEngineFactory/Engine.

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
@mch2 mch2 force-pushed the feature/delegation branch from 46e2002 to a583373 Compare March 24, 2026 00:56
int numDocs = maxDocId - minDocId;
BitSet bitset = new BitSet(numDocs);

Scorer scorer = context.getWeight().scorer(leaf);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are scoring for each call for each row group though, can't we model the delegates with same APIs as index filter provider / context.

if (requestContext.hasDelegation()) {
DelegationContext delegation = requestContext.getDelegationContext();
substraitBytes = SubstraitConverter.embedDelegation(
substraitBytes, delegation.getId(), null, "lucene-analytics-backend");
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't "lucene-analytics-backend" leaking in datafusion ?

this.datafusionQuery = query;
}

public IndexFilterTree getFilterTree() {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be still needed for boolean queries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants