Skip to content

⚡ Bolt: Optimized document metadata fetching #84

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nkissick-del wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚡ Bolt: Optimized document metadata fetching #84

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions api/db/services/doc_metadata_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -758,7 +758,11 @@ def get_metadata_for_documents(cls, doc_ids: Optional[List[str]], kb_id: str) ->
Dictionary mapping doc_id to meta_fields dict
"""
try:
results = cls._search_metadata(kb_id, condition={"kb_id": kb_id})
condition = {"kb_id": kb_id}
if doc_ids:
condition["id"] = doc_ids

results = cls._search_metadata(kb_id, condition=condition)
if not results:
return {}

Expand Down Expand Up @@ -818,7 +822,11 @@ def _meta_value_type(value):
return "string"

try:
results = cls._search_metadata(kb_id, condition={"kb_id": kb_id})
condition = {"kb_id": kb_id}
if doc_ids:
condition["id"] = doc_ids

results = cls._search_metadata(kb_id, condition=condition)
if not results:
return {}

Expand Down Expand Up @@ -997,7 +1005,11 @@ def _apply_deletes(meta):
return changed

try:
results = cls._search_metadata(kb_id, condition=None)
condition = {"kb_id": kb_id}
if doc_ids:
condition["id"] = doc_ids

results = cls._search_metadata(kb_id, condition=condition)
if not results:
results = [] # Treat as empty list if None

Expand Down
3 changes: 2 additions & 1 deletion api/db/services/document_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@ def get_list(cls, kb_id, page_number, items_per_page, orderby, desc, keywords, i
docs = docs.paginate(page_number, items_per_page)

docs_list = list(docs.dicts())
metadata_map = DocMetadataService.get_metadata_for_documents(None, kb_id)
doc_ids = [d["id"] for d in docs_list]
metadata_map = DocMetadataService.get_metadata_for_documents(doc_ids, kb_id)
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Guard the metadata lookup when there are no page results; an empty doc_ids list triggers a full KB metadata fetch in DocMetadataService and negates the performance optimization on empty pages.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At api/db/services/document_service.py, line 104:

<comment>Guard the metadata lookup when there are no page results; an empty doc_ids list triggers a full KB metadata fetch in DocMetadataService and negates the performance optimization on empty pages.</comment>

<file context>
@@ -100,7 +100,8 @@ def get_list(cls, kb_id, page_number, items_per_page, orderby, desc, keywords, i
         docs_list = list(docs.dicts())
-        metadata_map = DocMetadataService.get_metadata_for_documents(None, kb_id)
+        doc_ids = [d["id"] for d in docs_list]
+        metadata_map = DocMetadataService.get_metadata_for_documents(doc_ids, kb_id)
         for doc in docs_list:
             doc["meta_fields"] = metadata_map.get(doc["id"], {})
</file context>
Suggested change
metadata_map = DocMetadataService.get_metadata_for_documents(doc_ids, kb_id)
metadata_map = DocMetadataService.get_metadata_for_documents(doc_ids, kb_id) if doc_ids else {}
Fix with Cubic

for doc in docs_list:
doc["meta_fields"] = metadata_map.get(doc["id"], {})
return docs_list, count
Expand Down
6 changes: 6 additions & 0 deletions rag/utils/es_conn.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,12 @@ def search(
bool_query = Q("bool", must=[])
condition["kb_id"] = knowledgebase_ids
for k, v in condition.items():
if k == "id":
if isinstance(v, list):
bool_query.filter.append(Q("ids", values=v))
elif isinstance(v, str):
bool_query.filter.append(Q("ids", values=[v]))
continue
Comment on lines +129 to +133
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Missing empty list check for id filter. When v is an empty list, Q("ids", values=[]) will match no documents, which may cause unexpected zero results. The delete method in this file explicitly handles this case by skipping the filter when the list is empty. Consider adding a similar check here for consistency.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At rag/utils/es_conn.py, line 129:

<comment>Missing empty list check for `id` filter. When `v` is an empty list, `Q("ids", values=[])` will match no documents, which may cause unexpected zero results. The `delete` method in this file explicitly handles this case by skipping the filter when the list is empty. Consider adding a similar check here for consistency.</comment>

<file context>
@@ -125,6 +125,12 @@ def search(
         condition["kb_id"] = knowledgebase_ids
         for k, v in condition.items():
+            if k == "id":
+                if isinstance(v, list):
+                    bool_query.filter.append(Q("ids", values=v))
+                elif isinstance(v, str):
</file context>
Suggested change
if isinstance(v, list):
bool_query.filter.append(Q("ids", values=v))
elif isinstance(v, str):
bool_query.filter.append(Q("ids", values=[v]))
continue
if isinstance(v, list) and v:
bool_query.filter.append(Q("ids", values=v))
elif isinstance(v, str) and v:
bool_query.filter.append(Q("ids", values=[v]))
continue
Fix with Cubic

if k == "available_int":
if v == 0:
bool_query.filter.append(Q("range", available_int={"lt": 1}))
Expand Down
6 changes: 6 additions & 0 deletions rag/utils/opensearch_conn.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,12 @@ def search(
bqry = Q("bool", must=[])
condition["kb_id"] = knowledgebaseIds
for k, v in condition.items():
if k == "id":
if isinstance(v, list):
bqry.filter.append(Q("ids", values=v))
elif isinstance(v, str):
bqry.filter.append(Q("ids", values=[v]))
continue
Comment on lines +157 to +162
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Missing falsy value check for 'id' condition. If v is an empty list or empty string, this will add an ids filter with empty/invalid values, potentially causing unexpected query results (returning zero documents). Add a guard if not v: continue to match the existing pattern used for other condition keys.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At rag/utils/opensearch_conn.py, line 157:

<comment>Missing falsy value check for 'id' condition. If `v` is an empty list or empty string, this will add an ids filter with empty/invalid values, potentially causing unexpected query results (returning zero documents). Add a guard `if not v: continue` to match the existing pattern used for other condition keys.</comment>

<file context>
@@ -154,6 +154,12 @@ def search(
         bqry = Q("bool", must=[])
         condition["kb_id"] = knowledgebaseIds
         for k, v in condition.items():
+            if k == "id":
+                if isinstance(v, list):
+                    bqry.filter.append(Q("ids", values=v))
</file context>
Suggested change
if k == "id":
if isinstance(v, list):
bqry.filter.append(Q("ids", values=v))
elif isinstance(v, str):
bqry.filter.append(Q("ids", values=[v]))
continue
if k == "id":
if not v:
continue
if isinstance(v, list):
bqry.filter.append(Q("ids", values=v))
elif isinstance(v, str):
bqry.filter.append(Q("ids", values=[v]))
continue
Fix with Cubic

if k == "available_int":
if v == 0:
bqry.filter.append(Q("range", available_int={"lt": 1}))
Expand Down