fix: Folder Sync Improvements - Source Path Tracking & LightRAG Fallback #138
+395
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add get_kb_content() to list documents/images - Add /content API endpoint - Fix folder_id parameter in upload task - All tested and working on macOS
## Changes since last push: ### 1. Fixed folder sync state tracking (knowledge.py) - Added 'folder_id' parameter to run_upload_processing_task - Implemented source path tracking: Now stores original source paths from linked folders instead of destination paths in synced_files - This fixes the bug where files always appeared as 'new' after sync because detect_folder_changes checks source paths but synced_files contained destination paths ### 2. Added LightRAG fallback (add_documents.py) - When RAGAnything module is unavailable, gracefully falls back to LightRAG pipeline instead of raising ImportError - New _process_with_lightrag_fallback method handles text extraction and indexing for PDF, DOCX, TXT, and MD files - New _extract_text_content helper for extracting text from various document formats using PyMuPDF and python-docx ### 3. Improved error handling - Added null checks for processed_files in progress messages - Better logging for sync state updates
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes folder sync functionality with two key improvements:
1. Fixed New File Detection (Source Path Tracking)
Problem: Files always appeared as "new" after sync because:
detect_folder_changeschecks source paths (in linked folder)synced_filesstored destination paths (in KB raw directory)Solution: Now stores source paths in
synced_files, matching whatdetect_folder_changesexpects.2. LightRAG Fallback for RAGAnything
Problem: When RAGAnything module was unavailable, sync would fail with
ImportError.Solution: Gracefully falls back to LightRAG pipeline:
Files Changed
src/api/routers/knowledge.py- Addedfolder_idparam and source path trackingsrc/knowledge/add_documents.py- Added LightRAG fallback methodsTesting