Skip to content

fix: resilient LocalFile list() + notify feature gate + closure lifetime HRTB fix#103

Open
Copilot wants to merge 2 commits intomainfrom
copilot/upstream-sync-ignore-errors
Open

fix: resilient LocalFile list() + notify feature gate + closure lifetime HRTB fix#103
Copilot wants to merge 2 commits intomainfrom
copilot/upstream-sync-ignore-errors

Conversation

Copy link
Contributor

Copilot AI commented Mar 16, 2026

Ports upstream cocoindex PR #1753 to make LocalFile::list() resilient to per-entry errors during directory traversal. Also fixes two pre-existing CI failures introduced in the filesystem watch commit.

LocalFile list() resilience (upstream sync)

Adds try_ensure_metadata() — a non-propagating variant of ensure_metadata() that warns and returns Option<&Metadata> instead of erroring. All per-entry I/O errors in list() now warn and skip rather than aborting the entire listing:

// Before: single bad entry aborts the whole traversal
let file_type = entry.file_type().await?;
if file_type.is_symlink() && let Err(e) = ensure_metadata(...).await {
    if e.kind() == NotFound { warn!(...); continue; }
    Err(e)?;  // propagates
}

// After: all errors warn and skip
let file_type = match entry.file_type().await {
    Ok(ft) => ft,
    Err(e) => { warn!("Failed to get file type for {}: {e}", path.display()); continue; }
};
// symlink metadata, file size check, and ordinal fetch all use try_ensure_metadata

Pre-existing CI failures fixed

source-local-file missing dep:notify — The filesystem watch feature used notify in code but never added it to the source-local-file feature gate, causing E0432/E0433 on every non-full build:

source-local-file = [
  "dep:notify",   # added
  ...
]

apply_component_changes HRTB lifetime errorflat_map over Vec<&SetupChange<D>> triggered Rust's "implementation of FnOnce is not general enough" (E0308) when monomorphized for Neo4j's SetupComponentOperator. The return type of the inner closure depended on the outer closure's input lifetime, violating the implicit for<'0, '1> bound. Fixed by collecting futures into a Vec with explicit for loops instead of lazy iterator chains.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • test
    • Triggering command: /home/REDACTED/work/recoco/recoco/target/debug/deps/recoco_core-af5104cd93dc7568 /home/REDACTED/work/recoco/recoco/target/debug/deps/recoco_core-af5104cd93dc7568 e050294cd.rlib 7d0200b95c8.rlibtokio_stream -342df82e25d75b9--edition=2021 g/de�� lib/rustlib/x86_--error-format=json lib/rustlib/x86_--json=diagnostic-rendered-ansi,artifacts,future-incompat bin/rustc 0d177fd9ae1e00.sgit 0d177fd9ae1e00.sdiff 0d177fd9ae1e00.s--name-only bin/rustc 0d17�� 0d177fd9ae1e00.sembed-bitcode=no 0d177fd9ae1e00.s-C f/hyper-util-0.1.19/src/lib.rs 0d177fd9ae1e00.sbash 0d177fd9ae1e00.s--norc 0d177fd9ae1e00.s--noprofile 0d177fd9ae1e00.s--check-cfg (dns block)
    • Triggering command: /home/REDACTED/work/recoco/recoco/target/debug/deps/recoco_core-af5104cd93dc7568 /home/REDACTED/work/recoco/recoco/target/debug/deps/recoco_core-af5104cd93dc7568 debuginfo=2 -C debug-assertions=on c83-�� c83-cgu.2.rcgu.o--error-format=json c83-cgu.3.rcgu.o--json=diagnostic-rendered-ansi,artifacts,future-incompat bin/rustc c83-cgu.5.rcgu.o/home/REDACTED/.rustup/toolchains/stable-x86_64-REDACTED-linux-gnu/bin/rustc c83-cgu.6.rcgu.o--crate-name -o bin/rustc fc81�� fc81b6d-cgu.02.r--error-format=json fc81b6d-cgu.03.r--json=diagnostic-rendered-ansi,artifacts,future-incompat bin/rustc fc81b6d-cgu.05.rbash fc81b6d-cgu.06.r--norc fc81b6d-cgu.07.r--noprofile bin/rustc (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>[upstream-sync] LocalFile source: ignore errors and warn in list() for resilient directory listing (upstream PR #1753)</issue_title>
<issue_description>## Upstream Change Summary

Type: bug-fix / improvement
Difficulty: Easy
Recommendation: Adopt

Upstream changed the list() method for the LocalFile source to ignore all errors during directory listing and emit warnings instead of propagating errors. Previously, a single unreadable file or permission error could abort the entire listing. This makes the source more resilient when processing directories with mixed permissions.

Upstream References

Relevant Upstream Files / Areas

  • rust/cocoindex/src/ops/sources/local_file.rslist() method error handling

Recoco Considerations

  • Maps to crates/recoco-core/src/ops/sources/local_file.rs (or equivalent path)
  • Feature-gated behind source-local-file — no new feature gates needed
  • This change is purely in error handling logic, no API surface changes
  • Recoco uses the same local_file source implementation; this fix is directly applicable
  • Consider whether tracing/log crate integration aligns with recoco's logging approach (recoco may use tracing — ensure warnings use tracing::warn! not log::warn!)

Integration Notes

Simple and safe to adopt. Check that the list() function signature and surrounding code in recoco's local_file source matches the upstream before applying the patch. Related to filesystem watch support (issue #97) — both touch the same source file.
</issue_description>

<agent_instructions>Please implement this change.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

… closure lifetime bug

- Add try_ensure_metadata() helper: warns on error, returns Option<&Metadata>
- Change file_type().await? to match with warn+continue on error
- Remove old symlink error block, replace with try_ensure_metadata
- Use is_none_or for file size limit check (skip if metadata unavailable or too big)
- Use try_ensure_metadata with let..else for ordinal fetch
- Add dep:notify to source-local-file feature (fixes pre-existing CI failure)
- Fix apply_component_changes closure lifetime bug (HRTB E0308) in components.rs
  by using explicit for loops + Vec instead of flat_map iterator chains

Co-authored-by: bashandbone <89049923+bashandbone@users.noreply.github.com>
@cloudflare-workers-and-pages
Copy link

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
recoco-docs fd88c9a Mar 16 2026, 05:46 PM

Copilot AI changed the title [WIP] [upstream-sync] Update LocalFile source to ignore listing errors fix: resilient LocalFile list() + notify feature gate + closure lifetime HRTB fix Mar 16, 2026
Copilot AI requested a review from bashandbone March 16, 2026 17:51
@bashandbone bashandbone marked this pull request as ready for review March 16, 2026 21:19
Copilot AI review requested due to automatic review settings March 16, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[upstream-sync] LocalFile source: ignore errors and warn in list() for resilient directory listing (upstream PR #1753)

2 participants