Skip to content

Safety & Robustness Refactor#326

Open
danalec wants to merge 51 commits intophiresky:masterfrom
danalec:master
Open

Safety & Robustness Refactor#326
danalec wants to merge 51 commits intophiresky:masterfrom
danalec:master

Conversation

@danalec
Copy link
Copy Markdown

@danalec danalec commented Nov 24, 2025

  • Replace unsafe env mutations with safe std::env::set_var executed before spawning threads.
  • Remove unsafe lifetime transmute in ZIP adapter by restructuring stream iteration:
  • Replace unwrap/expect/panic! in non-test code with ? and contextual anyhow errors.
  • Harden external command calls (rg, ffprobe, ffmpeg, custom binaries):

…nRead entries into owned buffers and yield safe AsyncRead via SyncIoBridge to avoid lifetime hacks.
…std::env::set_var early in single-threaded setup to avoid UB and improve robustness.
…ontextual errors for piped IO; capture stderr and check exit codes for ffmpeg/ffprobe.
…ail splits and parsing safely; extract raw body via match to avoid panics.
…ure stderr and validate piped IO in rga.rs and custom adapters; replace expects with contextual errors.
…xt on metadata.modified() for robust error reporting.
…duplicate versions; no API changes expected for current usage.
…prove throughput for large files during preprocessing.
…rga-zip-extensions and --rga-ffmpeg-extensions; apply overrides in get_adapters_filtered using AdapterOverride wrapper.
…he parsed RGA_CONFIG for preproc; avoid repeated env JSON parsing.
@danalec
Copy link
Copy Markdown
Author

danalec commented Nov 24, 2025

Performance

  • Config caching for rga-preproc to avoid per-file config reads.
  • Buffered I/O policy where large reads benefit from BufReader.
  • Logging:
    • Standardize formatting; include durations with print_dur and consistent prefixes.
    • Improve adapter activation logs; show selected adapters and pre-globs.

…rides in preproc\n\nFix manifest duplication and update get_adapters_filtered call to pass config for overrides.
…\nAdd zip_extensions and ffmpeg_extensions examples to default config; tests verify overrides applied to metadata.
…ible versions to allow latest stable patches: anyhow 1.0, regex 1, serde_json 1.0, env_logger 0.10, log 0.4.
…topt provides required CLI; removing clap v4 reduces duplication in cargo tree.
…bility\n\nFix unresolved imports and async_zip API mismatch; keep features consistent.
… RGA_CONFIG propagation to process env\n\nReplace SyncIoBridge with StreamReader over Bytes; set RGA_CONFIG via Command.env to avoid unsafe env mutation.
…e reader()\n\nStore filename and sizes before taking mutable reader to satisfy borrow checker.
…der and pass PATH via Command.env\n\nIntroduce compute_exe_path() returning PATH value; use RUST_LOG through logger builder and avoid global env set.
…rap subprocess stdout and postproc streams to replace \r\n with \n for platform-agnostic outputs.
…th clap::Parser, update attributes and parse/command calls; remove structopt dep and add clap v4.
…alue_t for typed defaults and import CommandFactory for command() usage.
…mpression algorithm features\n\nFix compression level type; remove unused clap imports.
… on stable. Accept benign Windows ecosystem duplicates in cargo tree -d for now.
@danalec
Copy link
Copy Markdown
Author

danalec commented Nov 24, 2025

Dependency upgrades and CLI migration

  • Migrate CLI to clap v4 (replace structopt )
  • Align core crates: anyhow 1.0 , regex 1 , serde_json 1.0 , env_logger 0.10 , log 0.4
  • Bump schemars 0.9 , tree_magic_mini 3.1.6 , rusqlite 0.31 , tokio-rusqlite 0.5.1
  • Align async-compression 0.3.15 with algorithm features; keep async_zip 0.0.12 for API compatibility
  • Normalize CRLF → LF in adapters; build/tests/clippy all green
  • cargo tree -d duplicates minimized; remaining splits are upstream-driven

…Switch to clap v4 parsing; pass-through params integrated for list, preview, and fzf.
…\n\nAdd --rga-postproc-binary-marker, --rga-postproc-page-prefix, and --rga-postproc-page-include-empty; wire into postproc adapters.
…output wait\n\nWrap evince and open operations with contextual errors; improve fzf wait error context.
…ig\n\nAdd CLI flags and config keys with examples; clarify that overrides replace defaults and impact pre-glob.
@danalec
Copy link
Copy Markdown
Author

danalec commented Nov 24, 2025

Adapter extension overrides

  • Implemented config-driven extension overrides for ZIP and FFmpeg.
  • Added CLI flags --rga-zip-extensions and --rga-ffmpeg-extensions ; applied at adapter selection and reflected in pre-glob.
  • Added unit tests verifying override application; updated README and default config with examples.

FZF integration flags

  • Implemented rga-fzf passthrough flags:
    • --rg-params for list command.
    • --rg-preview-params for preview command.
    • --fzf-params for the fzf invocation.
  • Documented usage in README and default config.

Post-processing configuration

  • Added configurable binary marker ( --rga-postproc-binary-marker ).
  • Added page prefix ( --rga-postproc-page-prefix ) and empty page inclusion toggle ( --rga-postproc-page-include-empty ).
  • Wired into postproc adapters; tests updated accordingly.

Documentation

  • README: new “Adapter Extension Overrides” section and FZF flags subsection.
  • doc/config.default.jsonc : adapter override keys and FZF flags examples.

Replace unwrap with error propagation via anyhow::Context; no runtime panics.
Make lazy_static FROM_REGEX a Result<Regex> and use FROM_REGEX.as_ref()? at call site to avoid runtime panics.
@danalec
Copy link
Copy Markdown
Author

danalec commented Nov 24, 2025

  • Remove runtime panics from adapters by eliminating unwrap in regex construction and propagating errors with context.

- Add config_hash method to RgaConfig that hashes relevant configuration fields
- Include file modification time in cache key for real files
- Pass file_mtime_unix_ms through AdaptInfo chain for consistent caching
- Add safety limit for zip file memory pre-allocation (10MB max)
- Adjust buffer sizes based on accurate mode to reduce memory usage
…ecomputation

The loop_adapt function and related helpers now accept an active_adapters parameter, which is passed through recursive calls instead of recomputing the adapter list each time. This eliminates redundant work when processing nested archives.

- Modify choose_adapter to optionally accept precomputed active adapters
- Update buf_choose_adapter and loop_adapt to propagate active adapters
- Adjust caching logic to use provided adapters
- Simplify zip adapter by extracting make_zip_adapt_info helper
- Update all test calls to pass get_all_adapters result
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant