fix: handle Nirjas extraction errors and optimize file fallback #114

[Abh-igyan]

Description
Fixes a crash in the TF-IDF agent caused by unhandled exceptions during Nirjas comment extraction, and optimizes the file fallback mechanism by replacing inefficient in-memory reads with shutil.copyfileobj().
Changes

Wrapped commentExtract() call in a try/except block inside CommentPreprocessor.extract() to gracefully handle Nirjas parsing failures instead of crashing the entire scan
Replaced inFile.read().split('\n') pattern with shutil.copyfileobj() in the exception fallback path — avoids loading the entire file into memory
Applied the same shutil.copyfileobj() optimization to the unsupported file extension fallback (else block) which had the same inefficiency
Added logging.warning() to surface extraction failures without terminating the scan
Added shutil and logging to imports

shutil uses chunk based writing thus reducing RAM overload. Impressive right?

How to test

Clone the repository and install dependencies:
(remember to install poetry with curl and not apt (gives old version).
Use this: curl -sSL https://install.python-poetry.org | python3 -

bash

git clone https://github.com/fossology/atarashi
poetry install
poetry run preprocess
cd atarashi

Download BusyBox (the real-world project that triggers the crash):

bash

wget https://sources.buildroot.net/busybox/busybox-1.36.1.tar.bz2 #use this link (updated)
tar -xf busybox-1.36.1.tar.bz2

Run the TF-IDF agent scan on the extracted source tree:

bash
poetry run atarashi -a tfidf -s CosineSim ./busybox-1.36.1
#use poetry run and not just atarashi; atarashi is not installed in your global system path; it lives inside the Poetry virtual environment we just built.

Inferences:
The scan completes without raising IndexError: list index out of range
A WARNING log appears for any file Nirjas fails to parse
The scan continues and produces results for remaining files

Fixes #114

[hastagAB]

Hey @Abh-igyan, The fix LGTM!

As a follow-up, it'd be nice to: Narrow the except Exception to IndexError

But nothing blocking here. Thanks for the contribution! 🎉

[Abh-igyan]

"Thanks for the review @hastagAB ! Updated the except clause to catch IndexError specifically as suggested." Will try to think about those intricacies from now.
By the way, I have updated the PR description more elaborately step-by step to successfully run and test. As I had faced numerous issues while testing.

[Abh-igyan]

Hey @hastagAB <https://github.com/hastagAB> @Kaushl2208 <https://github.com/Kaushl2208> @shaheemazmalmmd <https://github.com/shaheemazmalmmd> @its-sushant <https://github.com/its-sushant> @GMishx <https://github.com/GMishx>*. *I've updated the except clause as suggested. Also ran Atarashi on BusyBox to verify the fix — scan completes without crashing now [image: resolved.png] While working on understanding the Nirjas-Atarashi pipeline for this fix, I also built a small POC exploring ML-based comment classification as part of my GSoC proposal: https://github.com/Abh-igyan/Nirjas-ML-POC (already shared on discussions page few days ago but received no feedback) Would really appreciate a quick look at the proposal draft <https://docs.google.com/document/d/1T2PqpRmekFOdYjxPqc1NnAv46URyeatT/edit?usp=sharing&ouid=109378995334582097647&rtpof=true&sd=true> if you have a moment before the deadline. Happy to share it here.

…

On Wed, 25 Mar 2026 at 16:06, Ayush Bhardwaj ***@***.***> wrote: ***@***.**** approved this pull request. Hey @Abh-igyan <https://github.com/Abh-igyan>, The fix LGTM! As a follow-up, it'd be nice to: Narrow the except Exception to IndexError But nothing blocking here. Thanks for the contribution! 🎉 — Reply to this email directly, view it on GitHub <#120?email_source=notifications&email_token=BNWHLCFGUQTVVWH4IMX2FPT4SOZDLA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTIMBQGU3DGNZWHA42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#pullrequestreview-4005637689>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BNWHLCHAQY7RPLOUTTQ6WQT4SOZDLAVCNFSM6AAAAACW4EL7DWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHM2DAMBVGYZTONRYHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>