issues with regression suite for nlp_text branch

[zurabksmlp]

As of now SMLP regression suite includes test 229 to 262 to cover NLP and LLM capabilities of SMLP. The LLM related modes llm, finetune and rag work also by re-using SMLP-trained and saved models to perform LLM tasks like generation, question-answering, summarization. Because most of these saved models contain large files that are beyond git allowed size limit (or at least some default limit), these saved models (which are directories with possibly multiple sub-directories and files) were not included in git and are not uploaded to the main repository -- they are included in local .gitignore file for nlp_text branch.

The model directories includid in .gitigmore are:
smlp/repo/smlp/regr_smlp/rag_models
smlp/repo/smlp/regr_smlp/finetune_models
smlp/repo/smlp/regr_smlp/llm_models

The tests that load the saved llm, finetune and rag models are:
RAG: 240,242,244,246,248
Finetune: 250,252,254,256
LLM: 260,262.

In addition: test 245 that trains RAG model has relatively high memory requirements. When run on a laptop using the command line the test (usually) succeeds while when this test is run using smlp_regr.py as part of regression, the process gets killed.

[onestardao]

Thanks for sharing the detailed breakdown. From your description, this issue seems closest to Problem #7: Memory Breaks Across Sessions — models lose continuity across runs, and in your case, test 245 being killed due to high memory demand looks like a symptom of that.

Over the past 60 days, we’ve seen over 600+ stars on the Problem Map repo that catalogs real-world LLM/RAG failure modes. One of the recurring fixes is applying a Semantic Firewall pattern — basically a lightweight guard that filters or rewrites outputs before they escalate into hard failures. It doesn’t require infra changes, but gives you a buffer so the process doesn’t collapse outright.

If you’d like, I can share the checklist of failure points we’ve compiled (used to systematically debug similar regressions). It could help spot whether it’s purely a memory issue or if continuity breaks are compounding it.