[FEATURE] question type classifier

[amirfz]

PROBLEM
Currently the qa_agent is limited to one RAG step regardless of the complexity of the question asked!

SOLUTION

• review the questions asked of sherpa in slack in the past 90 days
• identify broad categories of the question types
• find / create a training dataset for a classifier that determines the question type
• use the class and any necessary heuritics to determine paramters like the number of RAG steps
• rule out the need for a non-chain based solution strategy given the existing questions

ALTERNATIVES
n/a

OTHER INFO
n/a

[le-dawg]

Does it still make sense to rely so heavily on LangChain and force agentic design patterns through increasingly complex architectures that essentially emulate recurrence?

With models like DeepSeek-R1, proper reasoning capabilities are now available at low-compute scales and are relatively affordable to train. As with most ML systems, the true differentiator is data - datasets that demonstrate the reasoning behind using different tools in multi-task scenarios—that’s what Sherpa needs to move from PoC-hell to a production-ready reality. And producing them is cheap with DeepSeek.

In my opinion, the missing piece isn’t just improved RAG or more heuristics but intermediate tool use during reasoning. That’s where LangChain struggles and why a more native approach to reasoning might be the better path forward.

The intermediate tool use during reasoning is the key aspect that is missing in my opinion.