Develop Benchmark User Journey

[cbfrance]

Request from Paul

I only have one research question, it’d be great if we can develop a benchmark user journey with a sequence of questions and thought process (perhaps fill out Decision Workflow.md) cc/

[cbfrance]

Two phases perhaps:

• I think a good first test would be to ask questions of a single PDF, right at the start, almost as part of kickoff. (feature centric exploration of what is the new benchmark, qualitative/vibe check.)
• Then (during "Immersion" phase) we will answer the question more fully — what are they actually doing at the microscope, who do they talk to, what are their outputs?

This second question, which I think is the actual "benchmark user journey" this ticket represents is likely a deliverable at end of Immersion, after three weeks of research.

[cbfrance]

Related to to user journey mapping, I think there is also a question of how to do a "SonoEval" — we need a list of questions that we can automatically evaluate, each question would have associated acceptance criteria.

So this could be in phases too:

• Initial evaluation could be just a few questions about the single PDF scenario.
• later evaluation set could be more comprehensive, across more types of supported interactions.

[gutelius]

Could also just ask to inspect the final deliverable from the consulting company

[gutelius]

I'm intersted in documenting the catalog of questions a team of researchers might potentially ask of any corpus. It's clearly a spectrum. It's also partially dependent on the state of the art today - single researchers, scanning, current IR - versus what might be possible with a new product - e.g. TEAMs of experts actively creating collaborative inquiry experiences together that might themselves create new incremental knoweldge artifacts, and that could be used as inputs into generative-powered exploration and creation.

I'm less interested in making a slightly more clever IR engine than in looking towards creating something that's fundamentally different and better than generative-powereed search. It might be a while until we can fully realize some of this, but I don't want to lose the longer term disruptive vision we might build towards.