Blog Post: LLM / AI for using pharmaverse tools at Roche

[StefanThoma]

Blog Post

How we created an LLM to specifically help programmers use pharmaverse tools.

[bms63]

Hey @StefanThoma and @MathieuCayssol is this one getting worked on? we need a post for October and wondering if we can get this one out

[StefanThoma]

Hi @ben Straub ***@***.***> @mathieu Cayssol ***@***.***> is preparing something similar for the PHUSE that will be expanded upon for the blog. I think we will only get it out for November. Let me know what you think, @mathieu Cayssol ***@***.***>

…

On Fri, Oct 17, 2025 at 8:56 PM Ben Straub ***@***.***> wrote: *bms63* left a comment (pharmaverse/blog#312) <#312 (comment)> Hey @StefanThoma <https://github.com/StefanThoma> and @MathieuCayssol <https://github.com/MathieuCayssol> is this one getting worked on? we need a post for October and wondering if we can get this one out — Reply to this email directly, view it on GitHub <#312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJUWWEXHB4R3GPHZMRU2QVL3YE3U7AVCNFSM6AAAAAB77R3JECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMJWG4ZDEOJYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MathieuCayssol]

Here is the article. Will open the MR in the following days.

TL;DR

Finding and reusing clinical programming code is currently slow and scattered. The Code Search Assistant (CSA) fixes this by letting you search code by meaning, not just keywords. It breaks programs into manageable chunks, enriching each with metadata and a short AI summary. Then, when you query the CSA, it returns ranked snippets that match your plain-English question. Each result shows things like the file path, program name, study name, repository, and study description. The result is: quicker reuse of proven code, fewer duplicate implementations, and more consistent programs across studies.

Why this matters for pharmaverse programmers

Clinical reporting follows repeatable patterns—ADaM derivations, TFL layouts, and QC checks—but examples are scattered across templates and repositories. Templates cover most standard outputs; however, for non-standard requests or bespoke TFLs, keyword search often misses intent (e.g., “derive ABLFL in ADLB using a pre-dose baseline rule and analysis-visit windows”) because variable names and structures vary by study. CSA translates that intent into semantic retrieval, surfacing relevant, provenance-linked code snippets even when the wording doesn’t match exactly—so you can review, trust, and reuse.

What we built?

CSA is a focused agent inside our Clinical Analysis Assistant. A user asks a question; we create an embedding of that query, apply optional metadata filters, search the vector database, and return the most relevant code chunks alongside their origins.

Behind the scenes, we index code from our repositories, split it into coherent chunks, and generate short summaries that describe what each piece of code does, its inputs and outputs, and, for TLG programs, details such as titles and footnotes. We embed the summaries, store the vectors, and retain rich metadata such as programming language, file name, repository, and study description. The interface displays both the results and the parameters used so retrieval stays transparent and audit-friendly.

How it works

Understand the ask

We transform user's question into a semantic search query + metadata filters.

Retrieve with context

We perform a semantic search over the vector database + apply substring matching based on the metadata filters. Summaries help match intent (“derive AVALC from PARAMCD”) even if the snippet uses different variable names. The metadata filters help to narrow down the results to the most relevant snippets (e.g: adsl in the program name or phase III in the study description).

Return with provenance

Results show snippet text, file path, repo, and any available study metadata. You can jump to source for review and reuse.

Design principle: retrieval should be fast, explainable, and reproducibles. It should be possible to trace back the code to the original source.

Example

Live demo

What programmers can do today

With CSA, programmers can look up ADaM and TLG scripts by intent, such as "asking for an ADSL baseline flag with visit windowing" or "an AVALC mapping that treats missing values explicitly". They can locate TLG programs described in plain English. For instance, a table that splits columns by treatment arm and summarizes AVAL with mean and standard deviation and then adapt a proven layout. They can also compare similar implementations across repositories to converge on a standard approach while maintaining confidence through clear provenance back to source.

Metrics and usage

Over the observation window, the Code Search Assistant (CSA) handled 635 questions from 45 unique users, averaging 42 ± 14 weekly conversations and 92 ± 43 weekly questions. Satisfaction signals were positive: the broader Clinical Analysis Assistant (CAA) scored 3.39/5 for usefulness (n = 174), while the CSA specifically was rated at ~4/5, indicating strong early traction among active users.

Limitations & what’s next

Today, CSA focuses on fast, trustworthy discovery. We are working on tighter IDE integration via MCP so you can search directly from your editor. We also want to enable better study awareness so results are automatically filtered to the context of your study and clinical programming code. Finally, we are refining the data pipeline to continuously index changes without any manual effort and adding SDTM programs as well. These steps aim to make retrieval not only smarter but also more seamlessly woven into day-to-day programming.

[bms63]

Looks intriguing! I'm happy for this still to go out @manciniedoardo

@MathieuCayssol this might be our last post for a while. The blog team is disbanding, but someone might take over down the road.

[manciniedoardo]

sounds good, can also use the PR to fix the missing image on the last post :)