Trying to figure out how I can help / feature ideas / LLM work

[sliedes]

Hi!

This is a bit of "scratching an itch", but I think there's synergies where it can be useful for RECAP/courtlistener. I've been toying with an idea of developing something a bit customized since CourtListener doesn't seem to serve my current use case that well, but then, I guess the most productive course of action is to help CourtListener :) I didn't know where to talk of this, so I hope this is a good place.

I consider myself an "everything but UI/frontend" geek, including machine learning. I would suspect that my most unique skills lie outside web frameworks (I'm not very familiar with those, including for the backend, but I can figure it out; protocols and JSON etc. pose no problems). Right now I would also likely have some time, but I am bad at predictions, especially those concerning the future.

I think it helps if I describe what I do, what I want to do, and what I have thought about developing.

What I currently want to do

There have been developments in the (legal) world lately that have caused 100+ lawsuits around a vaguely defined theme. Being a law geek, I'd like to keep a track on what's going on. Previously I've done this with 5 or so lawsuits (and already wished for some tooling), but now it's getting a bit more unmanageable.

How I do it, and where the current UX fails me

Basically I have a tree of 100 open tabs of dockets that I refresh to see if there's updates.

There's some impedance between the courtlistener UI and what I want to do, as well as some things that I think just arent't there (and, I admit, might not be relevant for most users and thus possibly should not be implemented; take this all for background).

I'll try to rank the items for importance for my use case, as I currently perceive it; this is not intended to imply any kind of generic importance. Some of these are also the kind of things that I could try to solve with LLMs.

• One of the biggest things I'm missing is something to keep track of which documents, or which docket entries, I have already seen. (Importance for my use case: High)

  • One option here would be to use notes. I know there's a note tool that opens if I click Main Doc in a docket entry. I know that because I have clicked it, curious to see what's there. Generally, I just don't use that page. It has an embedded PDF. I see no point in embedded PDFs. It also has OCR'd text, in which I see a point, but when working on a computer, I prefer to just read the original PDF (using Download PDF). Thus, whatever note taking feature there is, I rarely even encounter it. I tested, and the note doesn't show in the docket, so it doesn't help me in that respect either. (Generally, importance of notes for my use case: Low to Medium.)

• To get a good overview of a case, or if I'm familiar with a case, I'd usually want to hide the less important stuff. Of course what is important is a value judgment, but usually admissions, appearances etc. administravia I would like to see hidden. (Importance for my use case: High. LLM potential if the data CourtListener has doesn't allow good classification.) Perhaps UI-wise it could be something that merely collapses those into a small expandable item saying "> 2 appearances, 1 motion pro hac vice, 1 order granting pro hac vice" or similar.

• Often docket text entries contain numbers of cases in courts of appeals, or other courts. I assume this is something that either is hard or just hasn't been implemented yet, but I'd love to see it. (Medium. LLM potential.)

• Docket text entries, as well as document texts, often mention other items in the docket, or some times items on other dockets. Links would be nice to have. (Same document: Low-Medium; Other dockets: Low. LLM potential.) A clever system could also presumably be able to track which motions are still live (Low; LLM potential, but trickier), or generally tracking the posture of the case.

Some thoughts on what I thought for my "custom tool" (some of which might make sense for CourtListener?)

When I was thinking of selectively downloading data to a graph database for personal use (although given the existing relational schema, I'm not sure it wouldn't be better to just run with that, even at small scale), here are some features that I thought I might want to implement:

• LLM summaries of documents, probably starting from text or OCR, either giving the model some case history or in isolation. (I think this would partially obviate the need to keep track of what I've seen—I can probably recognize a good summary of what I've read).
• Using these to understand the posture of a case.
• Keeping track of motions and deadlines.
• Showing a timeline of a case, or multiple cases on the same timeline (important developments, deadlines, ...)

Miscellaneous wishes

I looked a bit at the API. I can see that by default I don't have a lot of access (presumably to scaling reasons? also privacy?).

I appreciate these concerns, but I have one suggestion, even if you're open to giving me more access. For development purposes, it would likely be useful to have some amount of sample data accessible using the limited APIs too. I think a sane way that I assume should not overwhelm the systems could be to manually select a few suitable dockets/cases and allow access to those without extra permissions. I think this would let people play and see what they can do and then request for more access if they feel they're onto something.

[mlissner]

One of the biggest things I'm missing is something to keep track of which documents, or which docket entries, I have already seen.

I can understand this need, but I'm not sure notes is the way to do it. It feels like what you really want is something that highlights new content since the last time you loaded the page. This reminds me of Github's PR thing that gives you an option of "commits since last review."

The UX of this might be tricky, but I could see a few implementations:

1. Highlight the docket entries that are new since the last time you viewed the page. This is hard because of pagination, but probably doable?

2. Have a new filter in the filter bar that lets you filter to the same (easier on the backend, but the filter bar is so crowded already).

   

   Maaaaybe a clock icon next to the date input box that you can click? It could have a tooltip that says, "Since last time on this case" (or something), and when you click it, it could populate the first date box? Simple, unobtrusive, fairly easy to do?

I tested, and the note doesn't show in the docket,

That's a good idea. Notes on docket entries should appear on the docket. But it's hard from a UI perspective since notes can be long and they can be either on primary docs or attachments. Maybe such notes get crammed into an icon too?

If we start showing notes, people will want to edit them too...

I'd usually want to hide the less important stuff

We're working on docket entry classification now using an encoder model, but it'll take some time to get it right. Once that's done this feels more feasible (but so much judgement!)

Often docket text entries contain numbers of cases in courts of appeals, or other courts

Docket text entries, as well as document texts, often mention other items in the docket

These are both probably doable in some sense. We haven't tried. The trick is doing it well and dealing with 10,000 edge cases.

LLM summaries of documents

After we categorize the docs, we'll be able to write custom summarizers for each type of document. It's on the roadmap — But it's going to be impossible at scale, so we have to solve that.

The current solution to that is to have a credits system. You buy $20 of credit at a time, then you can summarize for a nickle or whatever by clicking a button.

Keeping track of motions and deadlines.

We have no ambition of doing this. Too scary and too risky.

Showing a timeline of a case, or multiple cases on the same timeline

Neat. Hard from a UI/design perspective, but doable, particularly once you have category tags.

For development purposes, it would likely be useful to have some amount of sample data accessible using the limited APIs too.

A fine idea, but not a priority for us. :)

---

So summarizing, there's a combination of things here. Some we'll do ourselves, eventually, others are new and interesting. My favorite ideas:

1. A button to filter to new stuff.
2. Adding notes to docket entries on the docket sheet.
3. Developing the credit system, and then adding summaries.

Thanks for writing in!

[sliedes]

One of the biggest things I'm missing is something to keep track of which documents, or which docket entries, I have already seen.

I can understand this need, but I'm not sure notes is the way to do it. It feels like what you really want is something that highlights new content since the last time you loaded the page. This reminds me of Github's PR thing that gives you an option of "commits since last review."

I agree that notes are not the perfect solution for this particular thing. I meant that in the absence of this, I could still see using notes to keep track of things... And, yes, I think that is a fairly good summary of what I'd like, with some caveats/stray thoughts (below).

The UX of this might be tricky, but I could see a few implementations:

1. Highlight the docket entries that are new since the last time you viewed the page. This is hard because of pagination, but probably doable?

Actually, why is there pagination? Is that a backend load issue or because of UX? I think I would personally prefer one long docket page (and believe that browsers would well handle thousands of entries), but I'm likely not typical.

Yeah, some kind of highlight, a NEW icon next to docket entries, bold, a red line between seen and unseen, or something.

One thing that springs to mind that there are actually two kinds of "new", and I'm not sure which is more interesting: What has been rendered before and, for example, what documents I have opened via the PDF link (I'm not sure if I'm the only one who uses those and not the embedded PDF page?).

2. Have a new filter in the filter bar that lets you filter to the same (easier on the backend, but the filter bar is _so_ crowded already).
   ![image](https://private-user-images.githubusercontent.com/236970/425296132-e7f47a87-3bf7-4655-bd2e-c306ac0551f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDI2MzI4NTUsIm5iZiI6MTc0MjYzMjU1NSwicGF0aCI6Ii8yMzY5NzAvNDI1Mjk2MTMyLWU3ZjQ3YTg3LTNiZjctNDY1NS1iZDJlLWMzMDZhYzA1NTFmMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDMyMlQwODM1NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00ZjkyYzhlODgxNWJmNTg5NGEzZWQ1OWJjNWM3MzkzOTExNmFmNWZiNGVkNDFiZTg4NWI2ZmE0Y2U4ZWJjYzFhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.UdJV-w7BDDYds-DUiFQTyWYA912Ue5PD1wiDMs8Xmdc)
   Maaaaybe a clock icon next to the date input box that you can click? It could have a tooltip that says, "Since last time on this case" (or something), and when you click it, it could populate the first date box? Simple, unobtrusive, fairly easy to do?

Wow. I agree it's cluttered. I don't think I've even ever realized there's filters there. For me it's been the cluttered bar that I use to click Desc (I often hope that was the default) and next. Search sounds useful, now that I see it. 🙈

I tested, and the note doesn't show in the docket,

That's a good idea. Notes on docket entries should appear on the docket. But it's hard from a UI perspective since notes can be long and they can be either on primary docs or attachments. Maybe such notes get crammed into an icon too?

If we start showing notes, people will want to edit them too...

I agree. Actually, I'm not sure I'd use the notes if I had to go to the embedded PDF page to edit them; the only use for that page that I have personally found is to read the OCR'd text, and that's how I discovered the notes feature and decided to try it to see if they are shown on the docket. :-)

Other than that, I think it would be very useful to show a sufficiently visible 📝 icon just to inform the user that there is a note for this entry. I guess clicking it or hovering on it could show the note. (I think UX-wise something that opens a small popup-ish text edit thing rather than takes one away from the docket page would be most useful? But that sounds like a lot of work.)

I'd usually want to hide the less important stuff

We're working on docket entry classification now using an encoder model, but it'll take some time to get it right. Once that's done this feels more feasible (but so much judgement!)

I agree, that sounds like a hard problem. And I think what's important is also... Hm.

So, I don't know if you've thought of this direction, but: Right now, the site feels mostly like there's a fairly static docket page that is displayed relatively as is for all users. If we consider a direction where a user would have more remembered state, I think the majority of what I'd want here would be probably best served by an ability to collapse docket entries that I find unimportant and expand entries that I find important. In fact, I'm pretty sure I'd like to have that even if there is a classifier, since "important for me" might be something different than for someone else, and "important today" might be something else than "important two months ago" (like, keeping track of live motions? IDK). Or some ability to tag, assign colors, mark, something. Perhaps the "what's new" thing could also use this?

Often docket text entries contain numbers of cases in courts of appeals, or other courts
Docket text entries, as well as document texts, often mention other items in the docket

These are both probably doable in some sense. We haven't tried. The trick is doing it well and dealing with 10,000 edge cases.

I understand this. I actually looked at some of your data, I think it was case names, and already saw that things are tricky and not always easily handled!

LLM summaries of documents

After we categorize the docs, we'll be able to write custom summarizers for each type of document. It's on the roadmap — But it's going to be impossible at scale, so we have to solve that.

The current solution to that is to have a credits system. You buy $20 of credit at a time, then you can summarize for a nickle or whatever by clicking a button.

I think this sounds good!

This is actually something I've thought of in other contexts. For example, for some Linux distributions, it would be nice if it was possible to donate time for things like autobuilders (perhaps targeted to packages that interest you). In practice, that's just impossible. Even here, I'd love to be able to donate CPU/GPU time for this, but even if there was a client that "anyone" can download (or a docker container), you'd run into the problem of how to keep malicious or broken submissions out.

The credit system is probably the most reasonably doable system.

Keeping track of motions and deadlines.

We have no ambition of doing this. Too scary and too risky.

Showing a timeline of a case, or multiple cases on the same timeline

Neat. Hard from a UI/design perspective, but doable, particularly once you have category tags.

I understand the scariness/riskiness.

Now, if I expand on the user state idea, this could still work if you let users add such metadata only for themselves? (And, if you want to get really fancy and build a facebook instead of courtlistener, ways to share and subscribe and ... 😛).

But I'd admit the motions/deadlines thing is also not for me the most interesting thing.

For development purposes, it would likely be useful to have some amount of sample data accessible using the limited APIs too.

A fine idea, but not a priority for us. :)

Makes sense :)

May I ask, are the interesting parts of the API closed more for backend load reasons, or privacy, or really both are very important reasons?

So summarizing, there's a combination of things here. Some we'll do ourselves, eventually, others are new and interesting. My favorite ideas:

1. A button to filter to new stuff.

2. Adding notes to docket entries on the docket sheet.

3. Developing the credit system, and then adding summaries.

I think this is a good favorite list, and I'd say tracks my ranking here quite well! As a data scientist, I'm of course the most excited for point 3, but as the law geek, I actually suspect numbers 1 and 2 are going to have the biggest impact for how I'm using it.

For the summaries: Have you considered the idea of a "case summary"? I think it's often much easier to figure out what's going on in a document than what's going on in a case. Of course, that would probably use some document summaries, and be tricky if a case is evolving...

Thanks for writing in!

Thanks for the kind reception :) And, indeed, I think I might be able to look at the LLM side as a volunteer a bit, if there's something suitably accessible where giving me a pointer is not so much work that it's a huge waste if I disappear (because I don't know how long/much time I have, say, in a few months).

[mlissner]

Actually, why is there pagination? Is that a backend load issue?

Yes. Some dockets have thousands of decisions, and even if there's only a couple hundred, we shouldn't make people load hundreds just to see the most recent.

Yeah, some kind of highlight, a NEW icon next to docket entries, bold, a red line between seen and unseen, or something.

Sure, that's not a bad idea.

I often hope [descending] was the default

You can tweak it in your profile: https://www.courtlistener.com/profile/settings/

I think it would be very useful to show a sufficiently visible 📝 icon just to inform the user that there is a note

Yeah, full note editing and display on docket entries makes sense. We should do that.

are the interesting parts of the API closed more for backend load reasons, or privacy, or really both are very important reasons?

Both, and a third. Historically, we had open APIs and we hoped that they'd be so cool people would open their wallets and support us. The support never came, so by closing them like this, we force people to talk to us and support us if it makes sense to do so.

Have you considered the idea of a "case summary"?

Yes, we'll be researching that soon. The plan is to summarize entries, then do a summary of the summaries.

---

Summarizing:

• We'll research ways to keep track of people's progress better.
• Note CRUD needs to be added to docket entries on the docket page.

I'm not sure what's a good LLM project, but if you want to work on either of the above, we'd certainly welcome it. First step is probably to get a couple designs done, then we can start getting user feedback.

@s-taube, what do you think? Can you run with these ideas, get a few designs together and we can see how people respond in your next round of interviews?