Empty predictions for long(er) contexts

[JackMurphy97]

Hey! Big fan of this piece of work! I'm trying this out on a tool response use case but getting nothing back. It works on the short Paris example in the docs but when I make the context a bit longer it seems to break, even though it's nowhere near the 4k max token limit. Any ideas what's going wrong here?

Even in the token level predictions, I'm getting back an empty list (instead of a list of token-level predictions).

The below is dummy data

context = """
TOOL RESPONSE:

{
"data": [
{
"id": "eating-out",
"attributes": {
"name": "Eating Out",
"discretionary": True,
"amount-this-period": "$22.31",
"daily-spend-this-period": "$2.79",
"average-transaction-amount-this-period": "$11.16",
"transactions-this-period": 2,
"ranking-this-period": 2,
"amount-previous-period": "$94.00",
"transactions-previous-period": 2,
"daily-spend-percentage-change": "-76%"
}
},
]
"meta": {
"amount-this-period": "$263",
"daily-spend-this-period": "$32.92",
"average-transaction-amount-this-period": "$21.95",
"transactions-this-period": 12,
"time-period-start-at": "2025-02-26",
"time-period-end-at": "2025-03-05",
}
}
"""
question = "How much did I spend on eating out"
answer = "You spent $263 on eating out in the last 2 weeks."

predictions = detector.predict(context=context, question=question, answer=answer, output_format="spans")
print("Span Predictions:", predictions)

predictions = detector.predict(context=context, question=question, answer=answer, output_format="tokens")
print("Token Predictions:", predictions)

[adaamko]

Hey @JackMurphy97 , thanks for your question and I'm really happy you like our work. I'd like to applaud you because you are our first issuer, and you already with a great question :) So thanks again!

To answer your question, it seems that you are passing the context as a string, but we expect it as a list, so you should pass it like this context=[context] if you are only using one context document. Let me know if it helps! I will build some validation mechanism soon that should catch things like this :)

Please also note that currently, we don't have many QA training samples with JSON data like this, but I hope we can improve on these soon! Currently, I think the performance would be better if you form a free text prompt from your json and pass it like this: detector.predict_prompt(prompt, answer)

Best,
Adam

[maxyousif15]

Also a big fan of this work!

Would it be possible to convert to a free form text using yaml? Whilst it is still somewhat structured, it should be more free-text like than a JSON dump.

Another question for you @adaamko - do you have a guide for contributions? I'd like to add some more strict types if you don't mind. Namely, looking at the signature here https://github.com/KRLabsOrg/LettuceDetect/blob/main/lettucedetect/models/inference.py#L68 and adding literal types for output_format.

[adaamko]

@maxyousif15 maybe, we could try. Also maybe you could just list the fields in a string like format and see if it works better. Im definitely planning to add much more training data soon.

Currently you could just fork the repository and just hit me with a PR and we can discuss there!

[wgetdddeb]

@JackMurphy97
My context was very big and this is how I made it work

from lettucedetect.models.inference import HallucinationDetector
import time 
# For a transformer-based approach:
detector = HallucinationDetector(
    method="transformer", model_path="KRLabsOrg/lettucedect-base-modernbert-en-v1"
)
def detect_hallusinations(context, question, answer):
    s = time.time()
    predictions = detector.predict(context=context, question=question, answer=answer, output_format="tokens")

    count_0, count_1 = 0, 0  

    for pred in predictions:
        if pred["pred"] == 0:
            count_0 += 1
        else:
            count_1 += 1
    print("Predictions:", count_0, count_1)
    if count_1 > count_0:
        print("hallucinated")
    print(time.time() - s)

I also noticed tokens output format takes less time and gives better accuracy (not sure about this)