Sporadic BamlValidationErrors during test runs

[gnguralnick]

Somewhat frequently when running tests I get a failure with the following error:

BamlValidationError: Failed to parse LLM response: Failed to coerce valu
e: <root>: Failed while parsing required fields: missing=1, unparsed=0
  - <root>: Missing required field: data

The error happens during check steps. Sometimes, rerunning the same test with no changes causes the error not to happen, so I'm not sure what the cause is.

This is with magnitude-core 0.2.22 and magnitude-test 0.3.1.

[anerli]

what LLM are you using?

[gnguralnick]

anthropic:claude-sonnet-4-20250514

[gnguralnick]

Example magnitude.config.ts:

import { type MagnitudeConfig } from 'magnitude-test';

export default {
    url: "<url>",
    browser: {
        contextOptions: {
            viewport: { width: 800, height: 600 },
            recordVideo: {
                dir: './videos/',
                size: { width: 800, height: 600 }
            }
        }
    }
} satisfies MagnitudeConfig;

Example tests.mag.ts:

import { test } from 'magnitude-test';

test.group('website functionality tests', { prompt: "Be sure to scroll through every page to view its full content. Scroll no more than 800px each time." }, () => {
    test("Verify that clicking on an art piece image from the list opens a dedicated description page with relevant content and images.", async (agent) => {
    await agent.act("Navigate to the website homepage.");
    await agent.act("Navigate to the page displaying the list of art pieces.");
    await agent.act("Scroll until the first art piece image is visible.");
    await agent.act("Click on the first art piece image displayed.");
    await agent.act("Scroll to the bottom of the page.");
    await agent.check("A dedicated description page for the clicked art piece is displayed.");
    await agent.check("A larger image of the art piece and descriptive text about it are visible on the page.");
    await agent.check("The page's color theme and font exhibit a historic, artsy touch that complements the Louvre's aesthetic.");
});
});

In this case the error happened on the check "A larger image of the art piece and descriptive text about it are visible on the page." But didn't happen on a second run. Sometimes it happens multiple times in a row

[anerli]

Hm thanks for the details. This occurs when the LLM isn't following the exact expected response structure, which usually Sonnet is good about so this is an interesting case.