Kompari xtask (html report, interactive test blessing, dead snapshots detection) by spirali · Pull Request #272 · linebender/parley

spirali · 2025-02-12T12:34:39Z

Adds command cargo xtask that allows to:

Creating report

cargo xtask report

creates HTML file with report containing snapshots that does not match

Interactive test blessing:

cargo xtask review

stars a local web server with the following page:

(screenshot obtained by putting an artificial error into each snapshot test)

Detecting dead snapshots

cargo xtask dead-snapshots

This PR fully replaces #185

tomcur

Very cool! I made some tests intentionally fail subtly, and I think this will definitely help with e.g. spotting when there are actual issues vs just rounding differences.

Some minor nits inline, and I think you intended to commit .cargo/config.toml or something similar, I added a comment about that on parley/tests/README.md.

tomcur · 2025-02-13T10:06:55Z

Cargo.lock

 [[package]]
 name = "accesskit"
-version = "0.17.0"
+version = "0.17.1"


I believe according to https://github.com/linebender/rfcs/blob/main/rfcs/0005-version-matrix.md#cargotoml-matches-cargolock, Cargo.toml should be updated to match these patch versions.

It should be fixed now by running cargo upgrade

I'm not sure, for example Cargo.toml still lists accesskit = "0.17", but it's locked to "0.17.1" in the lockfile.

Maybe @DJMcNab can chime in?

We should always specify the patch version in Cargo.toml, yes. So we never should have a dependency of the form accesskit = "0.17".

That's largely a thing for consistency, but it does have advantages for e.g. when we publish, largely in case any user decides to use the MSRV-aware resolver or minimal-versions.

xtask/src/main.rs

Cargo.toml

parley/tests/README.md

tomcur · 2025-02-13T10:22:45Z

xtask/src/main.rs

+    let args = Args::parse();
+    let mut diff_config = DirDiffConfig::new(snapshots_path, current_path);
+    diff_config.set_ignore_right_missing(true);
+    let actions = ActionsImpl();
+    let mut task = Task::new(diff_config, Box::new(actions));
+    task.run(&args)?;


Would it make sense to move more of this into kompari_tasks? If more repositories use Kompari, they'd all duplicate this code.

This was the original plan, but I still want to have a relatively high level of configurability, as it seems that the Vello testing framework works a bit differently than what I implemented for Parley.

tomcur · 2025-02-13T14:48:38Z

Cargo.lock

 [[package]]
 name = "accesskit"
-version = "0.17.0"
+version = "0.17.1"


I'm not sure, for example Cargo.toml still lists accesskit = "0.17", but it's locked to "0.17.1" in the lockfile.

Maybe @DJMcNab can chime in?

xtask/Cargo.toml

spirali · 2025-03-12T11:27:47Z

Kompari updated. It adds cargo xtask size-check that checks optimization opportunities for snapshots.

DJMcNab

I'm happy to just land this; I think there are more improvements to be made (such as using Kompari's diff algorithm internally, which I don't think this does). But it's important to keep this project moving.

All review comments are addressed, as far as I know

Co-authored-by: Tom Churchman <thomas@kepow.org>

DJMcNab

The concern about CI checking for xtask is blocking concern, but otherwise this is looking really good

DJMcNab · 2025-04-23T07:02:08Z

.github/workflows/ci.yml


      - name: cargo clippy
-        run: cargo hack clippy --workspace --locked --target ${{ matrix.target }} --optional-deps --each-feature --ignore-unknown-features --features std -- -D warnings
+        run: cargo hack clippy --package parley --package fontique --locked --target ${{ matrix.target }} --optional-deps --each-feature --ignore-unknown-features --features std -- -D warnings


Hmm. Why do we want to not run clippy on the xtask code itself? Is it just how much longer it takes (since it needs to check axum, etc.), or something else?

I am disabling it only for andoid, as kompari does not compile there because of:

Compiling libdeflate-sys v1.23.1 warning: libdeflate-sys@1.23.1: Compiler family detection failed due to error: ToolNotFound: failed to find tool "x86_64-linux-android-clang": No such file or directory (os error 2) error: failed to run custom build command for `libdeflate-sys v1.23.1`

Ah - that's unfortunate, but not too important at the moment.

Can this instead be --workspace --exclude xtask?

Thanks. I was looking for something like this in the documentation and could not find it so I have named the packages.

DJMcNab · 2025-04-23T07:02:29Z

xtask/README.md

+```bash
+cargo xtask report  # Creates report
+cargo xtask review  # Interactive test blessing
+cargo xtask dead-snaphosts  # Detects dead snapshots


Should we run this in CI?

This PR adds xtask similar to linebender/parley#272 for Parley. Because Vello has two types of tests: snapshots and comparisons (cpu vs. gpu), xtask is divided to two subcommands: ``` cargo xtask snapshots ... cargo xtask comparisons ... ``` The first one offers all Kompari's functionality (report, review, size-check, dead-snapshots detection). The second one offers only report as others do not make sense. Btw1: Current size check output: ![image](https://github.com/user-attachments/assets/3d51a8d1-9b7c-4a99-a0b0-c291a8f7588d) Btw2: Current dead snapshots report (This may be broken as some tests are skipped on my machine) ``` /home/ada/projects/vello/vello_tests/current/big_bitmap.png /home/ada/projects/vello/vello_tests/current/big_bitmap_apple.png /home/ada/projects/vello/vello_tests/current/big_colr.png /home/ada/projects/vello/vello_tests/current/bitmap_undef.png /home/ada/projects/vello/vello_tests/current/blurred_rounded_rect.png /home/ada/projects/vello/vello_tests/current/colr_undef.png /home/ada/projects/vello/vello_tests/current/deep_blend.png /home/ada/projects/vello/vello_tests/current/fill_types.png /home/ada/projects/vello/vello_tests/current/funky_paths.png /home/ada/projects/vello/vello_tests/current/gradient_extend.png /home/ada/projects/vello/vello_tests/current/image_extend_modes_bilinear.png /home/ada/projects/vello/vello_tests/current/image_extend_modes_nearest_neighbor.png /home/ada/projects/vello/vello_tests/current/image_sampling.png /home/ada/projects/vello/vello_tests/current/integer_translation.png /home/ada/projects/vello/vello_tests/current/little_bitmap.png /home/ada/projects/vello/vello_tests/current/little_colr.png /home/ada/projects/vello/vello_tests/current/longpathdash_butt.png /home/ada/projects/vello/vello_tests/current/many_clips.png /home/ada/projects/vello/vello_tests/current/non_integer_translation.png /home/ada/projects/vello/vello_tests/current/rounded_rectangle_watertight.png /home/ada/projects/vello/vello_tests/current/scaled_hinted.png /home/ada/projects/vello/vello_tests/current/simple_hinted.png /home/ada/projects/vello/vello_tests/current/smoke/data_image_roundtrip.png /home/ada/projects/vello/vello_tests/current/smoke/filled_circle.png /home/ada/projects/vello/vello_tests/current/smoke/filled_square.png /home/ada/projects/vello/vello_tests/current/smoke/two_emoji.png /home/ada/projects/vello/vello_tests/current/splash.png /home/ada/projects/vello/vello_tests/current/stroke_styles.png /home/ada/projects/vello/vello_tests/current/stroke_styles_non_uniform.png /home/ada/projects/vello/vello_tests/current/stroke_styles_skew.png /home/ada/projects/vello/vello_tests/current/tricky_strokes.png ```

spirali mentioned this pull request Feb 12, 2025

Report for snapshot testing (diff included) #185

Closed

DJMcNab requested a review from tomcur February 12, 2025 12:40

spirali force-pushed the kompari-xtask branch from 2b99f07 to 78a3551 Compare February 12, 2025 12:42

tomcur previously requested changes Feb 13, 2025

View reviewed changes

tomcur reviewed Feb 13, 2025

View reviewed changes

spirali force-pushed the kompari-xtask branch from ab495fc to b0c92d0 Compare March 21, 2025 18:31

DJMcNab approved these changes Mar 24, 2025

View reviewed changes

spirali force-pushed the kompari-xtask branch from b0c92d0 to 31b09b1 Compare March 25, 2025 15:03

spirali and others added 11 commits April 23, 2025 08:25

Added xtask into Parley; integrates kompari

28f049c

Readme updated

ed06985

Update parley/tests/README.md

e4664af

Co-authored-by: Tom Churchman <thomas@kepow.org>

Added missing file

438ac2f

Reformated by Taplo

bec2d2d

Docs fixed

ec437c6

Fixed copyright header

52ad621

Fixed missing attributes in Cargo.toml

60cac02

xtask publish & version

3b22bf5

Taplo format

ef87e89

Updated version of Kompari

aaaf89a

spirali force-pushed the kompari-xtask branch 2 times, most recently from 7d1c444 to 51176a3 Compare April 23, 2025 06:50

Disable xtask in andoid builds

903f4cc

spirali force-pushed the kompari-xtask branch from 51176a3 to 903f4cc Compare April 23, 2025 06:55

DJMcNab reviewed Apr 23, 2025

View reviewed changes

CI update

bec9d98

spirali added this pull request to the merge queue Apr 24, 2025

Merged via the queue into linebender:main with commit 8f343fa Apr 24, 2025
21 checks passed

spirali deleted the kompari-xtask branch April 24, 2025 12:22

spirali mentioned this pull request May 21, 2025

xtask with Kompari linebender/vello#1019

Merged

Conversation

spirali commented Feb 12, 2025

Creating report

Interactive test blessing:

Detecting dead snapshots

Uh oh!

tomcur left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spirali commented Mar 12, 2025

Uh oh!

DJMcNab left a comment

Choose a reason for hiding this comment

Uh oh!

DJMcNab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DJMcNab Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tomcur left a comment •

edited

Loading

DJMcNab Apr 23, 2025 •

edited

Loading