Graphics testing

[rocky]

Description

Recent changes seem to have broken non-vector-mode tick marks in ListPlot[]

How to Reproduce

mathicsscript --matplotlib -c 'ListPlot[{1, 1, 2}]'

Output Given

Expected behavior

No logplot scale, y axis goes from 0 to 2.

Your Environment

Current Mathics core development environment 9.0.1.dev0 and same for mathicsscript
mathics --version

Workarounds

Use last release ;-)

[bdlucas1]

See #1555 for analysis and fix.

We have a gap in testing, in that no (or maybe just a few?) tests check the generated SVG graphics. The tests in test_plot_detail.py can easily be enhanced to check that the SVG graphics for all the doctests aren't changed. If you'd like we can leave this issue open, assigned to me, to do that.

[bdlucas1]

I also submitted PR #1556, which is #1555 plus SVG tests. Please double-check one or the other, depending on whether you're ok with adding the SVG tests. I think they will help prevent mishaps like this in the future.

[bdlucas1]

As noted on PR #1556, it's probably too fragile, so suggest just going with PR #1555 at this point.

[rocky]

As noted on PR #1556, it's probably too fragile, so suggest just going with PR #1555 at this point.

Ok. As often happens here, there are good ideas, e. g., more robust testing or using compilation (or a more aggressive transformation) to speed up computation. But these ideas need iteration over time to get right.

I haven't been following the testing process in detail, other than we're testing at the S-expression level (as opposed to a rendered SVG level), which is a good thing.

The obvious things that would seem to be needed are:

• some way to indicate tolerance on numeric values, and possibly
• checking counts of data items when things can vary widely (e. g., all we careabout are there are n points, or n points that range between, x and y.)
• and as I alluded to in another PR, a GitHub workflows CI regime that works independently of other tests so that certain environment aspects can be isolated (version of numpy used, special OS environments, and Python versions). Isolating CI testing also allows the graphics tests, which will probably grow a bit, to be run in parallel with the other tests rather than in sequence with them.

[bdlucas1]

Some thoughts and experiences regarding testing plotting functions:

There are three places in the graphics pipeline where we could do testing:

• The Graphics* expression that is output from Plot*. This is what the
  tests in this directory currently do, in more or less detail. The
  more detailed tests that were recently added are based on doing a
  simple diff of a pretty-printed indented outline-form text
  representation of the Graphics* output compared to a reference
  file.

  This was initially found to be fragile because of inconsequential
  numerical differences on different platforms, but that was brought
  under control by limiting the precision and type information of the
  text representation of the Graphics* expression.

  I have found this to be effective at finding problems and easy to
  understand - determine whether a change is acceptable or harmful -
  when a code change changes the output.

• Here the pipeline splits into multiple options:

  • SVG output via core. This tests both Plot*->Graphics* and
    Graphics*->SVG code. Simple textual comparison of actual vs
    expected result was tried in this PR and immediately found to be
    fragile due to the insane precision of the numbers emitted to
    the SVG.

    Possibly this could be brought under control by limiting the
    precision of the numbers (as was done for the above Graphics*
    based tests), which would require finding all the spots in SVG
    rendering where numbers are emitted. However after doing so
    interpreting differences might be difficult due to the complex
    structure of SVG.

    A pretty-printer that displays SVG in a deconstructed indented
    outline form (as we do for the above Graphics*) that could
    easily be diffed might facilitate this.

  • High-performance rendering output in m3d via
    Plotly+Panel. There's no direct equivalent of SVG in this
    pipeline. The closest would be the internal data structures used
    by Plotly+Panel, but I would expect that to be fragile, subject
    to change, and difficult to interpret.

• The final rendered image. This is of course what the user cares
  about. We don't have this in our core testing, but I have been doing
  this in m3d as I do development in core for both pipelines
  (Plot*->Graphics*->SVG->PNG and Plot*->Graphics*->Plotly->PNG) for
  all the doctests and more. I (somewhat to my surprise) have found
  these tests to be the most robust, because they are insensitive to
  inconsequential numerical or other differences in the Graphics* or
  the SVG, partly because images have very limited precision, both
  spatially and in color depth.

  As for interpreting the results, in m3d I have a semi-automated UI
  that, when it detects a difference, displays the actual image, the
  reference image, and a diff image side-by-side, with a button
  offering to accept the actual output and save it as the new
  reference output. I have found this to be pretty useful for
  development testing.

  I don't know how robust the generated images are from system to
  system. My hunch is they might be portable due to the limited
  precision of images. I think it might be worth the experiment to set
  up a separate CI workflow where we can install the required software
  to generate images (initially, and maybe only, via the core SVG
  path), and try it out.

I think these tests are all useful and are complementary. For example,
if the first Plot*->Graphics* tests show that some code change has
changed the Graphics* but you think it is inconsequential, or is
expected, and image-based comparison can serve as a check on whether
that is so. Conversely, lacking effective testing on the SVG output,
an image comparison can flag unintended changes in the SVG generation
code.

Regarding the idea to do more sophisticated comparisons than simple
textual diffs, I tried something in that direction in test_plot.py
(see eval_and_check_structure). I found that to be difficult to set
up, and difficult to interpret the results. Probably that's because I
wasn't doing it right, but I couldn't figure out a better way to do
it.

On the other hand I found the doctest-based tests that were recently
added to be easy to set up, easy to interpret, and they give broad coverage.

[rocky]

I have found this to be effective at finding problems and easy to
understand - determine whether a change is acceptable or harmful -
when a code change changes the output.

As you say, it is also fragile and voluminous, both in the codebase and in changelogs. It doesn't really give us insight as to the changing nature of the underlying system, that numpy on some OSs and some Python versions and some numpy versions can use integers or floats (if that's the case), and that's okay.

While this kind of thing might be captured with the addition of comments, I noted that when "3" was changed to "3.1" no comment appeared there, but instead in a place where it is less likely to be seen.

With more work on a specific graphics testing system, it could be more declarative: that certain error bounds on values are okay, and maybe certain kinds of datatypes too (both float, and int, i.e., just numbers versus, say, must always be int). Or there could be a declaration of the dimensions and sizes of data. Explicit declaration is better than implicit, for example using 3.1 to try to constrain output to be floating point.

A pretty-printer that displays SVG in a deconstructed indented outline form (as we do for the above Graphics*) that could
easily be diffed might facilitate this.

The mathics-core SVG implementation currently always outputs in some kind of pretty-printed form for this reason.

High-performance rendering output in m3d via Plotly+Panel. There's no direct equivalent of SVG in this pipeline. The closest would be the internal data structures use by Plotly+Panel, but I would expect that to be fragile, subject to change, and difficult to interpret.

Testing of Plotly+Panel or any kind of add-on should be done inside the add-on system. It might be okay to have such individual tests inside the core if they run on demand, or on a time-based schedule, as opposed to running after every little commit.

And large-scale graphics should probably be isolated to its own test suite which is run in parallel rather than in sequence as part of the usual tests, which are expected to run relatively quickly.

On the other hand I found the doctest-based tests that were recently
added to be easy to set up, easy to interpret, and they give broad coverage.

I don't know what exactly you mean by doctest-based tests. Keep in mind that, in the long term, the homegrown doctest system will be replaced with something more standard. So extending our homegrown system is not a good idea, but if you want to extend something based on, say, Sphinx, or a non-homegrown system that is better suited for these kinds of things, that would be fine.

[bdlucas1]

I noted that when "3" was changed to "3.1" no comment appeared there, but instead in a place where it is less likely to be seen.

Fair point. This would make a great comment on the PR.

that certain error bounds on values are okay

That was the effect of limiting the precision in the textual output for the Plot*->Graphics* tests.

and maybe certain kinds of datatypes too (both float, and int, i.e., just numbers versus, say, must always be int).

That could be accomplished in the system based on text diffs by outputting "number" instead of the actual datatype. I contemplated doing that as a result of the case you mention above where 3 appeared as int on some systems and float on others, but opted not to because so far I've only seen this in that one test case. If this (int vs float confusion) occurs again I will probably make that change.

The mathics-core SVG implementation currently always outputs in some kind of pretty-printed form for this reason.

It's printed one SVG element per line, which is a start, but some of those lines can get quite long (e.g. a long polyline with many points is on a single line, along with other attributes describing color, style, etc.). What I had in mind is that for more effective diffing it could be printed in a form that shows each attribute on a separate line, indented to show containment, and breaks down points attributes so that each point is on a separate line. (I'm not necessarily suggesting that svg normally be generated in this form - I'm describing the form used for diffing in tests). But that, along with limiting precision, is work (and I'm not planning on doing it, at least not in the near future).

Testing of Plotly+Panel or any kind of add-on should be done inside the add-on system.

Right, agree. I just mentioned that case here for completeness. What could be done easily in core is the Plot*->Graphics*->SVG->PNG image-based test. (This requires something like cairosvg, so assuming that won't be made a core requirement the test would have to be conditioned on its availablity. TBD when and how it gets run.)

I don't know what exactly you mean by doctest-based tests.

Yes, sorry, that was vague and a bit of a misnomer. In the plot testing directory there is a file doc_tests.yaml that describes 120 or so tests originally obtained from doctests. But the mechanism (and the current set of tests) is not specific to doctests: the test case is run, the result is pretty-printed in outline form, and diffed against a reference file.

So what I was saying is that in practice I have found that mechanism effective: easy to set up, easy to interpret, and it gives very broad coverage.