[cairo] What does it take to get a make check to pass with the xcb target, using CAIRO_REF_DIR?

[Bryce Harrington]

On Wed, Jun 29, 2016 at 02:21:48PM -0400, darxus at chaosreigns.com wrote:
> At pq's suggestion, I tried running the tests twice against the same
> freshly generated reference set.  The failures were different between each
> run.
> 
> 
> For example:
> 
> Testing record2x-paint-alpha-clip-mask with xcb target (dev offset 0 scale: 1)
> 16 pixels differ (with maximum difference of 64) from reference image
> 
> Testing record2x-paint-alpha-clip-mask with xcb target (dev offset 0 scale: 1)
> 15 pixels differ (with maximum difference of 64) from reference image
> 
> < TEST: record1414x-paint-alpha-clip-mask TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
> > TEST: record1414x-paint-alpha-clip-mask TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
> 
> (Number of pixels which differ varied, and a couple tests switched between
> PASS and FAIL.)
> 
> 
> Example output from two identical test runs:
> 
> http://www.chaosreigns.com/tmp/cairo/cairo-ref.xcb.xvfb.2/record2x-paint-alpha-clip-mask.xcb.argb32.out.png
> http://www.chaosreigns.com/tmp/cairo/cairo-ref.xcb.xvfb.3/record2x-paint-alpha-clip-mask.xcb.argb32.out.png
> 
> In test/reference/record2x-paint-alpha-clip-mask.base.argb32.ref.png you
> can see that the four spots at the corners of the octagon are not supposed
> to be there.  The variation between the two test output images is in the
> extent of the glitching in the upper left spot.
> 
> 
> To see if the problem was specific to Xvfb, I re-ran the xcb tests on a
> regular X server (with nothing but an xterm running on it) with a Radeon
> driver.  Just like with Xvfb, the number of pixels differing varied, and
> tests switched between PASS and FAIL.  When running against the same
> reference set, and without any changes.
> 
> 
> With the default target set, running against a 24 bit Xvfb, with a
> reference set generated identically to the test results:
> 
> Target            Tests   Failures
> image              1066   2
> image16             533   1
> ps                    1
> ps2                1066   7
> ps3                1066   7
> recording          1066   4
> script              533   9
> xcb                1066   6
> xcb-fallback        533   1
> xcb-render-0_0     1066   2
> xcb-window          533   5
> xcb-window&         533   5
> xlib               1066  14
> xlib-fallback       533   1
> xlib-render-0_0     533  12
> xlib-window         533   7
> 
> (The image tests pass with an 8 bit Xvfb, but 2 of them crash with a 24 bit
> Xvfb, which substantially improves xcb results.)
> 
> 
> When was the last time these things passed?  Any ideas where to begin
> tracking down the problem?
> 
> 
> <pq> Darxus, if you ran it with different Cairo backends, or
> different hardware or different drivers, that might explain some, but
> really identical runs - that's very bad.

We do know there are certain tests with race conditions that will
alternate between passing and failing in two identical, sequential test
runs.  Those test cases you mention don't ring a bell, but if you see
it's the same set doing that each time, then that could be the case.  I
don't know what to do about those, and would not oppose disabling those
test cases (although I assume they were meant to test something
important...)
 
> <ebassi> It did use to pass its own test suite, until the downscaling
> algorithm was tweaked
> <Darxus> ebassi: When was the downscaling algorith tweaked?
> <ebassi> Darxus: Two or three years ago

Perhaps a small subset of the test suite passed at one point but,
general test runs have resulted in over half the test cases failing for
as long as I've been involved in the project.

Changing the downscaling algorithm did affect tests (of course) but I
recall those were dealt with at the time.  There may be some lingering
side effects there or reference images needing regenerated, and perhaps
some people still hold some discomfort about the change generally, but I
suspect that is a red herring for what you're looking at.  Easy enough
to prove though - you should be able to revert that specific change
fairly readily (I don't think there's been disruptive changes to that
chunk of code) and re-run your tests.

I wish I understood the causes of all the test breakages and failures.
In looking into it myself several times, it seems there are multiple
different factors at play, including some that are fundamental design
decisions that would be hard to change.  I am still uncertain about
whether it is appropriate to regenerate the reference images before
running the test suite, or if that might create false positives.  I
suspect a large chunk of the breakages are due to changes outside of
Cairo proper, which if true means that a lot of the references *should*
be regenerated, but would also suggest there's more complexity in the
render chain than the test suite is able to cover in a maintainable
level of coverage.

So I don't think there's any simple answers that would make everything
Just Work, and chasing down the causes for some of the issues look like
they might be a considerable amount of effort (and probably not result
in any tangible fixes other than maybe tweaks to the testsuite itself).
I might be wrong though, and would be happy if I am, and hope someone
could prove me so.

Bryce

> The script I used to determine that output varied when the reference set
> remained the same, took 43 minutes to run:
> 
> 
> Xvfb -screen 0 1680x1024x24 -ac -nolisten tcp :2 &
> 
> git clean -xfd
> ./autogen.sh --prefix=$HOME/install --enable-xcb
> make
> DISPLAY=:2 CAIRO_TEST_TARGET=xcb make test
> cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.1
> 
> git clean -xfd
> ./autogen.sh --prefix=$HOME/install --enable-xcb
> make
> DISPLAY=:2 CAIRO_REF_DIR=$HOME/source/cairo-ref.xcb.xvfb.1 CAIRO_TEST_TARGET=xcb make test
> cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.2
> 
> git clean -xfd
> ./autogen.sh --prefix=$HOME/install --enable-xcb
> make
> DISPLAY=:2 CAIRO_REF_DIR=$HOME/source/cairo-ref.xcb.xvfb.1 CAIRO_TEST_TARGET=xcb make test
> cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.3
> -- 
> cairo mailing list
> cairo at cairographics.org
> https://lists.cairographics.org/mailman/listinfo/cairo