[cairo] What does it take to get a make check to pass with the xcb target, using CAIRO_REF_DIR?

[darxus at chaosreigns.com]

At pq's suggestion, I tried running the tests twice against the same
freshly generated reference set.  The failures were different between each
run.


For example:

Testing record2x-paint-alpha-clip-mask with xcb target (dev offset 0 scale: 1)
16 pixels differ (with maximum difference of 64) from reference image

Testing record2x-paint-alpha-clip-mask with xcb target (dev offset 0 scale: 1)
15 pixels differ (with maximum difference of 64) from reference image

< TEST: record1414x-paint-alpha-clip-mask TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
> TEST: record1414x-paint-alpha-clip-mask TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL

(Number of pixels which differ varied, and a couple tests switched between
PASS and FAIL.)


Example output from two identical test runs:

http://www.chaosreigns.com/tmp/cairo/cairo-ref.xcb.xvfb.2/record2x-paint-alpha-clip-mask.xcb.argb32.out.png
http://www.chaosreigns.com/tmp/cairo/cairo-ref.xcb.xvfb.3/record2x-paint-alpha-clip-mask.xcb.argb32.out.png

In test/reference/record2x-paint-alpha-clip-mask.base.argb32.ref.png you
can see that the four spots at the corners of the octagon are not supposed
to be there.  The variation between the two test output images is in the
extent of the glitching in the upper left spot.


To see if the problem was specific to Xvfb, I re-ran the xcb tests on a
regular X server (with nothing but an xterm running on it) with a Radeon
driver.  Just like with Xvfb, the number of pixels differing varied, and
tests switched between PASS and FAIL.  When running against the same
reference set, and without any changes.


With the default target set, running against a 24 bit Xvfb, with a
reference set generated identically to the test results:

Target            Tests   Failures
image              1066   2
image16             533   1
ps                    1
ps2                1066   7
ps3                1066   7
recording          1066   4
script              533   9
xcb                1066   6
xcb-fallback        533   1
xcb-render-0_0     1066   2
xcb-window          533   5
xcb-window&         533   5
xlib               1066  14
xlib-fallback       533   1
xlib-render-0_0     533  12
xlib-window         533   7

(The image tests pass with an 8 bit Xvfb, but 2 of them crash with a 24 bit
Xvfb, which substantially improves xcb results.)


When was the last time these things passed?  Any ideas where to begin
tracking down the problem?


<pq> Darxus, if you ran it with different Cairo backends, or
different hardware or different drivers, that might explain some, but
really identical runs - that's very bad.

<ebassi> It did use to pass its own test suite, until the downscaling
algorithm was tweaked
<Darxus> ebassi: When was the downscaling algorith tweaked?
<ebassi> Darxus: Two or three years ago


The script I used to determine that output varied when the reference set
remained the same, took 43 minutes to run:


Xvfb -screen 0 1680x1024x24 -ac -nolisten tcp :2 &

git clean -xfd
./autogen.sh --prefix=$HOME/install --enable-xcb
make
DISPLAY=:2 CAIRO_TEST_TARGET=xcb make test
cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.1

git clean -xfd
./autogen.sh --prefix=$HOME/install --enable-xcb
make
DISPLAY=:2 CAIRO_REF_DIR=$HOME/source/cairo-ref.xcb.xvfb.1 CAIRO_TEST_TARGET=xcb make test
cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.2

git clean -xfd
./autogen.sh --prefix=$HOME/install --enable-xcb
make
DISPLAY=:2 CAIRO_REF_DIR=$HOME/source/cairo-ref.xcb.xvfb.1 CAIRO_TEST_TARGET=xcb make test
cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.3