[cairo] What does it take to get a make check to pass with the xcb target, using CAIRO_REF_DIR?

Thu Jun 23 22:26:42 UTC 2016

A few years ago, I got make check to work with the image target, with help:
https://lists.cairographics.org/archives/cairo/2013-June/024411.html

This still works.  

But I just tried it with the xcb, and the xlib target, running on Xvfb,
creating my own reference directory, and testing against it, and that
fails.  I've read through test/README.  What does it take to get it to
work?

Script that I ran to test it:
#########################################################################
Xvfb :3 &

git clean -xfd
./autogen.sh --prefix=$HOME/install --enable-gl --enable-xcb
make
DISPLAY=:3 CAIRO_TEST_TARGET=xcb make test
cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.1

git clean -xfd
./autogen.sh --prefix=$HOME/install --enable-gl --enable-xcb
make
DISPLAY=:3 CAIRO_REF_DIR=$HOME/source/cairo-ref.xcb.xvfb.1 CAIRO_TEST_TARGET=xcb make test
cp -a test/output/ $HOME/source/cairo-ref.xcb.xvfb.2

git clean -xfd
./autogen.sh --prefix=$HOME/install --enable-gl --enable-xcb
make
DISPLAY=:3 CAIRO_REF_DIR=$HOME/source/cairo-ref.xcb.xvfb.2 CAIRO_TEST_TARGET=xcb make test
#########################################################################

(Yes, I did that an extra time, just to make sure it didn't help.)

The contents of test/output and the reference directory ended up being
basically identical.  There are a few files that only exist in the xvfb.2
directory.  And there are a couple tests that started working on that last
run, which I'm pretty sure shouldn't have changed anything?

< TEST: record2x-paint-alpha-clip-mask TARGET: xcb FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
> TEST: record2x-paint-alpha-clip-mask TARGET: xcb FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS

< TEST: record1414x-paint-alpha-clip-mask TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
> TEST: record1414x-paint-alpha-clip-mask TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS

darxus at dancer:~/source/cairo-ref.xcb.xvfb.1$ grep FAIL *.log | wc -l
2653
darxus at dancer:~/source/cairo-ref.xcb.xvfb.1$ grep PASS *.log | wc -l
2166

darxus at dancer:~/source/cairo-ref.xcb.xvfb.2$ grep FAIL *.log | wc -l
1598
darxus at dancer:~/source/cairo-ref.xcb.xvfb.2$ grep PASS *.log | wc -l
2118

darxus at dancer:~/source/cairo/test/output$ grep FAIL *.log | wc -l
1590
darxus at dancer:~/source/cairo/test/output$ grep PASS *.log | wc -l
2126

I'm not sure what information is useful to give you.

a1-bug.log:
Testing a1-bug with xcb target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/a1-bug.xcb.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/a1-bug.xcb.argb32.out.png
DIFFERENCE: output/a1-bug.xcb.argb32.diff.png
TEST: a1-bug TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing a1-bug with xcb target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/a1-bug.xcb.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/a1-bug.xcb.rgb24.out.png
DIFFERENCE: output/a1-bug.xcb.rgb24.diff.png
TEST: a1-bug TARGET: xcb FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing a1-bug with xcb-window target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: a1-bug TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing a1-bug with xcb-window& target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: a1-bug TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing a1-bug with xcb-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/a1-bug.xcb-render-0_0.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/a1-bug.xcb-render-0_0.argb32.out.png
DIFFERENCE: output/a1-bug.xcb-render-0_0.argb32.diff.png
TEST: a1-bug TARGET: xcb-render-0_0 FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing a1-bug with xcb-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/a1-bug.xcb-render-0_0.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/a1-bug.xcb-render-0_0.rgb24.out.png
DIFFERENCE: output/a1-bug.xcb-render-0_0.rgb24.diff.png
TEST: a1-bug TARGET: xcb-render-0_0 FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing a1-bug with xcb-fallback target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: a1-bug TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL

overlapping-boxes.log:
Testing overlapping-boxes with xcb target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/overlapping-boxes.xcb.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/overlapping-boxes.xcb.argb32.out.png
DIFFERENCE: output/overlapping-boxes.xcb.argb32.diff.png
TEST: overlapping-boxes TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing overlapping-boxes with xcb target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/overlapping-boxes.xcb.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/overlapping-boxes.xcb.rgb24.out.png
DIFFERENCE: output/overlapping-boxes.xcb.rgb24.diff.png
TEST: overlapping-boxes TARGET: xcb FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing overlapping-boxes with xcb-window target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: overlapping-boxes TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing overlapping-boxes with xcb-window& target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: overlapping-boxes TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing overlapping-boxes with xcb-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/overlapping-boxes.xcb-render-0_0.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/overlapping-boxes.xcb-render-0_0.argb32.out.png
DIFFERENCE: output/overlapping-boxes.xcb-render-0_0.argb32.diff.png
TEST: overlapping-boxes TARGET: xcb-render-0_0 FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing overlapping-boxes with xcb-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/overlapping-boxes.xcb-render-0_0.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/overlapping-boxes.xcb-render-0_0.rgb24.out.png
DIFFERENCE: output/overlapping-boxes.xcb-render-0_0.rgb24.diff.png
TEST: overlapping-boxes TARGET: xcb-render-0_0 FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing overlapping-boxes with xcb-fallback target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: overlapping-boxes TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL

surface-pattern-scale-down-extend-none.log:
Testing surface-pattern-scale-down-extend-none with xcb target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/surface-pattern-scale-down-extend-none.xcb.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/surface-pattern-scale-down-extend-none.xcb.argb32.out.png
DIFFERENCE: output/surface-pattern-scale-down-extend-none.xcb.argb32.diff.png
TEST: surface-pattern-scale-down-extend-none TARGET: xcb FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing surface-pattern-scale-down-extend-none with xcb target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/surface-pattern-scale-down-extend-none.xcb.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/surface-pattern-scale-down-extend-none.xcb.rgb24.out.png
DIFFERENCE: output/surface-pattern-scale-down-extend-none.xcb.rgb24.diff.png
TEST: surface-pattern-scale-down-extend-none TARGET: xcb FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing surface-pattern-scale-down-extend-none with xcb-window target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: surface-pattern-scale-down-extend-none TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing surface-pattern-scale-down-extend-none with xcb-window& target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: surface-pattern-scale-down-extend-none TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing surface-pattern-scale-down-extend-none with xcb-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/surface-pattern-scale-down-extend-none.xcb-render-0_0.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/surface-pattern-scale-down-extend-none.xcb-render-0_0.argb32.out.png
DIFFERENCE: output/surface-pattern-scale-down-extend-none.xcb-render-0_0.argb32.diff.png
TEST: surface-pattern-scale-down-extend-none TARGET: xcb-render-0_0 FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing surface-pattern-scale-down-extend-none with xcb-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/surface-pattern-scale-down-extend-none.xcb-render-0_0.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xcb.xvfb.2/surface-pattern-scale-down-extend-none.xcb-render-0_0.rgb24.out.png
DIFFERENCE: output/surface-pattern-scale-down-extend-none.xcb-render-0_0.rgb24.diff.png
TEST: surface-pattern-scale-down-extend-none TARGET: xcb-render-0_0 FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing surface-pattern-scale-down-extend-none with xcb-fallback target (dev offset 0 scale: 1)
Error: Created an error surface: invalid value for an input cairo_format_t
TEST: surface-pattern-scale-down-extend-none TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL

Examples from the doing the same thing with the xlib target:

record1414x-fill-alpha.log:
Testing record1414x-fill-alpha with xlib target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/record1414x-fill-alpha.xlib.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/record1414x-fill-alpha.xlib.argb32.out.png
DIFFERENCE: output/record1414x-fill-alpha.xlib.argb32.diff.png
TEST: record1414x-fill-alpha TARGET: xlib FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing record1414x-fill-alpha with xlib target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/record1414x-fill-alpha.xlib.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/record1414x-fill-alpha.xlib.rgb24.out.png
DIFFERENCE: output/record1414x-fill-alpha.xlib.rgb24.diff.png
TEST: record1414x-fill-alpha TARGET: xlib FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing record1414x-fill-alpha with xlib-window target (dev offset 0 scale: 1)
Error: Created surface has content 0 (expected 4096)
TEST: record1414x-fill-alpha TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing record1414x-fill-alpha with xlib-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/record1414x-fill-alpha.xlib-render-0_0.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/record1414x-fill-alpha.xlib-render-0_0.rgb24.out.png
DIFFERENCE: output/record1414x-fill-alpha.xlib-render-0_0.rgb24.diff.png
TEST: record1414x-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing record1414x-fill-alpha with xlib-fallback target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/record1414x-fill-alpha.xlib-fallback.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/record1414x-fill-alpha.xlib-fallback.rgb24.out.png
DIFFERENCE: output/record1414x-fill-alpha.xlib-fallback.rgb24.diff.png
TEST: record1414x-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS

recordflip-paint-alpha.log:
Testing recordflip-paint-alpha with xlib target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/recordflip-paint-alpha.xlib.argb32.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/recordflip-paint-alpha.xlib.argb32.out.png
DIFFERENCE: output/recordflip-paint-alpha.xlib.argb32.diff.png
TEST: recordflip-paint-alpha TARGET: xlib FORMAT: argb32 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing recordflip-paint-alpha with xlib target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/recordflip-paint-alpha.xlib.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/recordflip-paint-alpha.xlib.rgb24.out.png
DIFFERENCE: output/recordflip-paint-alpha.xlib.rgb24.diff.png
TEST: recordflip-paint-alpha TARGET: xlib FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing recordflip-paint-alpha with xlib-window target (dev offset 0 scale: 1)
Error: Created surface has content 0 (expected 4096)
TEST: recordflip-paint-alpha TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: FAIL
Testing recordflip-paint-alpha with xlib-render-0_0 target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/recordflip-paint-alpha.xlib-render-0_0.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/recordflip-paint-alpha.xlib-render-0_0.rgb24.out.png
DIFFERENCE: output/recordflip-paint-alpha.xlib-render-0_0.rgb24.diff.png
TEST: recordflip-paint-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS
Testing recordflip-paint-alpha with xlib-fallback target (dev offset 0 scale: 1)
PNG file exactly matches reference image.
OUTPUT: output/recordflip-paint-alpha.xlib-fallback.rgb24.out.png
REFERENCE: /home/darxus/source/cairo-ref.xlib.xvfb.2/recordflip-paint-alpha.xlib-fallback.rgb24.out.png
DIFFERENCE: output/recordflip-paint-alpha.xlib-fallback.rgb24.diff.png
TEST: recordflip-paint-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SCALE: 1 SIMILAR: 0 RESULT: PASS