[cairo] Got make check to work using CAIRO_REF_DIR, for image target

[Bryce Harrington]

On Thu, Jun 20, 2013 at 01:42:40PM -0400, darxus at chaosreigns.com wrote:
> DISPLAY=:1 CAIRO_TEST_TARGET=image /usr/bin/time make test  # this will fail, 1:04.28elapsed
> # 457 Passed, 44 Failed [0 crashed, 13 expected], 31 Skipped
> #             ^^ Note the number that fail, make sure this changes next
> #                time, or you got CAIRO_REF_DIR wrong.
> 
> The contents of my CAIRO_REF_DIR:
> http://www.chaosreigns.com/wayland/dl/cairo-ref.1.xvfb.tar.gz
> 
> This was done with the current git master, commit
> b7331f0c52cc64f2c224eac502afa6c50a1a8d8b .
> 
> image is only one of many cairo test targets.  It's just the one that
> seemed simplest to get working.  

A full run (on my system) appears to exercise on the order of 13,000
tests, with 2708 failures as of yesterday.  Running it again today to
see where we're at.
 
> Getting cairo's make test to pass against the existing reference images is
> hard.  The test/README documents how you can run make test, copy the output
> images to a new reference directory, make your changes, and then run make
> test again against your own reference images, using CAIRO_REF_DIR.

It's really interesting that this drives down the failure count.  But
presumably those bad reference images are bad for a reason, and should
be corrected so the test suite passes without needing to override them.

I poked through the reference images a bit yesterday to try and
understand the history there, but without any major discoveries.  There
are a handful of pdf bugs mentioned towards the end of test/README, all
of which are marked as fixed upstream, yet the referenced tests are
still failing.  But I can't tell if those tests are failing for the same
reasons they were originally; my guess is that the original issue has
long since been resolved and the tests are failing for a new or
unrelated reason.  I didn't find any .xfail.png images for those
particular tests; poking through git logs it appears they used to be
there (under different naming scheme) but got dropped along the way
(perhaps because they were passing at the time?)

> Xvfb is a software X server recommended in test/README because apparently X
> server driver differences cause problems.
>
> Although after I ran the above, I ran make test again with DISPLAY=:0,
> which is a normal X server with (open source) radeon drivers, and again
> it passed.  I guess that's cool?  I don't actually know if the image target
> hits anything where driver differences could be relevant.

I also get the same results with Xvfb and with a live X driver (radeon).
It might be interesting to compare test results across a few drivers;
perhaps this caveat is no longer necessary?

If there are still some variances due to driver bugs, those issues would
be worth bubbling up to bug trackers.  (I might try to tackle this
testing some time in the future if no one beats me to it.)

Bryce