[cairo] How can I find real FAIL in result of cairo-test-suite?

[Chris Wilson]

On Wed, 22 Feb 2012 05:25:51 +0000 (GMT), Deokjin Kim <deokjin81.kim at samsung.com> wrote:
> Hello all,
> 
> When I executed cairo-test-suite for image backend in my environment, the result is below.
> 376 Passed, 61 Failed [0 crashed, 10 expected], 26 Skipped
> 
> As I know, some failures are real FAIL(may be caused by cairo itself) and others are caused by test environment(difference of library version, font files, and so on).
> How can I distinguish real FAIL from non-real FAIL?

Experience, i.e. by knowing what the test is and how cairo should
behave. One short-cut to gaining that inside knowledge is to run with
the base compositor (the test-base surface) and comparing. The difference
between that compositor and the current image compositor will be in the
antialiasing of geometry and the use of clip masks. Basically anything that
appears to be antialiasing differences is typically just noise unless you
know you have altered those algorithms, differences around the edges of
glyphs are that you just have a different font and/or fontconfig
overrides. At this moment in time, you should see 6 failures in the
image backend, 2 critical clipping bugs (which are likely to be the same
in different guises) and 4 bogons caused by replaying recording surfaces
through tiles.
 
> Can I know test environment when you create reference images?

x86-64 debian sid, with ttf-dejavu 2.33-2:
93502d0d0445d1fe1c9f51e51b3e0169266346ce /usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf
1cd336329f45f241002ded61893d91e3acd04436 /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
0458c0f0fb57f3eb8ced62f26fe7c5ed4e6a9a68 /usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf

> Content of README file("Getting the elusive zero failures") is insufficient.

I've just pushed the updated sha1sums to the README as well.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre