[cairo] How can I find real FAIL in result of cairo-test-suite?

[Deokjin Kim]

Hello Chris,

When I tested image backend with ttf-dejavu font files, I still had many failures.
    376 Passed, 62 Failed [0 crashed, 12 expected], 25 Skipped
For reference, below is the result of test base compositor.
    400 Passed, 38 Failed [0 crashed, 5 expected], 25 Skipped
I want to know which test case is real FAIL(caused by cairo itself).

For example, I have tested with a1-bug.
As I know, image which is produced by image compositor is compared with a1-bug.ref.png.
Then a1-bug.ref.png file is made by what kind of compositor? (Do you make reference images using test base compositor?)
When I checked git log, you updated reference images sometimes.

Please explain your environment for reference images.

Thank you,
Deokjin Kim
------- Original Message -------
Sender : Chris Wilson<chris at chris-wilson.co.uk>
Date : 2012-02-22 18:34 (GMT+09:00)
Title : Re: [cairo] How can I find real FAIL in result of cairo-test-suite?

On Wed, 22 Feb 2012 05:25:51 +0000 (GMT), Deokjin Kim wrote:
> Hello all,
> 
> When I executed cairo-test-suite for image backend in my environment, the result is below.
> 376 Passed, 61 Failed [0 crashed, 10 expected], 26 Skipped
> 
> As I know, some failures are real FAIL(may be caused by cairo itself) and others are caused by test environment(difference of library version, font files, and so on).
> How can I distinguish real FAIL from non-real FAIL?

Experience, i.e. by knowing what the test is and how cairo should
behave. One short-cut to gaining that inside knowledge is to run with
the base compositor (the test-base surface) and comparing. The difference
between that compositor and the current image compositor will be in the
antialiasing of geometry and the use of clip masks. Basically anything that
appears to be antialiasing differences is typically just noise unless you
know you have altered those algorithms, differences around the edges of
glyphs are that you just have a different font and/or fontconfig
overrides. At this moment in time, you should see 6 failures in the
image backend, 2 critical clipping bugs (which are likely to be the same
in different guises) and 4 bogons caused by replaying recording surfaces
through tiles.

> Can I know test environment when you create reference images?

x86-64 debian sid, with ttf-dejavu 2.33-2:
93502d0d0445d1fe1c9f51e51b3e0169266346ce /usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf
1cd336329f45f241002ded61893d91e3acd04436 /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
0458c0f0fb57f3eb8ced62f26fe7c5ed4e6a9a68 /usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf

> Content of README file("Getting the elusive zero failures") is insufficient.

I've just pushed the updated sha1sums to the README as well.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre