[cairo] Cairo test comparison 1.12.14 vs 1.12.15~git

[Uli Schlachter]

Hi,

On 13.06.2013 02:22, Bryce W. Harrington wrote:
> What follows is a comparison of the cairo test suite run against
> 1.12.14, and against current git (as of June 12th).
> 
> # Passed:           8983   9202  +219
> # Failed:           3287   3384   +97
> # Expected Failed:   231    298   +85
> # Error:               1      0    -1
> # Crash:             194    232   +38
> # Untested:          575    575    +0

Wow, so many new tests...
(This reminds me that the test suite should explain the reason for "untested"
tests better...)

[...]
> Some of the record tests are now expected to fail (I don't know why):
> 
> PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record90-fill-alpha TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record90-fill-alpha TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record90-fill-alpha TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record90-fill-alpha TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record90-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> PASS -> XFAIL # TEST: record90-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0

Uhm. This is weird. These tests are XFAIL because
test/reference/record1414x-fill-alpha.xfail.png and
record1414x-fill-alpha.base.xfail.png exist. However, these files were last
touched in commit 03df52a833f6 which was already part of 1.11.4. Also, the only
commit touching the test suite was 592f5944239 and reverting this one does not
change the result.

In other words: It doesn't look to me like these tests became XFAILs on purpose
and I can't find anything that would cause this.

> Some of the bug-seams tests had been expected to fail, but are still failing.
> 
> XFAIL -> FAIL # TEST: bug-seams TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> XFAIL -> FAIL # TEST: bug-seams TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> XFAIL -> FAIL # TEST: bug-seams TARGET: recording FORMAT: argb32 OFFSET: 0 SIMILAR: 0
> XFAIL -> FAIL # TEST: bug-seams TARGET: recording FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
> XFAIL -> FAIL # TEST: bug-seams TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0

Same as above: The reference images weren't touched since 1.11.4.

> The pthread-same-source test seems to be crashy:
[...]
> This one is a bit mysterious.  I attempted re-running the test in isolation a few dozen times but it appears
> to always FAIL, both in 1.12.14 and 1.12.15.  So I'm wondering if the PASS in 1.12.14 is spurious.

Yeah, this test does hit a bug, but it only manages to hit it randomly. Oh, the
joy of threads... :-)

I don't have much time currently. If this ever changes, I will try building
1.12.14 to reproduce these results and then perhaps do a git bisect to figure
out what the heck happened. (But I wouldn't mind if someone else would do this
first, volunteers?)

Cheers,
Uli
-- 
- Captain, I think I should tell you I've never
  actually landed a starship before.
- That's all right, Lieutenant, neither have I.