[cairo] Cairo test comparison 1.12.14 vs 1.12.15~git

[Bryce W. Harrington]

What follows is a comparison of the cairo test suite run against
1.12.14, and against current git (as of June 12th).

# Passed:           8983   9202  +219
# Failed:           3287   3384   +97
# Expected Failed:   231    298   +85
# Error:               1      0    -1
# Crash:             194    232   +38
# Untested:          575    575    +0

The test runs were done sequentially on the same machine, so
there is no variation in dependency libraries.  For each release
I used the test suite as included in that release.  The test
machine is running:

  Ubuntu 12.04 LTS
  X Server 1.11.3
  Linux kernel 3.2.0-37
  mesa 8.0.4

Of the tests present in both releases that actually ran, most
results were the same.

Of the tests which changed results, most changed for the better:

ERROR -> FAIL # TEST: xcb-huge-image-shm TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: image16 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: ps2 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: ps3 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: recording FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: recording FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: svg11 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: svg12 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xlib FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: bitmap-font TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: clip-text TARGET: pdf FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: clip-text TARGET: pdf FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: clip-text TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: device-offset TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: linear-gradient-subset TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: linear-gradient-subset TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: linear-gradient TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: linear-gradient TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: pdf-surface-source TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: pdf-surface-source TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: pdf-surface-source TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: pdf-surface-source TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: ps-surface-source TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: ps-surface-source TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: ps-surface-source TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: ps-surface-source TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: smask-fill TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: smask-fill TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: smask-mask TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: smask-mask TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: smask-paint TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: smask-paint TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: svg-surface-source TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: svg-surface-source TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: svg-surface-source TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: svg-surface-source TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: text-pattern TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> PASS # TEST: text-pattern TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: image16 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: pdf FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: pdf FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: ps2 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: ps2 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: ps3 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: ps3 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: recording FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: recording FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: image16 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: pdf FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: pdf FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: ps2 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: ps2 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: ps3 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: ps3 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: recording FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: recording FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: image16 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: pdf FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: pdf FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: ps2 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: ps2 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: ps3 FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: ps3 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: recording FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: recording FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xcb FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xcb FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xcb-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xcb-window& FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xlib FORMAT: argb32 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xlib FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> XFAIL # TEST: record90-fill-alpha TARGET: xlib-window FORMAT: rgb24 OFFSET: 0 SIMILAR: 0

Some of the record tests are now expected to fail (I don't know why):

PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record1414x-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record2x-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record90-fill-alpha TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record90-fill-alpha TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record90-fill-alpha TARGET: script FORMAT: argb32 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record90-fill-alpha TARGET: xcb-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record90-fill-alpha TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
PASS -> XFAIL # TEST: record90-fill-alpha TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0

Some of the bug-seams tests had been expected to fail, but are still failing.

XFAIL -> FAIL # TEST: bug-seams TARGET: image FORMAT: argb32 OFFSET: 0 SIMILAR: 0
XFAIL -> FAIL # TEST: bug-seams TARGET: image FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
XFAIL -> FAIL # TEST: bug-seams TARGET: recording FORMAT: argb32 OFFSET: 0 SIMILAR: 0
XFAIL -> FAIL # TEST: bug-seams TARGET: recording FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
XFAIL -> FAIL # TEST: bug-seams TARGET: xlib-fallback FORMAT: rgb24 OFFSET: 0 SIMILAR: 0

The pthread-same-source test seems to be crashy:

FAIL -> CRASHED # TEST: pthread-same-source TARGET: ps3 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0
FAIL -> CRASHED # TEST: pthread-same-source TARGET: svg11 FORMAT: argb32 OFFSET: 0 SIMILAR: 0

However, in running this test repeatedly both against 1.12.14 and 1.12.15, in both cases it seems to CRASH
50% of the time and FAIL the rest.  So it's just random variation that this results combination would show up.
Thus this doesn't really count as a behavior change.

PASS -> FAIL # TEST: ps-surface-source TARGET: xlib-render-0_0 FORMAT: rgb24 OFFSET: 0 SIMILAR: 0

This one is a bit mysterious.  I attempted re-running the test in isolation a few dozen times but it appears
to always FAIL, both in 1.12.14 and 1.12.15.  So I'm wondering if the PASS in 1.12.14 is spurious.


--
Bryce Harrington
Senior Open Source Developer  -  b.harrington at samsung.com
Open Source Group             -  Samsung Research America