[cairo] Automated testing of Cairo

[Bryce Harrington]

On Fri, Aug 18, 2006 at 03:30:58AM -0700, Carl Worth wrote:
> 	Is cairo creating correct .ps and .svg output and it's just
> 	the tools we are using (gs and svg2png_librsvg) to render it
> 	to a .png that are broken? Or is this a bug in cairo that is
> 	exposed on the systems you are using but not the ones we
> 	usually test cairo on for some reason?
> 
> Bryce, you might even be of some direct help in chasing these
> down. You could pick a text test failure out of an SVG column and go
> look at the -svg-argb32-out.svg file and see if it looks like it
> should render to the same result as the -ref.png image. If not, then
> we've identified a bug in cairo.

I picked out a few of the failed svg cases from this testrun:

   http://crucible.osdl.org/runs/1553/test_output/cairo-test/amd01/test/
 
Several of the cases I saw were rendered incorrectly.  The dashes appear to
be offset a bit compared with the reference.  The gradients seem to not
have the right degree of alpha transparency.

The mask tests look okay to me; the diff image is all black.  Perhaps
there are extremely minor differences that are just summing up since
the test image is dimensionally large?

The push-group tests look identical to me, but if you zoom in closely in
gimp you'll see that there is banding in each of the gradients, but they
appear to be slightly offset from each other.

text-pattern shows some differences in how the letters are being
anti-aliased.  Actually the rendered -svg version looks better to me
than the -ref.

I don't really have enough time to look at all the images, but it
appears to me that most of the failures appear to be legitimate, and
they seem to involve some sort of offset.  Maybe it's a 64-bit issue?

> > One thing I'd suggest is if a test fails, that the testsuite insert some
> > sort of "error" image.
> 
> The test suite should be doing this...

Well, clearly it isn't since there's many broken images.
 
> > http://crucible.osdl.org/runs/1539/test_output/cairo-test/amd01/test/
> >
> > The broken image links don't really communicate that the test itself is
> > failing; instead you wonder if something didn't get copied to the
> > webserver correctly.
> 
> I think something very much like that is happening. For example, one
> of the broken links on that page is to a missing image named:
> 
> 	glyph-cache-pressure-svg-argb32-diff.png
> 
> Meanwhile, by poking around on amd01 I did notice that there's a file
> of that name here:
> 
> /usr/src/cairo-1.2.2-g6122cc85/test/glyph-cache-pressure-svg-argb32-diff.png
> 
> So maybe the image-copying script is picking up the -ref and -out
> images but missing the -diff images or something?

Hmm, I'm not sure; I did a similar analysis and looked for about a dozen
different broken images, and in all cases the files were missing on the
test machine as well.

Besides, if it was an issue with the image copying script, then I'd
expect the broken images to appear more irregularly.  For instance, in
this case, only the svg diff images are missing:

 http://crucible.osdl.org/runs/1526/test_output/cairo-test/nfs11/test/
 
Bryce