[cairo] Automated testing of Cairo

[Carl Worth]

On Wed, 16 Aug 2006 13:00:54 -0700, Bryce Harrington wrote:
>
> Ah cool.  The bit I was unsure about was making it report 1.2.3 instead
> of 1.2.2, but if you're okay with it showing 1.2.2 in the above, that's
> great.  :-)

Oh, that. No, I still don't like the version being wrong. ;-) But now
that I understand what you were referring to, let me go back and
re-read what you wrote:

> >                    If it gets really annoying, let me know; I'll need to
> > rewrite that code anyway at some point.

OK. Let's leave it as an in-the-future thing. But it would be nice to
have fixed. For example, I'd like it to be clear from the report which
runs are from releases and which are from random in-development points
later on. For example, in our testing of 1.2.2 at release time, we had
no unexpected failures, (at least on the systems we used). But since
1.2.2 we do have some expected failures in the test suite. So it would
be nice to have the report make that story clear with the version
numbers it uses.

It doesn't really matter for now since we don't yet have _any_
versions of 1.2 on that page that are showing no failures. ;-)

>   http://crucible.osdl.org/runs/cairo_branches.html
>
> The numbers shown are (fails / fails+passes).

This looks good. Particularly, now since I had you count only separate
tests and not tests X backends X 2 content X 2 offsets. The only funny
things I still see is that the count of total tests is differing
across the systems even for the same version of cairo. A very small
difference, (5 tests or fewer), is actually expected if there are
different backends enabled on some systems. This is because there are
a few backend-specific tests that won't be run at all if the backend
isn't enabled. So 88 tests without xlib and 89 tests with xlib looks
fine.

But 176 tests instead of 88 tests looks like some miscounting is
happening somewhere.

>                                              I didn't want to show the
> fails without the totals because then it'd look like Cairo is
> regressing; 69 fails in 1.2.2 compared with 10 in 1.0.0 sounds bad until
> you realize there were only 110 tests in 1.0.0, and now there are 735. ;-)

Yes, that's worth pointing out. But now the totals are down to numbers
like 89 as I expect rather than up at 735.

> If you have other ideas on how to make the display more immediately
> useful, let me know and I'll see what I can do.

Maybe it would help to add the total test count OK cells too. The idea
there is that if we transition 89 tests to 1 failure/90 tests that's
quite different from transitioning from 89 tests to 1 failure/89
tests. In the first case we've added a new test, so maybe the failure
isn't unexpected, (we're just testing for a bug that has perhaps been
present before). But in the second case it's likely that a new bug has
been introduced.

But without the number of tests being shown in the OK cells, it's hard
to distinguish the two cases.

> Well, here is what's in gentoo for x86 stable:
>
>  gnome-base/librsvg-2.12.7  USE="zlib -debug -doc -gnome -nsplugin"

That might be too old of a version of librsvg, (though we won't really
know until we try as I don't see any svg backend testing happening
yet). Emmanuel Pacaud is cairo's SVG backend maintainer and he
confirms that librsvg 2.14 is sufficient for testing cairo.

>  app-text/poppler-0.5.1-r1  USE="jpeg"

The version of poppler I'm using advertises itself as 0.5.3 so that's
hopefully sufficient, (though it may not be if what I'm running is
from cvs and poppler doesn't bump it's version number until just
before releases). Hopefully poppler is at least is doing a
post-release version bump.

As for what version of poppler is necessary, that's something I don't
know yet.

> I notice on some platforms, the ESP Ghostscript isn't available through
> gentoo.  They have some virtual package thingee set up such that when
> you 'emerge ghostscript', you might get the ESP version or the GNU
> version.

Strange. Let me know if you can't get a version of ghostscript
installed that makes the test suite happy.

> You have access now; feel free to log in and poke around.  See the
> tutorial at http://crucible.osdl.org for how to lock the machine if you
> want to prevent it from running tests while you're investigating it.

Thanks for the details. I'll see if I can chase down this failure.

> The logs/ directory contains the details.  For kernel stuff we break
> these up into separate files for configure, make, make check, etc. but
> for cairo its all in one file.  For convenience, I've now also
> hyperlinked this log file from the cairo_branch.html page.

I mentioned this on irc, but there appears to be a web server problem
in that the log files are being served with a mime type of a gzipped
tar file rather than plain text, so they're difficult to browse on the
web. Presumably some change to apache configuration would fix
that. Otherwise, perhaps renaming the files to not have a .log
extension rather than .tar.gz.log would workaround the problem.

I took a look at the ppc configure failure and the relevant output
appears to be:

	checking for cairo's PNG backend...
	configure: WARNING: Could not find libpng in the pkg-config search path
	checking whether cairo's PNG backend could be enabled... no
	configure: error: requested PNG backend could not be enabled
	CONFIGURE STATUS: 1

Is the libpng devel package simply not installed on that machine?

-Car


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freedesktop.org/archives/cairo/attachments/20060816/b6355b2d/attachment.pgp