[cairo] External cairo-test-suite

[Carl Worth]

On Mon, 2008-09-01 at 18:57 +0100, Chris Wilson wrote:
> I've cooked up a patch to compile all the 190 (and rising!) tests into a
> single binary.

This sounds fantastic, Chris!

> Another improvement is that each test is grouped by keywords so that you
> can just run all the text tests, or all the clip tests etc. The XFAIL
> status is moved to the test itself (and becomes just another keyword) so
> that the information is maintained along with the test. Hence
> "./cairo-test-suite XFAIL" will execute all the known failures. The
> tests are also able to specify required library versions (and as a
> special case whether it needs a raster target, again moving the
> information from an array inside cairo-test.c into the test itself).

All sounds very nice, thanks!

> So I'm soliciting feedback - especially from Behdad since I've hacked
> about the test/Makefile.am quite a bit and from Vlad wrt to the
> buildbots. However, everyone feel free to pitch in with ideas and
> improvements.

I haven't looke specifically at what you've got here, but one thing that
came from my recent test-suite polishing for 1.8 is that we really want
XFAIL to apply on a backend-specific basis for a test. If I know that
our ps backend is emitting something bogus for the degenerate-path, I
want to mark that XFAIL, but I don't want to make that then hide any
regressions that appear in that test for other backends.

For that matter, I don't want it to hide regressions in the XFAIL
backend either. So we'll probably want to have something like:

	degenerate-path-ps-xfail.png

and have the test suite first compare against -ref.png and if that fails
then if there's also a -xfail.png then test against that, (match==XFAIL
and mismatch=FAIL).

Make sense?

-Carl