[cairo] Automated testing of Cairo

[Carl Worth]

[Dropping desktop_architects from the cross-posting here.]

On Thu, 17 Aug 2006 01:37:30 -0700, Bryce Harrington wrote:
> I wrote a new git patch puller for cairo, and grabbed daily patches
> since 1.2.2 and ran them through:
>
>   http://crucible.osdl.org/runs/cairo_report.html

Hi Bryce,

I'm glad you're getting more updates once again. This is going to be a
very nice system when it's all running smoothly.

> Not sure why the last two patches are failing to apply, but ignoring

It's really ironic that you're running into a "failed to patch"
situation when you've got a git clone of cairo's repository just
sitting there on your machine. That clone will give you the exact
state of the cairo source at any point in its development history with
cryptographically strong assurance that you're getting exactly the
right thing.

You're only making things much harder on yourself than they should
be by extracting a tar file from git, extracting a patch from git, and
then trying to apply that patch.

Instead you should just be doing something like this in a clone of
cairo's repository:

	git checkout -b crucible-branch <sha1>

I understand that you're adding git support to an existing,
patch-based system. But instead of trying to get that to work without
changing the "system" I would strongly advise first changing the
system to embrace what git is really good at, (such as tracking the
entire state of a repository). Once you've done that, it would even be
much simpler to just use git itself for tracking projects based on tar
files, or patches, or anything else.

> those, the only noteworthy change in behavior since 1.2.2 is on the amd
> system in the font-matrix-translation test.  That started between 8/10
> and 8/11 and only seems to make an appearance on the AMD system.

Once we get your tests all performing correctly---

Here's my diagnosis of the current failures:

1) SVG failing many tests

   This is due to a bug in the test suite where the svg2png program is
   linked against an old system version of cairo rather than the
   version of cairo being tested. It's too bad you ran into the
   "failed to patch" problem when you did, since the latest 6122cc85
   commit from Behdad is to fix this exact issue.

2) PS failing a handful of tests

   This is the ghostscript version discrepancy we discussed earlier.

3) Some systems fail every test using text

   I'm not sure if this is a missing Vera font, or if there's a
   freetype version issue or something else.

4) ft-text-vertical-layout failing on all backends

   I'm not sure what's happening here, but this is definitely a test
   that has been more fragile than others. Behdad has recently
   re-worked this test, (it now adds a new dependency on another font,
   Nimbus something or other?---Behdad, we should get in the habit of
   documenting test suite dependencies in test/README or better, check
   for them and give warnings in the logs about what's missing).

---so once we get all of those issues resolved, then we have the
opportunity to do some very interesting things.

For example, when crucible is pulling new versions on daily basis and
things switch from OK to BAD then it's in the perfect spot to do
something much better than just reporting "something bad happened on
this day". Instead, crucible can use the two commit IDs and start a
"git bisect" session to get more reports on intermediate versions. And
the end result of that will be that crucible will end up pointing to
exactly the commit that introduced the bug. This kind of thing will be
extremely valuable and is exactly the kind of thing that is so easy
once crucible really starts embracing git.

With that kind of thing in place, you could adjust the frequency,
(say, crucible pulling stuff out only once a week or whatever), and it
would still drill its way down the the one bad commit when
needed. Obviously, using less frequent pulls would introduce lag into
the system, but it might be nice to be able to do this if you end up
having machines that can't keep up with the daily load, (I'm thinking
about arm systems for example).

Anyway, I continue to be very encouraged by the progress here. And I
hope I can continue to be of help in getting this system working as
well as possible.

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freedesktop.org/archives/cairo/attachments/20060817/877a7fdb/attachment.pgp