[cairo] A better image comparison tool

[Carl Worth]

On Wed, 2008-09-24 at 09:51 +0200, Adib Taraben wrote:
> There is also a tool for perceptual image difference 
> http://pdiff.sourceforge.net/

Actually, we found that and integrated it into cairo's test suite a long
time ago. We've been underwhelmed in a few ways:

1. It's *really*, *really* slow. It slows down the test suite much more
than I'd like.

2. Lots of false negatives. Adding pdiff didn't let us remove nearly as
many of the backend-specific reference images as I would have liked.

3. A few false positives. This is the real killer. One of the things we
do in the cairo test suite is to have fairly small reference images, (we
like to keep the test suite running fast), and they tend to have fairly
high-frequency content. There was a case that was something like a 10x10
image with a 2-pixel-wide white cross on a black background. And when we
had a bug that made the horizontal portion of the cross disappear, pdiff
still thought the result was "visually similar" and the bug slipped
through unnoticed. That's really unacceptable and something we need to
avoid.

One of the things I'm really hoping for with Antoine's work is a tighter
coupling of the comparison tool with the needs of our test suite. If we
understand the comparison tool enough to know what minimum feature sizes
we need to avoid false positives, then we can do that. Or if the author
of the comparison tool can look at our current reference images and see
what features we're testing for and tune for that, then that will work
too.

Those are things that we're just plain missing by just grabbing the
pdiff code and trying to use it.

-Carl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.cairographics.org/archives/cairo/attachments/20080924/03481c83/attachment.pgp