[cairo] cairo 1.4.12 on Solaris 10, Sparc

[Carl Worth]

On Thu, 20 Dec 2007 15:12:08 +0900, "Nguyen Vu Hung" wrote:
> I think that "make test" is some kind of regression test suite we've made.
> And in this case, because the development system is ( slight
> )different ( Solaris ), we can use "make test" to find bugs.

Yes, "make test" is a regression test suite that's designed to help us
find bugs.

The problem is that "make test" is more strict than we really
want. What would be ideal is if the automatic test suite could verify
that the results are "perceptually identical" to the reference
images---that is, that to a human the reference image and the output
image look identical.

But we have not yet succeeded in writing a computer program that can
effectively do that. So for now, we keep "make test" on the strict
side of things, (preferring false negatives that require a human to
verify that there's no real problem in the reported failure, as
opposed to false positives that would lead to unreported bugs). And we
try to eliminate as many variables as can introduce differences in the
test suite output.

> > You can look at the index.html file that "make test" created to see if
> > the differences look imperceptible.
> >
> # I can't post it now. But
> For test cases those passed, output are OK. Otherwise, I can't see the
> png files.

I don't know what happened with index.html. But just pick a test that
failed, and compare the -out.png to the corresponding -ref.png
images. If they look the same to a human, then the test is reporting a
false negative because it is too strict, (as I described above). You
can also look at the -diff.png image to see which pixels
differ. Often, in tests with text, the pixels around the border of
each glyph differ due to different rasterization of the underlying
font system, (such as freetype).

And if so, then the failures are nothing to worry about. The
underlying font system is out of cairo's control and minor differences
in rasterization from one system to another are not likely to be a
problem.

> It is not clear for me. Maybe I've misunderstand the purpose of make test.
> If the test cases are correct, then make test *should* report valid result.
> But as you said, make test has reported misleading failures.
> So should we consider to check the test suites?

We know that the test suite is broken in the case of tests that deal
with text. What I was proposing in the paragraph you're responding to
here was an idea for improving the test suite to no longer have this
problem. It was really an idea for Behdad as he might tackle this
stuff in the upcoming months,

Thanks for the report. From everything I can see your installation of
cairo is working just fine.

Have fun,

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.cairographics.org/archives/cairo/attachments/20071220/97d8f322/attachment-0001.pgp