[cairo] Time for a release ?

[Bryce Harrington]

On Thu, Nov 19, 2020 at 10:23:33PM +0000, Stuart Axon wrote:
> [1] Locking issues
> 
> I ran the tests in a Windows VM with Meson and quite a few timed out, it
> seems like quite a few will be the ones with locking issues.
> 
> https://gist.github.com/stuaxo/1d125dc83888a28f60c684718619a112
> 
> There were a couple of tests that popped up dialog boxes under Windows when
> they failed, which seems not idea for CI.
> 
> I'll see if I can do the same under Linux and possibly MacOS to see if the
> same ones time out.
>
> [2]  Rendering issues:
> 
> It sounds like there are different sources for these, so maybe it's worth
> having tickets for the different kinds of failure, once a baseline is
> established.
> 
> Maybe we could make sanity checks for the different failure types (or
> designate existing tests for these) ?
>
> Pixman:  As mentioned elsewhere, changes that affect rendering here will
> mean ref images need updating.

For rendering issues in the image backend, I tended to find that a
common root cause would often be a changed assumption about how drawing
operations should behave.  For example, algorithm changes in pixman can
result in a pixel to flip on or off, or to be set to a slightly
different shade of grey.  Legitimate performance or quality improvements
to blending or dithering algorithms can cause these kinds of changes
too.  Typically for those cases the rendering behavior is kind of a
judgment call, so not much to do except redo the reference images to
match the new behavior.

> Ghostscript:  We could render some known postscript through this and check
> the output.

With backends like ps, a common situation is that a given operation, or
some portion of it, may not be supported there.  For example, Postscript
doesn't have gradients, so test cases involving shading or shadows or
hinting may sometimes run up against this.  This can especially show up
with pdf where newer versions of the standard support drawing features
that older versions did not.

> Fonts and operating systems:
> 
> Do we expect output on differing operating systems to be the same?  If not,
> maybe per-OS reference images is could work (e.g. Linux/MacOS/Windows would
> be expected to be similar, but whichever OS you are running on should be an
> exact match).

A foundational principle for Cairo is that rendering output should be
reliably identical not only cross-platform but also cross-device
(i.e. what's on your monitor matches what's printed on paper, and
rendering to PNG, PDF, or SVG should all be identical.)

In practice, no of course actual renderings will differ from one
platform to the next.

Honestly, though, it's even more complicated than that due to dependence
on fonts, dependency versions, etc.  So I wouldn't expect images on
Ubuntu to match ones from Fedora, nor even ones rendered on Ubuntu 16.04
to match those on Ubuntu 20.04.  Achieving decent apples-to-apples
comparison gets to be a bit of an combinatorial explosion...

I'm skeptical at the practicality of central collection of all these
different reference images.  If we could start over from scratch, I think
a better engineered solutions would use the reference images *only* as
visual references, perhaps with a registry of git SHAs where the
rendering has been manually verified to be correct for a given set of
platform parameters.  You'd use the testsuite by first generating a
collection of "truth" renderings using a known-good git tag (e.g. the
last stable release) and then running it against your branch (or the
next commit in the CI cycle, or whatever) and doing the image-diff
operation between those.

Bryce