[cairo] Documenting how to get test targets to pass that can

[Bryce Harrington]

On Thu, Aug 18, 2016 at 02:41:36PM -0400, darxus at chaosreigns.com wrote:
> I'd like to add some stuff to the test/README to more accurately represent
> the current state of tests, and provide instructions to get as many
> targets to pass as possible.
> 
> How about a section like this, maybe before "Tailoring tests running"
> (right after the initial command line examples)?
> 
> I'm happy to provide a patch, but I have a feeling some discussion should
> happen.
>
> Avoiding tests known to fail
> ----------------------------
> 
> As of August 2016, there are a number of tests known to fail even with
> updated references, by generating diffenet output each run, or crashing.
> The following will get the maximum number of targets to pass, by using
> freshly generated reference images (which should not be necessary).  This
> may be useful for generating new references, making changes to cairo, then
> running the tests against your references, before submitting your changes.
>
> Xvfb -screen 0 1680x1024x24 -ac -nolisten tcp :2 &
> 
> ./autogen.sh --enable-xcb
> make
> # To update reference images, will error:
> DISPLAY=:2 CAIRO_TEST_TARGET=xcb,image,image16 make test 
> # The new reference images:
> cp -a test/output/ $HOME/cairo-ref.xcb-image.xvfb
> 
> ./autogen.sh --enable-xcb
> make
> DISPLAY=:2 CAIRO_REF_DIR=$HOME/cairo-ref.xcb-image.xvfb CAIRO_TEST_TARGET=xcb,image,image16 make test

Good thinking, although it looks like this may already be covered.

There's a section under "What if I can't make my system match"
describing the basic essense of process.  Perhaps that could be improved
(e.g. moving it to a more notable spot in the file) but it looks like it
covers the main points of copying the output from a 'before' run, and
then using it via CAIRO_REF_DIR for your 'after' run.

The use of Xvfb is also documented in test/README, although in a
different section.  But this example really doesn't depend on the use of
Xvfb so probably clearer to omit it here.

(One thing that *isn't* documented, that I suppose we're touching on
tangentially here, is how to determine when the reference images
*should* be updated, officially.  I think if we had a clearer process
for that, it'd get done more and maybe reduce the need for the local
workaround in the first place.)

Bryce