[cairo] What does it take to get a make check to pass with the xcb target, using CAIRO_REF_DIR?

[Bryce Harrington]

On Fri, Jul 08, 2016 at 08:58:40AM -0400, darxus at chaosreigns.com wrote:
> On 07/07, Bryce Harrington wrote:
> > When doing releases I also try to do a run and compare with the prior
> > release to spot regressions.  I'd really love to be able to do a
> > thorough git bisect search to locate which changes caused which tests to
> > fail, but as the suite takes a while to run it'd be pretty time
> > consuming.
> 
> I think bisecting sounds like great fun, except for the irregularity of
> problems.  I wrote a script to build a random commit, back through 2010
> (because they seemed to not be successfully building before that).  All of
> them either gave me no test-suite.log, or, for the 22 that gave me a log,
> the test suite had at least 6 failures.  
>
> Nothing passed all tests.  Does anybody have any ideas when all tests for
> the xcb target might have passed?  Is this evidence that the FAILs are
> caused by an external library?

When I was running regular test runs, I noticed several times that after
doing a system upgrade, the quantity of test failures changed
significantly, for the same Cairo git checkout version.  That experience
is what has me thinking that external causes may be playing a big role.
(And if it's external sources, that may make bisection searching less
likely to produce fruit.)
 
> This is the output, plus the perl script I wrote:
> http://www.chaosreigns.com/cairo/tests.tar.gz
> (The script is kind of crude.)
> 
> 
> This is a filename with a unix timestamp (git log -1 --pretty=format:%ct),
> an underscore, the commit hash, a colon, the number of FAILs, some spaces,
> then a GMT date calculated from the unix timestamp:
> 
> darxus at dancer:~/source/cairolog$ grep -l test * | xargs grep -c ' FAIL'
> 1264201313_c0458b456007f718747be7fd690e674df5026059:7    Fri Jan 22 23:01:53 2010
> 1269204078_247ef2bd63e755963f7930ac06c79b95aed2adb4:6    Sun Mar 21 20:41:18 2010
> 1271277994_54670ec13d64efa94f552b5473c1f15a9db1cecd:145  Wed Apr 14 20:46:34 2010
> 1273693338_094f0e0fa0153f290061635eed51e8d1dbe2cf4a:5    Wed May 12 19:42:18 2010
> 1276288178_d31e2418d6d9920205a154f21becca12a7b84c2a:7    Fri Jun 11 20:29:38 2010
> 1278883932_fbb0a260b707cb5f02a14cc368c6f2f0d63564c3:146  Sun Jul 11 21:32:12 2010
> 1286969384_90d50cd92315d6760069ad8062aba5e297370b20:6    Wed Oct 13 11:29:44 2010
> 1292229969_d9ba8337ab456ae0e232d3c603cb41cea984ebea:32   Mon Dec 13 08:46:09 2010
> 1292427140_e4b78424ac82588bcb9b855d5b6d5872050d33f9:7    Wed Dec 15 15:32:20 2010
> 1297614844_c3605bd3adcae1f12731230a1ea599d15e3c8cad:7    Sun Feb 13 16:34:04 2011
> 1311937629_58df191946b7d275c5706dc40e1cf3dffa7cceae:8    Fri Jul 29 11:07:09 2011
> 1313177159_bb24f165ff57973347b34956a371c6b33d2d9b59:55   Fri Aug 12 19:25:59 2011
> 1342113299_3ed581149f82b653e1987e746467ad8ae9ce3ca2:7    Thu Jul 12 17:14:59 2012
> 1348565464_31291611625c21bc9ec2827cf4ccc8a1f18392ed:55   Tue Sep 25 09:31:04 2012
> 1405128883_d5ffe67008afe46c030b31afc0803321b6479990:7    Sat Jul 12 01:34:43 2014
> 1413893554_f032133e6d5ad05157fc46609d8c63103028342c:7    Tue Oct 21 12:12:34 2014
> 1423145626_3cf862f6d973755cd9824c2224fbe0a623c47ff1:7    Thu Feb  5 14:13:46 2015
> 1425713393_58df191946b7d275c5706dc40e1cf3dffa7cceae:7    Sat Mar  7 07:29:53 2015
> 1434656331_ae608035c7b7133826a608d45e067c3875a1aceb:7    Thu Jun 18 19:38:51 2015
> 1435360353_70cc8f250b5669e757b4f044571ba0f71e3dea9e:7    Fri Jun 26 23:12:33 2015
> 1465125216_caa4c9fdeb3aecd9a4288114e75d24ec931cd01b:6    Sun Jun  5 11:13:36 2016
> 1467845160_f9b65ae1fc91bc558a01c2ad7be5a121c6f10818:7    Wed Jul  6 22:46:00 2016

I'm not sure I'm following how the timestamp is being calculated here.
Spot checking a few of the commits, the AuthorDate and CommitDate of the
commit are significantly off from the GMT date you're listing (by a year
or more).

Looking at the last commit, I get:

$ git log -1 --pretty=format:%ct f9b65ae1fc91bc558a01c2ad7be5a121c6f10818
1413893554
$ date --date='@1413893554'
Tue Oct 21 05:12:34 PDT 2014

$ git show f9b65ae1fc91bc558a01c2ad7be5a121c6f10818 | head -n5
commit f9b65ae1fc91bc558a01c2ad7be5a121c6f10818
Author:     Adrian Johnson <ajohnson at redneon.com>
AuthorDate: Tue Oct 21 22:35:12 2014 +1030
Commit:     Adrian Johnson <ajohnson at redneon.com>
CommitDate: Tue Oct 21 22:42:34 2014 +1030

I'm confused where your Jul 6, 2016 date comes from.  Am I looking at
the right thing?

> Of the ones that had fewer than 10 FAILs, the test names mostly started
> with "record", with a few "subsurface".  
> 
> The most recent one with no test-suite.log was      
> 1466331752_bf5adaf3942388e58ad3bda30173e53b214df885 Sun Jun 19 10:22:32 2016 
> Which I manually checked, and it did fail to build:
> test/cairo-test-runner.c:130: undefined reference to `cairo_boilerplate_xmalloc'

Hmm.  Commit bf5adaf39 seems to be from 2012?

Looking at the commits around and prior to June 19th, 2016, I'm not
spotting anything that would have changed the build system, test runner,
or build system that would cause a failed link like that.

Side note...  Looking at the git logs I noticed Adrian Johnson recently
updated the reference images about a month ago.

> The second most recent one with no test-suite.log was:
> 1465125216_afe6f4f0519606c4bc7e9b705b0cae75692d7af2 Sun Jun  5 11:13:36 2016 
> It also failed to build:
> ./autogen.sh: running `./configure --cache-file=config.cache --disable-static --enable-test-surfaces --prefix=/home/darxus/install --enable-xcb'
> configure: creating cache config.cache
> configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."
> 
> (Which, so far, confirms my expectation that empty output files are due to
> failures to build.)



> This is how many I attempted to build per year that produced no
> test-suite.log.  The ones from before 2010 were from before I started to
> restrict the date range (and go to sleep):
> 
> $ grep -L test * | cut -d'_' -f1 | ~/bin/unixtime.pl | awk '{print $5}' | uniq -c
>       1 2002
>       3 2003
>       3 2004
>      16 2005
>      14 2006
>      15 2007
>      23 2008
>       6 2009
>     100 2010
>      80 2011
>      40 2012
>      22 2013
>      18 2014
>       7 2015
>       2 2016
>
> This is how many total I ran:
> 
> darxus at dancer:~/source/cairolog$ ls *_* | cut -d'_' -f1 | ~/bin/unixtime.pl | awk '{print $5}' | uniq -c
>       1 2002
>       3 2003
>       3 2004
>      17 2005
>      14 2006
>      15 2007
>      23 2008
>       6 2009
>     109 2010
>      84 2011
>      42 2012
>      22 2013
>      20 2014
>      11 2015
>       4 2016

Well, I would certainly not be surprised if there was a cutoff date, and
earlier cairos just don't build due to API/ABI changes in dependencies,
or other build-system level changes that just make it unbuildable.
However, it looks like pretty much all your runs are failing to produce
test logs, including recent years, which shouldn't be quite *that*
bad...  Could there be an error in your timestamp calculation?

Bryce