[cairo] [PATCH] perf: Move macro-benchmark documentation to cairo-traces

[Bryce W. Harrington]

On Tue, Jul 09, 2013 at 10:22:15AM +0100, Chris Wilson wrote:
> On Tue, Jul 09, 2013 at 02:08:03AM +0000, Bryce W. Harrington wrote:
> > The macro benchmarks were moved to a separate repository some time ago,
> > but the perf README still refers to these tests as if they were still
> > present, which may lead to some confusion.  Instead, consolidate the
> > macro benchmark documentation with the macro benchmarks, and focus this
> > README on just the (still in tree) micro-benchmarks.
> 
> Throw in a comment for the purpose of macro level tests and for when
> comparing performance using cairo-perf-trace is preferred over the
> micro benchmarks (and make perf). And a TODO for a control language for
> crafting and running small sets of micro benchmarks.

Sounds good, will follow up with these changes directly.

> Also if make perf still exists and being used, we should use it run
> cairo-perf-trace over the cairo-traces/benchmark if present.  -Chris

Yes, make perf does still exist, and runs both the micro-benchmarks and
cairo-perf-trace.  And when cairo-traces is copied or symlinked to
cairo/perf/cairo-traces, cairo-perf-trace definitely does find
uncompressed cairo-script *.trace files under cairo-traces/benchmark and
runs them fine.  I've not run the entire kit to be certain, but it does
look like make perf will run all the tests, if everything's properly set
up.

The one hitch I ran into is that cairo-traces' Makefile by default
creates .trace files which aren't plain text cairo-script, but rather
some binary format generated by csi-bind (which I don't yet grok...)
The error messages weren't terribly helpful - something about invalid
value for input sizes (see http://paste.ubuntu.com/5856936/).  But
I found that these worked fine if I run them with the cairo-perf-trace
script from my system (i.e. after installing the cairo-perf-tools .deb),
but *not* with the cairo-perf-trace from current git.

However, I found if I manually lzma -cd the compressed traces to plain
text cairo-script .trace files, then everything started working (the files
must be named *.trace though; *.cs aren't found).  Looks
like it'd take forever to run through all the macro benchmarks
completely, and it bogs my system down noticeably while it does run.
(The full micro benchmarks take a long time too...  I wonder how long
running both micro and macro sequentially would take?  4-5 days?)

I'm not sure what the csi-bind step does, or if omitting it is going to
make the test results less valid.  I took a shot at redoing
cairo-traces' Makefile to link against my local cairo checkout rather
than the system cairo, but no dice - still got the exact same failures.


Anyway, this is all a bit clunky and could benefit from some
improvements...

1.  I'm concerned that we have both binary and plain text trace files
    sharing the .trace suffix; if these truly are different types of
    files maybe we need two suffixes to distinguish between them?
    (Possibly I'm just misunderstanding things here...)

2.  cairo-traces' Makefile builds csi-bind and csi-trace, but I wonder
    if perhaps these two programs should move over to cairo/util?  Then
    cairo-traces could focus purely on data file packaging, with no need
    for software build logic.

3.  Currently (if I understand correctly) you have to manually
    uncompress the .lzma files before you can run cairo-perf-trace.  But
    couldn't c-p-t take care of that itself internally?  If it was able
    to accept .lzma files as arguments, that'd simplify some of the
    usage directions at least.

Bryce
--
Bryce Harrington
Senior Open Source Developer  -  b.harrington at samsung.com
Open Source Group             -  Samsung Research America