[cairo] Malloc failure testing

[Chris Wilson]

Recently I've been experimenting with injecting allocation failures into
cairo. The process I've used is to write a valgrind skin that can
arbitrarily cause an allocation to fail. By instructing the injector to
cause every allocation to fail only once and by causing the test harness
to loop until it completes a test without encountering a
CAIRO_STATUS_NO_SUCCESS, I can be confident that every single call site
will have received one failure. Of course this is not the same as every
single branch for every failure pattern...

Due to various complications, the primary one being certain tests
generate CARIO_STATUS_NO_MEMORY all by themselves, I adapted cairo-perf
as my test framework. These adaptations are available in my
malloc-testing branch,
http://gitweb.freedesktop.org/?p=users/ickle/cairo;a=shortlog;h=malloc-testing

The valgrind skin is available at
http://gitweb.freedesktop.org/?p=users/ickle/valgrind;a=shortlog;h=memfault
[Note this skin also doubles as a malloc profiler]

To use:
../libtool --mode=execute valgrind --tool=memfault \
--fail-every=1 --fault-regex=cairo* \
--nofault-fn=xmalloc --nofault-fn=xasprintf --nofault-fn=_XReply \
--nofault-regex=Fc* --nofault-regex=ft_* --nofault-regex=FT_* \
./cairo-perf

which says trigger allocation failures if the call emanated from a cairo
function, but avoid xmalloc, xasprintf, and _XReply as these call exit
on failure.
FontConfig and FreeType functions must be protected from failures as
they are subject to their own bugs. :-(

After program termination, valgrind will output the stack trace if
cairo-perf aborted, the location of the last few injected faults, all
blocks not freed and a summary table of all allocators.
'valgrind --tool=memfault --help' will provide a list of understood
comandline options.

At the moment, all I feel that is missing from the skin is the ability to
apply leak suppressions - a trick I've yet to learn from memcheck, and I
fancy a source code annotator to check coverage and show mallocs per
line - which suggests outputing the stacktraces in a common format and
offloading the job to another program. Other suggestions are welcome.

However, the cairo side leaves slightly more to be desired.
1) It should be targetting cairo-test instead/as well as cairo-perf.
2) How do I determine if the malloc failure caused side effects? So far
there have been a few cases where an earlier malloc failure triggers a
XRenderBadGlyph error, allocated but not used cache entries etc.
But how can we detect more subtle errors?
3) How do we systematically test all possible failure sources? Such as
we know this function can generate CAIRO_STATUS_SUCCESS,
CARIO_STATUS_NO_MEMORY, CAIRO_STATUS_INVALID_MATRIX so can we
automatically trigger those errors for all its callers. Those returns
should be documented as part of the function interface...
And can this procedure be extended to all of cairo's base libraries?
4) The things I don't know.

Anyway, I hope this serves as food for thought and encourages a few more
people to routinely check cairo and think of new ways of breaking it.
:-)
--
Chris Wilson