[cairo] pixman "Simplify clipping rule" problems

[Siarhei Siamashka]

Hi,

Sorry for not being able to report it earlier, but I was on a vacation.
Since the commit 78ca4eea6467dbb6b9da1198b9526750a0a8dca3 (Simplify clipping
rule), scaling test reveals some problems:

git reset --hard 78ca4eea6467dbb6b9da1198b9526750a0a8dca3
./autogen.sh
./configure --disable-mmx --disable-sse2 --disable-shared
make
valgrind test/scaling-test

...
==17940== Invalid read of size 4
==17940==    at 0x8061669: fbCompositeSrc_x888x0565 (pixman-fast-path.c:787)
==17940==    by 0x80535C0: walk_region_internal (pixman-utils.c:596)
==17940==    by 0x8053B2F: _pixman_run_fast_path (pixman-utils.c:847)
==17940==    by 0x8061CEF: fast_path_composite (pixman-fast-path.c:1214)
==17940==    by 0x80504AA: pixman_image_composite (pixman-pict.c:132)
==17940==    by 0x8048C9A: test_composite (scaling-test.c:280)
==17940==    by 0x80490D6: main (scaling-test.c:325)
==17940==  Address 0x41e97e4 is 4 bytes before a block of size 480 alloc'd
==17940==    at 0x4024D2E: malloc (vg_replace_malloc.c:207)
==17940==    by 0x80489D0: test_composite (scaling-test.c:196)
==17940==    by 0x80490D6: main (scaling-test.c:325)
==17940==
==17940== Invalid read of size 1
==17940==    at 0x4025940: memcpy (mc_replace_strmem.c:402)
==17940==    by 0x8061EEA: fbCompositeSrc_8888xx888 (pixman-fast-path.c:1017)
==17940==    by 0x80535C0: walk_region_internal (pixman-utils.c:596)
==17940==    by 0x8053B2F: _pixman_run_fast_path (pixman-utils.c:847)
==17940==    by 0x8061CEF: fast_path_composite (pixman-fast-path.c:1214)
==17940==    by 0x80504AA: pixman_image_composite (pixman-pict.c:132)
==17940==    by 0x8048C9A: test_composite (scaling-test.c:280)
==17940==    by 0x80490D6: main (scaling-test.c:325)
==17940==  Address 0x43fc263 is 5 bytes before a block of size 96 alloc'd
==17940==    at 0x4024D2E: malloc (vg_replace_malloc.c:207)
==17940==    by 0x80489F1: test_composite (scaling-test.c:197)
==17940==    by 0x80490D6: main (scaling-test.c:325)
...

The final crc32 checksum is also different, but it is to be expected because
source clipping behavior changed.

The main source of problem are invalid memory accesses. The test deliberately
uses x, y, width, height parameters to sometimes refer to pixels outside of
source and destination images to check how pixman handles clipping. Before
"Simplify clipping rule" commit, it had no problems with this.

Is it a pixman bug or we have some new rules for pixman clients regarding the
valid range of input values for x, y, width, height?

Also I still wonder how many clients can be affected by this change and if the
old problems like http://bugs.freedesktop.org/show_bug.cgi?id=11620 could be
reincarnated. Regarding potential invalid memory accesses, it definitely would
be not nice if clients could now crash X server by malformed XRender requests.

-- 
Best regards,
Siarhei Siamashka