[cairo] [Pixman] Planar YUV support

[Siarhei Siamashka]

On Wed, Mar 2, 2011 at 4:16 PM, Soeren Sandmann <sandmann at cs.au.dk> wrote:
> Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:
>
>> I'm not a big fan of "let's make this totally universal and future
>> proof" approach if only a very small fraction of this functionality is
>> going to be actually used. Moreover, I suspect that trying to be too
>> general was responsible for slowing down the original "Planar YUV
>> support" plan.
>
> Part of what was derailing that plan may have been my insisting on being
> precise about how these formats fit into the image processing pipeline,
> including how it related to gamma correction and other colorspace
> issues. I still think this is important. However, we can probably leave
> out some specific features, if there is a credible story about how to do
> them later.
>
> The pipeline as it is now:
>
>    1 convert image sample to a8r8g8b8
>    2 extend sample grid in all directions according to repeat
>    3 interpolate between sample according to filter
>    4 transform
>    5 resample
>    6 combine
>    7 store

What is the difference between "3 interpolate between sample according
to filter" and "5 resample"?

> To add support for potentially subsampled YUV, some additional stages
> have to be inserted before the first:
>
>   -2 interpolate subsampled components of YUV to get the same
>      resolution as the Y plane
>
>   -1 if the format is planar, stitch together components to form YUV
>      pixels
>
>    0 convert to sRGB
>
> Stage -2 is important because the filter used in that interpolation
> should probably be user-specifiable eventually, which has the
> implication that whatever simple support is added first, it needs to be
> clear what filter precisely is being used.
>
> Stage 0 is a color space conversion and need to eventually be
> configurable too, which means it has to be specified which matrix is
> being used.

I'm not totally sure about stage -2. So source interpolation is going
to happen twice in the pipeline, once when getting rid of subsampling,
and the second time when applying transform? To me it looks more
natural to just start with the transform, for each pixel in the
destination space get a transformed fractional coordinate in the
source image, look it up in the source YUV grid and interpolate each
color component for each color plane separately according to the
selected filter, and finally convert interpolated YUV color to
a8r8g8b8 according to the used color matrix. Though I see some trouble
using such approach when handling repeat for odd sized images with
subsampling (imagine handling of NORMAL repeat for fractional sized
RGB images as a somewhat similar example). I don't know what would be
the best solution here.

The following PDF file has some nice pictures showing chroma siting in
a very comprehensive way (for the people not familiar with this
concept, but who might be interested to know what we are talking
about):
http://www.poynton.com/PDFs/Chroma_subsampling_notation.pdf

> There is also the question of how to *write* to a subsampled YUV
> image. I don't particularly like read-only image formats, but writing to
> YUV is not simple when you consider clip rectangles, because subsampling
> involves a filter that probably extends outside the clip boxes.

Writing to subsampled YUV images somewhat resembles writing to
transformed RGB images to me (if this was supported by pixman of
course). Both are inherently lossy to the point of becoming
ridiculous. And if a sequence of compositing operations has to be
applied to YUV destination image, then each step would introduce a
major distortion by itself. I see only one recipe here - never allow
to do this :-) If really high quality image processing has to be done,
then any subsampled YUV destination image needs to be converted to
some intermediate format (preferably losslessly). Then all the
composition has to be done with this intermediate format and only
converted to the target format at the very last step. A variation of
this is to hide all these steps from the end user by doing everything
behind the scene, but it may be too complex and inefficient.

As I don't have a serious need for writing to YUV images at this
point, I would like to skip this YUV writing part completely for now
:-)

> What Andrea is getting at, is presumably how to specify image formats in
> the API. A fully general API like he suggests is perhaps interesting at
> some point, but I agree it shouldn't be prerequisite for getting some
> YUV support in.

I think he tried to suggest that if any pixman API extension is needed
at all, then it's better to do it flexible enough and more future
proof. Please correct me if I'm wrong.

Right now I'm only interested in a new function
'pixman_image_create_planar' (with or without 'color_space' argument)
from:
http://cgit.freedesktop.org/~company/pixman/tree/pixman/pixman.h?h=planar#n764

The color matrix and chroma siting can be assumed to be set to some
sane default, with the API to actually tweak them to be introduced in
pixman later.

-- 
Best regards,
Siarhei Siamashka