[cairo] [Pixman] Planar YUV support

[Andrea Canciani]

On Thu, Mar 3, 2011 at 6:58 PM, Siarhei Siamashka
<siarhei.siamashka at gmail.com> wrote:
> On Wed, Mar 2, 2011 at 4:16 PM, Soeren Sandmann <sandmann at cs.au.dk> wrote:
>> Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:
>>
>>> I'm not a big fan of "let's make this totally universal and future
>>> proof" approach if only a very small fraction of this functionality is
>>> going to be actually used. Moreover, I suspect that trying to be too
>>> general was responsible for slowing down the original "Planar YUV
>>> support" plan.
>>
>> Part of what was derailing that plan may have been my insisting on being
>> precise about how these formats fit into the image processing pipeline,
>> including how it related to gamma correction and other colorspace
>> issues. I still think this is important. However, we can probably leave
>> out some specific features, if there is a credible story about how to do
>> them later.
>>
>> The pipeline as it is now:
>>
>>    1 convert image sample to a8r8g8b8
>>    2 extend sample grid in all directions according to repeat
>>    3 interpolate between sample according to filter
>>    4 transform
>>    5 resample
>>    6 combine
>>    7 store
>
> What is the difference between "3 interpolate between sample according
> to filter" and "5 resample"?
>
>> To add support for potentially subsampled YUV, some additional stages
>> have to be inserted before the first:
>>
>>   -2 interpolate subsampled components of YUV to get the same
>>      resolution as the Y plane
>>
>>   -1 if the format is planar, stitch together components to form YUV
>>      pixels
>>
>>    0 convert to sRGB
>>
>> Stage -2 is important because the filter used in that interpolation
>> should probably be user-specifiable eventually, which has the
>> implication that whatever simple support is added first, it needs to be
>> clear what filter precisely is being used.
>>
>> Stage 0 is a color space conversion and need to eventually be
>> configurable too, which means it has to be specified which matrix is
>> being used.
>
> I'm not totally sure about stage -2. So source interpolation is going
> to happen twice in the pipeline, once when getting rid of subsampling,
> and the second time when applying transform? To me it looks more
> natural to just start with the transform, for each pixel in the
> destination space get a transformed fractional coordinate in the
> source image, look it up in the source YUV grid and interpolate each
> color component for each color plane separately according to the
> selected filter, and finally convert interpolated YUV color to
> a8r8g8b8 according to the used color matrix. Though I see some trouble
> using such approach when handling repeat for odd sized images with
> subsampling (imagine handling of NORMAL repeat for fractional sized
> RGB images as a somewhat similar example). I don't know what would be
> the best solution here.

Are subsampled images allowed to have a size which is not a multiple
of the downsampling? I think that most video formats do not allow this
and pixman should not either.

>
> The following PDF file has some nice pictures showing chroma siting in
> a very comprehensive way (for the people not familiar with this
> concept, but who might be interested to know what we are talking
> about):
> http://www.poynton.com/PDFs/Chroma_subsampling_notation.pdf
>
>> There is also the question of how to *write* to a subsampled YUV
>> image. I don't particularly like read-only image formats, but writing to
>> YUV is not simple when you consider clip rectangles, because subsampling
>> involves a filter that probably extends outside the clip boxes.
>
> Writing to subsampled YUV images somewhat resembles writing to
> transformed RGB images to me (if this was supported by pixman of
> course). Both are inherently lossy to the point of becoming
> ridiculous. And if a sequence of compositing operations has to be
> applied to YUV destination image, then each step would introduce a
> major distortion by itself. I see only one recipe here - never allow
> to do this :-) If really high quality image processing has to be done,
> then any subsampled YUV destination image needs to be converted to
> some intermediate format (preferably losslessly). Then all the
> composition has to be done with this intermediate format and only
> converted to the target format at the very last step. A variation of
> this is to hide all these steps from the end user by doing everything
> behind the scene, but it may be too complex and inefficient.

There are two possible approaches to writing on subsampled images,
but both are quite complicated. Since you do not need to add write
support to subsampled images, this can be safely deferred and it
probably should be, as we're already tackling multiple problems
with new formats and YUV support.

>
> As I don't have a serious need for writing to YUV images at this
> point, I would like to skip this YUV writing part completely for now
> :-)
>
>> What Andrea is getting at, is presumably how to specify image formats in
>> the API. A fully general API like he suggests is perhaps interesting at
>> some point, but I agree it shouldn't be prerequisite for getting some
>> YUV support in.
>
> I think he tried to suggest that if any pixman API extension is needed
> at all, then it's better to do it flexible enough and more future
> proof. Please correct me if I'm wrong.

Yes, that's the idea.

>
> Right now I'm only interested in a new function
> 'pixman_image_create_planar' (with or without 'color_space' argument)
> from:
> http://cgit.freedesktop.org/~company/pixman/tree/pixman/pixman.h?h=planar#n764

pixman_image_t *pixman_image_create_planar
(pixman_format_code_t          format,
						      int                           width,
						      int                           height,
						      uint32_t                    **bits,
						      int                          *rowstrides_bytes);

is probably the right API to add in a short timeline (the number of planes can
be computed from the format, the color space and the memory layout are
both described by the format code).

I'm basically ok with this, but I hope to evolve it into:

pixman_image_t *pixman_image_create_planar           (pixman_format_t
        *format,

pixman_color_space_t         *color_space,
						      int                           width,
						      int                           height,
						      uint8_t                    **bits,
						      int                          *rowstrides_bytes);

in time for 0.22, as I think this would be more future-proof.

A first implementation would probably allow color_space and format
derived from a pixman_format_code, but this can then be extended.

I think that we might also want to have some more flexibility about the
alignment of planes and the row_strides, which it the reason why in
the second prototype I used uint8_t **bits.

I'm not sure if the YUV planar images we want to support are guaranteed
to have a row_stride %4==0 for each plane (implying that
width %(4*max_horizontal_downsampling)==0). If this is not the case,
we probably want to make strides (and maybe bits, too) more flexible.
(I expect that width%max_horizontal_downsampling==0, but this is a
much looser condition)

>
> The color matrix and chroma siting can be assumed to be set to some
> sane default, with the API to actually tweak them to be introduced in
> pixman later.

Agreed.

Andrea