[cairo] [Pixman] Planar YUV support

[Soeren Sandmann]

Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:

> I'm not a big fan of "let's make this totally universal and future
> proof" approach if only a very small fraction of this functionality is
> going to be actually used. Moreover, I suspect that trying to be too
> general was responsible for slowing down the original "Planar YUV
> support" plan.

Part of what was derailing that plan may have been my insisting on being
precise about how these formats fit into the image processing pipeline,
including how it related to gamma correction and other colorspace
issues. I still think this is important. However, we can probably leave
out some specific features, if there is a credible story about how to do
them later.

The pipeline as it is now:

    1 convert image sample to a8r8g8b8
    2 extend sample grid in all directions according to repeat
    3 interpolate between sample according to filter
    4 transform 
    5 resample
    6 combine
    7 store

To add support for potentially subsampled YUV, some additional stages
have to be inserted before the first:

   -2 interpolate subsampled components of YUV to get the same
      resolution as the Y plane

   -1 if the format is planar, stitch together components to form YUV
      pixels

    0 convert to sRGB

Stage -2 is important because the filter used in that interpolation
should probably be user-specifiable eventually, which has the
implication that whatever simple support is added first, it needs to be
clear what filter precisely is being used.

Stage 0 is a color space conversion and need to eventually be
configurable too, which means it has to be specified which matrix is
being used.

There is also the question of how to *write* to a subsampled YUV
image. I don't particularly like read-only image formats, but writing to
YUV is not simple when you consider clip rectangles, because subsampling
involves a filter that probably extends outside the clip boxes.

What Andrea is getting at, is presumably how to specify image formats in
the API. A fully general API like he suggests is perhaps interesting at
some point, but I agree it shouldn't be prerequisite for getting some
YUV support in.

> Surely an alternative solution for html5 video fallbacks is to do
> YUV->RGB conversion of video frames outside of cairo/pixman, and only
> let cairo/pixman handle the RGB data. The only real problem with this
> solution is that such multi pass data processing is somewhat slower.
> We are speaking about at least tens of percents of performance here.
> And pixman support for native single pass YUV->RGB conversion combined
> with scaling could help.

YUV conversion is definitely within the scope of pixman, both YUV->RGB
and RGB->YUV. However, I'm not sure the existing YUV formats are very
well though out.


Soren