[cairo] Planar YUV support
Benjamin Otte
otte at redhat.com
Tue Mar 9 08:09:57 PST 2010
Hey,
Here's an update on my work on YUV planar support, also known as "what
we got from the gstreamer-cairo hackfest". I've reworked my experimental
code from back then to incorporate the things we agreed on. It is not
complete yet, but only misses details. So if you're a maintainer of the
libraries in question, now is a good time for a review and complaints or
issues with the general design of the code.
The code can be found in these 3 places:
http://cgit.freedesktop.org/~company/cairo/log?h=planar
http://cgit.freedesktop.org/~company/pixman/log?h=planar
http://cgit.freedesktop.org/~company/gst-plugins-cairo
I constantly rebase and update it while I work on it, so that the final
merge contains a proper set of patches.
I'd like to merge this stuff to the respective master branches soon
after Cairo 1.10 is out, so that it gets enough testing and exposure
before it's contained in the next major Cairo release, which I hope will
happen before the end of September.
I'll now give a short description of what these patches do, what the
issues are and what's missing.
pixman:
The code adds a pixman_color_space_t enum that can be used to specify
the color space the values are in. Valid values so far are ARGB,
ARGB_UNMULTIPLIED, YCBCR_SD, YCBCR_HD and YCBCR_JPEG. Various new
pixman_format_code_t's were added to match the planar formats used in
YUV. Finally a new constructor for images was added:
pixman_image_t *
pixman_image_create_planar (pixman_format_code_t format,
pixman_color_space_t color_space,
int width,
int height,
unsigned int num_planes,
uint32_t **bits,
int *rowstrides);
This constructor combines the above features to add support for all the
image representations commonly used. So if you want to create a
pixman_image for a GdkPixbuf with alpha channel, you'd do:
uint32_t *bits = (uint32_t *) gdk_pixbuf_get_data (pixbuf);
int stride = gdk_pixbuf_get_rowstride (pixbuf);
pixman_image_create_planar (PIXMAN_r8g8b8a8,
PIXMAN_COLOR_SPACE_ARGB_UNMULTIPLIED,
gdk_pixbuf_get_width (pixbuf),
gdk_pixbuf_get_height (pixbuf),
1,
&bits,
&stride);
Or to create an image from a GStreamer I420 buffer, you would do:
for (i = 0; i < 3; i++) {
bits[i] = (uint32_t *) (buffer_data +
gst_video_get_component_offset (GST_VIDEO_FORMAT_I420,
i, width, height));
strides[i] = gst_video_format_get_row_stride (GST_VIDEO_FORMAT_I420,
i, width);
}
pixman_image_create_planar (PIXMAN_y420,
PIXMAN_COLOR_SPACE_YCBCR_SD,
width, height,
3, bits, strides);
That is roughly anything you need to know as a user of pixman.
Details missing in the implementation that I intend to fix:
- The conversion matrices for HD and JPEG color spaces are still
missing. The ones I randomly copied so far were all wrong. And I was too
lazy to do the math myself yet. (I blame COG for getting them wrong.)
- Writing for i420 is not implemented. I haven't found a way to
implement it yet that passed my not-ugly test.
- There is no fast paths at all for the YUV-related formats yet. I
certainly want to add the ones that are needed, but that requires some
real life tests first.
- Getting the fetchers right for subsampled formats and different
filters.
Things that I don't intend to fix before merging, but would be happy to
see others have a go at:
- formats I consider "unimportant" such as 4:1:1 video or the other
4:2:0 chroma varieties other than the MPEG1 one that GStreamer uses and
that I've implemented.
cairo:
The only change visible to users of Cairo will be the addition of the
cairo_color_space_t enum, additions to the cairo_format_t enum that
expose the required pixman formats and a new constructor:
cairo_public cairo_surface_t *
cairo_image_surface_create_planar (cairo_format_t format,
cairo_color_space_t color_space,
int width,
int height,
unsigned int n_planes,
char ** data,
int * strides);
This constructor matches the pixman constructor above. Even the examples
would look very similar. For the GdkPixbuf you'd do:
uint32_t *bits = (uint32_t *) gdk_pixbuf_get_data (pixbuf);
int stride = gdk_pixbuf_get_rowstride (pixbuf);
cairo_image_surface_create_planar (CAIRO_FORMAT_RGBA32,
CAIRO_COLOR_SPACE_ARGB_UNMULTIPLIED,
gdk_pixbuf_get_width (pixbuf),
gdk_pixbuf_get_height (pixbuf),
1,
&bits,
&stride);
In fact, I'm using a gdk patch right now that does exactly this. (I also
have a WebKit patch that does this to improve <canvas> and svg filters.)
To create an image from a GStreamer I420 buffer, you would do:
for (i = 0; i < 3; i++) {
bits[i] = (uint32_t *) (buffer_data +
gst_video_get_component_offset (GST_VIDEO_FORMAT_I420,
i, width, height));
strides[i] = gst_video_format_get_row_stride (GST_VIDEO_FORMAT_I420,
i, width);
}
cairo_image_surface_create_planar (CAIRO_FORMAT_PLANAR_420,
CAIRO_COLOR_SPACE_YCBCR_SD,
width, height,
3, bits, strides);
And that is exactly what gst-plugins-cairo does.
I actually had to write quite a bit of code in Cairo to make this work,
because a lot of the backends assume they get image surfaces in one of
the 3 common format: A8, RGB24 or ARGB32. And this of course is no
longer true.
To be sure to catch all the cases, I removed the image_surface's data
and stride members. With planar images, they don't make any sense
anyway. That of course causes a lot of compilation failures and I
haven't yet fixed all of them. In particular the backends that don't
work on my computer (Most notably win32 and CoreGraphics) or
experimentaql haven't been fixed.
Another thing I haven't decided on and need some input is what
backend(s) to focus on for accelerated uploads (and downlaods) of YUV
image data. The obvious choices are GL and XCB, but GL is still
experimental and XCB suffers from X not having any support for YUV (no,
xv doesn't count). (Before anyone asks: In the long run both should be
well supported, but I still need something to start with).
gst-plugins-cairo:
There are no real changes to the API since the hackfest. I just updated
the internals to conform to the new API and thread-safety guarantees
from Cairo. (And that was mostly deleting code).
My first goal here is to make the gstreamer sink the default videosink.
And after that work on improved embedding of video streams (think
browsers, but also Clutter) and a live editing framework for video
editors like Pitivi. (annotating Byzanz screencasts should be trivial!)
What I've been wondering about though is what subset of currently
supported YUV/RGB formats is a sane subset. It doesn't make a lot of
sense to me to support all those crazy formats when no one uses them.
I'm in particular thinking about Y41B, NV21, and all those weird RGB
formats. Can we agree on a list of formats that we want to support
"natively" as opposed to "convert asap to a native format"?
So, that's the current state of the gstreamer-cairo saga, now it's
Søren's, Carl's, Chris' and others turn to comment on it if they wanna
avoid me merging it as is. ;)
Cheers,
Benjamin
More information about the cairo
mailing list