[cairo] Release management for Cairo

[Emmanuele Bassi]

Hi;

thanks for your perspective. It's really appreciated.

On Mon, 26 Apr 2021 at 11:28, Heiko Lewin <hlewin at gmx.de> wrote:

> We'd like to add API and functionality to it, but the lack of response for
> those changes makes iterating and fixing issues very complicated; we are,
> after all, stretched too thin ourselves. Not knowing if a change will
> effectively land, or be part of a release, naturally leads to workarounds;
> downstream distributors end up either shipping snapshots of Cairo, or we
> end up with issue reports because we have to depend on unstable features.
> This inevitably leads to friction between end users, distributors, and
> maintainers, and instead of working together we end up trying to route
> around one another. This is an untenable situation for everyone.
>
> I am not sure what changes you mean and haven't seen any discussion not
> MRs for them, but I will try to make some general points. cairo is a
> relatively old software project. You can see already where things that did
> not really fit into the design were added. This makes adding new APIs
> problematic. Of course it comes down to the concrete matter of things you
> want to add, but in general my advice, whenever it is possible to layer
> some feature on top of cairo, would be to do so. There are things that
> cannot be done "from the outside", though, and for these things chances are
> you will find an open ear to listen to ideas.
>
>From the GTK developers perspective, we *already* add things on the
"outside" of Cairo. Sadly, glyph composition is still something that has to
live in multiple components, and its implementation is dictated by how text
rendering works more than considerations on the initial Cairo design.

The text rendering stack on Linux is already fairly complex, and in various
cases its complexity is the result of Conway's Law more than actual
software engineering practices. This situation isn't helped if entire
components simply cease to function as part of the stack.

>
> Ideally, I'd like to help with the maintenance of Cairo. I am not an
> expert in the tesselation code, or in font rendering, or in the image
> scaling code; but I can deal with making releases, keeping the CI running,
> automating the website maintenance, triaging issues, and fixing the build.
> More importantly, since Cairo is still a GTK dependency, I can spend my
> work time on those tasks.
>
> As cairo is seriously lacking development resources, you then should have
> a look at the unstable features you mentioned and see what you can do to
> fix the problem. I think most issues can be fixed without expert knowledge
> but only require someone to dig into the code - if the features are already
> implemented, that is. If you stumble about something that does need special
> insight, well.... good luck. Most of the code is orphaned.
>

Yes, that's why I want to drop a lot of this orphaned code—especially the
one that is known to never be enabled.

>
> Of course, since this might look too good to be true: my offer comes with
> strings attached, and this is where things might get controversial.
>
> First of all: Cairo is full of conditionally built code that is barely
> maintained, and seldom enabled. This makes maintenance more complicated
> because nobody is really exercising the whole code base except for some
> long tail end user. Cairo has too many science experiments, and too many
> small fiefdoms, and not enough people that care about the whole project.
> People end up enabling some random feature without realising that it's a
> trap; downstream packagers don't really know what to enable at compile
> time, either.
>
> As a description of a symptom this might be true, of course. But the easy
> advice would be to not enable any feature that you don't miss.
>
This is not "the easy" way, sadly.

Code costs, even code that isn't built. Every time you have to change
things, you have to verify that your change covers every use case—except
people don't do that, because nobody builds niche backends. The test suite
doesn't even run for them: it barely runs for the image surface.

More importantly: it gives out a false perception of what Cairo can do, and
makes people waste their time. People look at Cairo today and think that
cairo-gl is a viable solution for rendering vectors on the GPU. It is very
much not, unless you like your rendering to be wrong and slow; but maybe
that's enough for your use case—except that now that you sunk a ton of time
and effort, you realise that nobody is actually shipping Cairo with the GL
support enabled. Anywhere. Now you have to figure out a replacement, and
you will never use Cairo again.

Leaving dead code in a project is an abdication of responsibility—even when
done in the hope that somebody will turn up and fix it. Spoiler: nobody
will, because the broken code will drift further and further, and at some
point it'll require a rewrite, which is a *daunting* perspective for
anybody, let alone a first time contributor.

> I am not quite sure what you mean by the care for the whole project versus
> "fiefdoms". Different parts of cairo were written by different people who
> seem to have lost interest in the course of time. This, of course, is
> perfectly fine, but now it can be felt that major parts of cairo are
> lacking experts. You will definitely not find the monarch of backend X who
> closely watches everything that happens in his domain and strives to
> realize his greater plan - unfortunately.
>
Well, at the very least we still have people who understand the Xlib and
XCB backends; we might even have people with enough knowledge, or
willingness to hack on the Windows backend and the Quartz. The problem is,
of course, that we have a chicken and egg problem: nobody feels entitled to
take ownership of some piece of code because they don't know if their
changes will ever be released.

As an example: for a long while, the GTK project has had issues finding
people willing to contribute to the Windows and macOS backends. Our code
was not great, but more importantly: the people who originally wrote it
either left the project or simply stopped caring. This kept people from
contributing, and in turn made people work around issues in GTK even though
they could contribute a fix upstream.

We ended up breaking this spiral through the introduction of a reliable CI
pipeline, which allowed us to at least know if a change would build—the
minimum bar to clear. We ensured that our build system would work on every
platform. We started refactoring our own internals to make sure that people
could contribute to GTK without having to learn X11. We reduced the amount
of build options and toggles, to create a reliable development environment.
We also decided to empower more people with small contributions: not by
telling them "now you maintain this subsystem", but involving them into the
decisions regarding larger changes in the project. We turned fiefdoms into
commons.

Many of these changes are the result of better infrastructure—the same
infrastructure Freedesktop now has—but more importantly: it was a change in
how we maintain the project.

>
> In the eyes of the lack of dev-resources this might seem good and well.
> But how do you come to this distinction? From my point of view the
> "supported backends" are just as abandoned as is the rest. For example,
> there is no actve developer with a deeper understanding of PDF, or PS. The
> quartz-backend sometimes gets fixed by mac-users that stumble upon bugs
> (thanks for that!), but has open issues that nobody can really work on.
>
The list of "supported" backends I identified are actually backends that
are a) enabled by every common Linux distribution b) used by GTK and
Firefox. In other words: those are the backends that are exercised by the
vast majority of remaining Cairo users.

The Xlib backend is used by GTK on X11; the Windows backend is used by GTK
on Windows; the Quartz backend is, for all its problems, still used by GTK3
on macOS. The PS and PDF backends are used by GTK applications when it
comes to printing, as well as Firefox on Linux/Unix.

If there are issues on those, we can at least verify that they don't
regress. To be honest, I'd be more than happy to drop the Quartz backend:
it's basically busted and has been so since the introduction of color
spaces and HiDPI displays on macOS, almost 10 years ago.

> My stance on this issue would be more like: If there are no concrete
> complaints, things seem to work just fine. And if there are complaint: This
> is open-source. Fix your problem!
>
I'm sorry to say, but this is pretty terrible advice and it demonstrably
does not work—as you may have noticed, nobody has shown up to fix Cairo. On
the contrary, projects have stopped using Cairo except for very specific
domains, and they are scrambling trying to find replacements.

In part this is caused by the design of Cairo, which does not match what
general purpose computers have been moving towards over the past 15
years—dedicated GPU cores with programmable pipelines; but I'd contend that
the biggest contributor to the exodus is literally the uncertainty in how
Cairo is maintained since Carl Worth (and Chris Wilson) have left the
project.

> It's not that I would find the proposal indiscussable ( I have no real
> interest in those backends - why not drop PS, Xlib, Win32 and Tee as well?
> ), but I do not see a strong reason to do so. Most of the
> removal-candidates on your list are listed as experimantal/incomplete.
> That's it: if someone has interest in working on those, they are welcome.
> Lack of active development, however, cannot be criterion for removal: You
> would have to throw away far more than those...
>
Why? And, also: why would it be a problem to throw away more code?

The Cairo-drm backend has been a science experiment since day 1; it hasn't
seen changes in the past 10 years, and it doesn't even build on recent
versions of the Linux DRM subsystem, as far as I can tell. The Qt backend
does not build, and it's another science experiment, just like the old Skia
backend was. DirectFB is entirely abandoned upstream, to the point that the
domain itself has lapsed and it's now squatted by a random blog. Cairo-cogl
was another science experiment, and Cogl is unmaintained upstream (I know:
I was one of the maintainers of Cogl, and we never used Cairo-cogl to begin
with).

The Cairo-GL backend is not enabled by any Linux distribution; the only
known consumer of that API was WebKitGTK, which has moved on for its
compositing because cairo-gl is known to have performance issues and
because nobody would dare enable cairo-gl on platforms shipping WebKitGTK.

In practice, all the backends I listed are either not enabled by default
because they are known to be broken, or they are tied to unmaintained
dependencies.

I know that people randomly show up to fix a backend or two, every once in
a while. I appreciate their work, I really do, but a patch every five years
is not a sustainable maintenance position.

Plus, it's not like the code is erased from the whole of existence; if
people wish to resurrect a backend, they can still find it in the Git
history, or in older stable branches.

> The reason you give (easier development) is abstract in the light of
> concrete circumstances. If, for example, a feature would need to be
> implemented in (some) backends, you definately will have to implement it in
> the image backend. If you have done that you already have implemented the
> default-fallback for backends lacking a specialized implementation. Such a
> fallback isn't performant, pretty or elegant but could be improved later -
> if there is someone interested in doing so, that is.
>
> You would have to come up with concrete proposals, but in my eyes cairo is
> more or less "finished". No new features, no new responsibilities.
> Improving what is there by fixing bugs is the top priority. cairo has a
> pretty narrow scope (vector graphics), which makes it a viable choice if
> that is exactly what you need. You don't get that often with less than 2MB
> of library size these days.
>
I value your perspective, but I'd like to point out that it is *incredibly*
narrow.

I do have to care about the whole desktop thing—both Linux and not. Text
rendering still very much relies on Cairo, and text rendering does not
cease to exist just because we have figured out Unicode.

On top of that, while we have successfully moved a lot of rendering to the
GPU, Cairo is still very much the fallback on various platforms.

Basically: while from a certain perspective Cairo might be feature
complete, it turns out that what Cairo does cover more use cases than that,
and in that regard, Cairo is not "finished" at all.

This is, essentially, the currently exercised subset of Cairo that GTK
> needs in order to work, plus the requirements of other users of Cairo
> according to Debian's code search. I'd like to be able to say: "if you want
> to keep a backend alive, please show up and maintain it", but the truth is:
> none of the backends I'd like to remove are really used by anything that is
> currently relevant outside of a moderately long tail of niche users. If you
> rely on custom builds for your project/product, I'm sure you can also
> continue using Cairo 1.16. I'd be happy to backport all the relevant
> changes to the 1.16 branch, before sealing it for good.
>
> I am here. We still at least compile the gl backend. I fix bugs. Now what?
> Have you fixed the PDF or quartz backend recently? Those are not feature
> complete.
>
> This is a bad argument of course. I hope you see why.
>
Thanks for fixing bugs, but I hope you see why random fixes does not imply
that the backend is maintained or even used. At the very least, we know
that the Quartz and PDF backends *are* used, because they are enabled by
default and we know people use Cairo-based toolkits on macOS, and print
documents.

>
> With the Autotools build gone, it's possible to avoid targeting the
> minimum common denominator, and ensure that Cairo can build properly on
> multiple platforms. GTK currently builds Cairo as a Meson sub-project on
> Windows (with the MSVC toolchain) and macOS, which gives us a functional
> baseline.
>
> We recently had a short discussion about dropping autotools recently.
> There are a few things autotools does that are missing from meson as for
> now. Mostly the release-packaging stuff. As for me: I will not port that
> over. Maybe you should talk to Tim how this could be accomplished. Despite
> of the fact that we currently use autotools for cairo I would not really
> mind it disappearing.
>

A bunch of those scripts are all kinds of redundant. Switching to Meson has
the advantage of giving us a very reliable "dist" command, which ensures
that we have a release archive that matches the contents of the Git
repository; it is not self-hosting, which reduces the issues with the
tarball containing random files. A bunch of the tests done at release time
can be moved to a dist script. Finally, a lot of the checks should just be
part of the CI pipeline that runs for every merge request. Having a
Makefile rule upload tarballs on a remote server is… quaint… and to be
honest, I'd rather have releases hosted on GitLab, as it would likely
reduce the sysadmin effort, and lower the barrier to contribution, since it
would remove the need to give out SSH access to people.


> I have started these changes in a separate branch, called "Tanis"[3]
> available on GitLab:
>
>   https://gitlab.freedesktop.org/ebassi/cairo/-/tree/tanis
>
> Some of the commits there are fairly uncontroversial, and deal with fixing
> compiler warnings or other small build issues. The commits prefixed with
> "tanis" are the ones that I feel are going to be more controversial.
>
> As I said above, I am not really involved with the development of Cairo; I
> am just a downstream consumer of it. The larger Cairo community might not
> be enthused by somebody dropping by and asking to assume the role of
> release manager with a laundry list of changes that drop code and features.
> I *do* empathise with that, and I'm entirely prepared to a "get lost" reply.
>
>
> Sadly, it seems that the other option is keeping Cairo on life support,
> landing changes but never really releasing them; since there are still
> downstream users of Cairo, and those projects do have quite some traction
> in the larger free and open source ecosystem, either those projects will
> accelerate dropping Cairo, or will be forced to perform a soft fork, with
> an ABI compatible release from a separate branch or repository. Neither of
> those solutions would be acceptable to me, and I'm sure multiple
> downstreams would also find it very hard to swallow.
>
> If you do updates frequently, I would advice to always use the latest
> master version. Occasionally a new bug might be introduced but if you look
> at the list of previous issues it might seem that things can only get
> better.
>
That's not how things work in the larger free and open source ecosystem,
sadly.

Downstream distributors do not want to ship random Git snapshots, and
consumers of the Cairo API really want to be able to depend on an actual
stable release.


> So, I guess my question is: would this plan be acceptable to the people
> still involved with the development of Cairo?
>
> To give some information: We use cairo in a commercial product and I got
> into development by fixing some simple issues that occurred in that scope.
> Now I am deemed an active developer by some people and try to have a look
> at merge-requests and new issues. I occasionally do work on cairo that is
> not covered by the commercial interest.
>
> I would be fine with the removal of backends as I do not use them. But
> when talking about responsibility I have to say, I do not know. Every now
> and then someone comes along who does seem to use the more exotic stuff.
> And - given that cairo is free and open-source - what right would I have to
> remove code that seemingly does not cause any problems? You say people
> enable stuff and realize it does not work. You say it would ease
> development to remove backends. From what I see I just cannot say I am
> sure, you are right.
>
> I do not feel comfortable making radical decisions on such an old project.
>
Which is why I volunteer for that. I am much more comfortable with this,
and I think one of the problems with Cairo as it stands today is that
nobody feels empowered to make *any* decision.

Again: I appreciate your perspective and contributions, and commercial
users of Cairo are very much an important part of the community.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cairographics.org/archives/cairo/attachments/20210426/25d636bf/attachment-0001.htm>