December days: Ubuntu

[cjwatson]

jack asked me about Ubuntu.

My direct involvement with what would become Ubuntu started in sunny Málaga, back in February 2004. I'd gone there for a free software conference, partly to talk about Debian release management (which I was heavily involved in back then) and partly to spend some time with a few other like-minded developers who were concerned about the state of dpkg maintenance and wanted to talk about plans for improving it. While I was there - if memory serves, in a lecture theatre listening to one of my first experiences of the marvel that is simultaneous translation - this e-mail popped up:

From: Mark Shuttleworth
To: Colin Watson
Subject: New Debian-related project

Hi Colin

We haven't met, but I've appreciated the contribution you make to
Debian. I'm starting a new project that I hope will be very positive for
Debian, and wondered if you would like to discuss it by telephone. It
will be open source, build on Debian and contribute much work back to
Debian itself.

I'm based in London, so we are in the same timezone which should make it
easier to find a good time to speak if you are interested. Let me have a
phone number and a time to call, and I'll give you a ring.

Cheers,
Mark

Well then, that's interesting. Non-spammy enough that I didn't delete it out of hand (although I heard later that some people who received similar messages did!), but at the time I was a year or so into working at nCipher and rather enjoying it, so I didn't feel a particularly urgent need to follow it up. A few days later the then Debian Project Leader (who happened to be a friend of mine) got in touch and said, hey, this guy is for real, you should probably reply and see what he wants. OK, so I replied and we arranged a phone call, which is perhaps the oddest job interview I've ever had, really more of a pitch but definitely one I thought was very promising indeed, and at the end Mark asked what I was earning at the moment and named a rather bigger number. I mean, I was interested already but that kind of grabs one's attention. I was still a bit hesitant because I'd only been at nCipher a year and didn't want to look like I was job-hopping, but I went and talked it over with ghoti (we'd been going out for a few months) and eventually decided that this was something I'd kick myself for turning down.

So, I went to a meeting in April, and then started in mid-May. I have a later "December days" post scheduled about working from home, but it was certainly very odd for a while. Fortunately I was already used to talking with Debian folks using IRC and e-mail, so it wasn't a horrible culture shock. At that point there was no Ubuntu and certainly no Canonical; there was a holding company called Fieldwave, and my first contract was with them. Everything was "under the radar" in startup terminology, and there was a really fun kind of subversive feel to the operation right at first.

I was initially supposed to be working on the bug tracking system, which made sense because at the time I was maintaining Debian's BTS. I was also a Debian installer hacker by that point, though, and so a short time into my new job the release manager got in touch with me and said something along the lines of "hi Colin, do you think you could quickly put together some installable CD images for us?" Right. That took centre stage and my work on the bug tracking system never really happened - I just sort of unofficially moved sideways into working on the distribution itself, which is the sort of thing that tends to happen anyway when you have a handful of people all scrabbling to do whatever needs to be done. Putting together images ended up involving a mix of installer hacking, fixing up other bits of the system that happened to cause problems, and doing general release management kind of work to make sure everything stayed consistent, which are all things I turn out to be good at but ended up putting me right in the middle of Ubuntu development; for the first year, I handled the mechanics of every single release we ever did (including development milestones), wrote a good deal of software to that end, and tested most of the images along the way, and it wasn't until there was a milestone release while I was very thoroughly out of contact on honeymoon that other people had to start learning how to handle releases.

Of course over ten years I've done a lot of different things, including a distinctly ill-conceived stint in management (no, really, past Colin, this is not something you're good at and you'll hurt yourself trying, just don't); I've worked in project governance and I've been the Ubuntu release manager; but most of my work can be traced back in some way to working on the installer. I really love the fundamental nature of that work, particularly with the approach that d-i takes: you start with a bare kernel and some absolutely minimal tools, and gradually bootstrap your way up to a fully-functional system. For some years, each time we started a new release cycle, something different went wrong that meant that I had to go back and figure out how to make that bootstrapping process work properly again, and it was always a tremendously enjoyable exercise. Later on I started doing boot loader work, notably on GRUB, which is kind of the same thing only more so.

As for what we've produced: I've seen it go from the underdog everyone loved, Debian but with more predictable releases and better support, through to being top of the Linux desktop market and then being a source of controversy as it started making lots of extensive user interface changes with Unity and so forth, to the situation today where we're a major force in cloud installations and trying to make headway on phones. For the most part I've worked in lower parts of the stack - I used to say only somewhat jokingly that my job ended once graphics came up - and so I haven't really been directly involved in very many of the things that have upset people, and I've always been able to stay quite closely connected to Debian development.

I don't like everything we've done, and really it would be surprising if I did. But, ten years ago, Linux was a thing that geeks used but it was really quite difficult to give it to people whose main focus wasn't the computer itself. We weren't the only ones working on that by any means, but Ubuntu put a lot of effort into fixing that and doing really good integration, and the difference between then and now is like night and day. I'm very proud of what we've managed to do. It will be interesting to watch how it continues from a slightly greater remove as of next year.

This post is part of my December days series. Please prompt me!

Comments

[cjwatson]

Ah, right, yes. Not a stupid question at all - having been completely immersed in this forever I sometimes sort of forget that not everyone is! And I tend to use first-person pronouns for both Debian and Ubuntu depending on which hat I'm wearing for a given sentence, which probably doesn't help much.

I've been helping to develop Debian since 2000 or so, and the initial pool of Ubuntu developers mostly came from the Debian community (also GNOME and one or two other places). Part of the legend, which in this case I think is mostly true, is that Mark went on an icebreaker trip to Antarctica in late 2003 and took a copy of the debian-devel mailing list archives with him for light bedtime reading so that he could get an idea of whom to hire. I always did free software work because it seemed like a useful and ethical thing to do with my skillset rather than because it might get me hired, but in this case it turned out to help both ways.

Anyway, Debian is technically called a distribution, in that it takes source code from upstream developers (the standard metaphor is that of a river of code flowing downstream), packages it so that people can get at it in a unified standard way without having to figure out how to cope with all the different ways developers release their software, and distributes it from a central place. Ubuntu is also a distribution, but is technically a Debian derivative: it takes large parts of its source code from Debian (which of course distributes its source code and build scripts and such), builds it independently, and assembles it differently.

Since it's free software the licences in general allow for modification and redistribution. To start with, Ubuntu made a fairly minimal set of changes on top, basically things like making the installation process flow the way we wanted, adjusting the default desktop to the way we wanted it to look, and fixing some bugs in advance of the relevant Debian maintainer getting round to it. As time has gone on our set of changes has got a good deal larger, and in some cases we now either bypass Debian and take code directly from upstream or else do our own upstream development; for instance the Unity desktop environment is developed by Canonical and not packaged in Debian. However, the bulk of the packages available in the Ubuntu archive still come basically unmodified from Debian, even though most of the stuff that's front-and-centre is bespoke or modified in some way. A lot of this is for economic reasons, because it would be really expensive in both time and money to replicate the huge archive of 20000+ source packages that Debian has already put together (a lot of which is niche in some way, of course, but it's often very useful to have that breadth available), and it wouldn't actually make things any better for users if we tried. We do spend a considerable amount of effort just keeping up - any time we make changes to a package, we have to merge those forward to newer versions until such time as they're integrated further upstream from us, if ever - but a lot of that is at least semi-automatable and it's still a lot easier than doing it all from scratch.

In terms of the community, it's been complicated. On the one hand, Canonical employs a bunch of people to do free software development, and a lot of that ends up flowing back to Debian; we send a lot of patches back in bug reports, and there are quite a few packages we just maintain in Debian directly for various reasons (maybe we have the best person for the job, maybe it's just simpler that way, or whatever). That sort of thing is generally supposed to be good. On the other hand, a lot of people resented us creating our own distribution rather than putting the same effort purely into making Debian better, we often get accused of not contributing enough back (sometimes fairly, sometimes not), and some people just don't like the kinds of changes we make. There are plenty of developers who consider themselves part of both communities, myself included. By this point things have stabilised - Ubuntu is obviously around for the long term, some people in Debian are pretty much resolved to dislike whatever we do, but a lot of others are happy to work with us so we can spend a reasonable chunk of time actually getting stuff done rather than arguing about it.

Debian and Ubuntu are two major examples, but if you don't try to be quite so general-purpose then setting up a Linux distribution doesn't require hundreds of people, especially if you base your work on something that already exists, and it's often a useful way to try out different approaches at a medium scale. As a result the distribution world is fearsomely complicated and entangled; there's an amazing chart on Wikipedia (https://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Distribution_Timeline.svg) which tries to visualise everything anyone's ever done in this space and how they relate to each other, and I've only even heard of a ridiculously small fraction of them.

[liv]

Thank you so much for this, it's really helpful and clear, and I have definitely learnt something.

I'm really glad I asked rather than trying to piece this together by searching online. The way you explain this reminds me of how I answer when people ask me questions about Judaism, I try to be fair in explaining all sides of controversies, but obviously do have my own view.

That timeline is indeed amazing! Somebody really had fun with data viz, didn't they?

[cjwatson]

I'm glad that helped. Yes, you can get the headline answer online, but for something like this it's really hard to put together a proper view of how the relationship fits together. And, while a lot of Ubuntu (though not all) is driven by a company which can have a sort of official view of things, Debian is an association of several hundred developers and expecting a unified opinion on just about anything is thus hopeless.

I'd love to have a version of that timeline scaled by log(number of developers) or something so that it's easier to see the large-scale structure. Number of users would be even better but of course nobody really has any way to measure that short of terrible proxy metrics.