I was honored to present my ideas about Ubuntu & Debian at Debconf 7, with a number of Ubuntu and Debian developers present, including Mark (first time I saw him in person) and DPL Sam Hocevar. There was no slide projector, so I had to read my slides, even though everyone says you shouldn’t do that… 🙂

I was very impressed with the quality of the people at Debconf. I met lots of smart geeks, and even several anthropologists who study Debian as a social organization. You do not ship an operating system by accident, just as you don’t build an airplane by accident.

Here are my slides.
Here is a link to all of the Debconf 7 videos.
Here is a link to my talk, and the subsequent discussion.

A couple of thoughts from the discussion after the slides:

One of the big pieces of confusion is that Mark believes that publishing patches in multiple / granular formats is helpful to Debian. Lars Risan, who had a beer with Mark after my talk said that Mark strongly disagreed with my analogy of throwing code over the wall and that Ubuntu is doing everything technically possible to make its changes available to Debian. However, a major point in my talk is that throwing patches over the wall is not helpful because the person on the other side needs to get up to speed on the patch. You can’t just hand someone 100s or 1000s of lines of code because that person will have to take time to learn what is going on (to integrate them, fix any problems, etc.) and this takes a lot of time, perhaps as much time as if the Ubuntu patch doesn’t exist. This is why it took Debian many months to integrate modular X even though it had Ubuntu’s patches. Because it takes so much time, Ubuntu’s patches in practical terms are not helpful to Debian. The format, frequency, etc. of the patches doesn’t matter, it is the time to learn something which is the cost. If two people in different teams are each learning the same thing, they are not “standing on the shoulders of giants” but re-inventing and re-learning from scratch, just like the old, dark proprietary software days. I believe that, even though I spent some minutes on this topic, Mark did not understand this concept. Perhaps he would disagree with the scope of the problem, etc., but he would need to understand it first to disagree with it.

One of the points in my talk is that Ubuntu is exploiting a loophole in the GPL. You can of course fork code and do with it what you want, but that the spirit of GPL is to cooperate, work in the same codebase, and to fork only under extreme circumstances. Two examples I cited are the Linux kernel and Wikipedia. Jonathan Riddell has responded that Ubuntu is not exploiting a loophole in the GPL because he disagrees that there is only one Wikipedia and one Linux kernel. However, he doesn’t explain what he means. Perhaps he thinks there are multiple Linux kernels because each distro ships their own. Along these lines, Mark at one point responded to my point that there is only one kernel by asking: “Which one is that, the Red Hat kernel?” Only if Ubuntu had at least considered using the Red Hat kernel would his answer have been a substantive rebuttal. The larger point is that the various distro kernels differ from the mainline kernels mostly because they contain backports of fixes from the mainline kernel and other minor tweaks. These changes are very small in nature compared to the chasm of the separate buglist, greatly divergent code, etc. which exist between Ubuntu and Debian. Maybe I should have said that there is “one Linux kernel and Wikipedia team.” However, I see these statements as largely synonymous.

One of the points I made is that Ubuntu did not show up to Debconf with a list of workitems for Debian: Ubuntu takes whatever they can get from Debian, but any Debian limitations they just fix on their own. Mark responded by saying that the idea that Ubuntu should demand a list of workitems from Debian is a misunderstanding of the way free software works. However, I believe he is nitpicking my choice of the word “workitems.” My point is that much of the design and feature work that happens in Ubuntu is going on in a separate community/world. If the cross-organizational collaboration was healthy, Ubuntu would come to Debconf with a list of things like: “One of Ubuntu’s biggest problems is (x). How can we work together on this problem?” This doesn’t happen because Ubuntu is attacking problems on their own, without the deep and rich consultation if the communities were together. And, another downside of this situation is that the center of gravity on a number of areas is moving away from Debian.

Mark believes that Ubuntu is good for Debian, because so many people use apt-get. This is a bad way of measuring “good.” What about quantifiable metrics? Here is one: I would measure something as good for Debian if it increases the number of DDs. Anthony Towns told me here that the number of DDs has not dramatically increased in the last 3 years, and the number of new maintainers joining every year is even slightly decreasing. Therefore, this is one metric which demonstrates that Ubuntu has not been good for Debian. Mark says that he thinks it would be great if Debian had 10,000 DDs, but if the Debian userbase is not growing, how is it going to get there?!

Mark said that: “Ubuntu could have derived from Red Hat, SuSE, Gentoo…” However, Mark did not do this, and instead forked Debian and hired a number of its best developers. Furthermore, I also disagree with the premise that he needed to derive from any codebase at all. A major theme of my talk is that what makes Debian so great is that it supports many hardware platforms and contains many software packages, built to work together. Why could his improvements not be done directly into Debian, just like HP and many other companies and interests have done? Finally, if Mark thinks he could have had so much success deriving from any other distro, Debian is not the special thing he claims it to be.

A number of Debianites agreed with me on many aspects of these issues, but who have given up trying to convince Mark, etc. Several told me that they were worried about the center of gravity shifting away from Debian, for example. I believe a reasonable compromise to minimize the damage to the community is to let users use Ubuntu, but to encourage all developers to join Debian. I met someone working on MOTU games, and his group by themselves realized that doing their work directly in Debian was a better approach.

A final thought: if Debian took all of Ubuntu’s patches, which Mark would like Debian to do, and shipped on the same day, how would a user decide which distro to install?

What do you think? Post your comments below.

Debian is the the quietest big Linux distro. I see hourly posts on Distrowatch, Slashdot and Digg about the latest builds of Ubuntu and SUSE, and even Mark Shuttleworth’s wearing of a KDE t-shirt is considered news. I presume that things are fine inside Debian and that no gnus is good gnus, but also I believe, as Oscar Wilde said: “What’s worse than being talked about is not being talked about.”

I’ve written a number of posts about Ubuntu and Debian on this website in the last few weeks, saying that: in principle I should be running Debian, that Ubuntu is too small a team for the size of their customer base and buglist, questioning the size of the fork Ubuntu has made, questioning whether Debian is stagnant, etc. I decided to see if I could test any of these assumptions, so I grabbed the latest Etch bits from the Debian-testing tree and installed it on a 2-year-old Sony VAIO laptop.

Even though I had grabbed just a daily build, the very nice Debian installer recognized my hardware, it generally installed and ran like a champ. However, my hopes and expectations for Linux in 2006 are greater:

• The mouse pointer was very slow; it took 15 swipes to move the mouse across the screen and tweaking the speed of the mouse in the Gnome UI didn’t help. There is a bug in Debian’s database which suggested a workaround and that the problem isn’t specific to Debian, but I haven’t seen this problem on the other distros. I plugged in a USB mouse and kept going.
• My ipw 2100 didn’t work. I know there is a firmware freedom issue, but the drivers ship with the kernel, and they work on other distros. I just plugged in an Ethernet cable, which worked right away, and kept going.
• Menus: There are simple Ubuntu-style menus, and then a deeply nested set of Debian menus. I’m not sure if this is part a transition, but I do think they should ship with one nice simple default set.
• There is a a lot of crufty old apps which get installed by default: mutt, sh, tcsh, xeyes, etc., etc. Given that it is so easy to install software, even if you assume that Debian is for experts, why not ship the prettiest and most popular and perhaps put the old command-line stuff into a meta-package?
• I tried to hibernate and received an error message saying that it didn’t appear software support was compiled into the kernel. Neither was any support compiled into the UI for me to choose hibernate from the logout menu.
• Sound didn’t work. Totem-gstreamer would crash saying “failed to connect to the d-bus daemon.”
• Everything is built against gstreamer .08 even though .10 is in the repositories; I presume this is being worked on.
• Apache is 2.0, latest is 2.2
• Drupal is 4.5.8, latest is 4.7
• GCC 4.0.3, latest is 4.1
• OpenOffice 2.0.1, latest is 2.0.2
• Tomboy 0.3.3, latest is 0.3.5. No F-Spot or Beagle
• No Ekiga
• Tried to install the free ATI 3-d drivers (the proprietary ones I wish I had the freedom to install are nowhere to be found) but it couldn’t install because it complained about missing xserver-xorg-core 1:0.99.0-1, which it said wasn’t installable.
• I installed the Xfce Window Manager which worked fine, but I didn’t see the application menu for starting any apps, like Xubuntu has.
• Lots of little things: the “Add to Panel…” dialog box is lamer than the one that ships with Dapper. Sorry I don’t have a screenshot. The dialog to change the desktop background had repaint issues.
• On the good side, Debian ships with a more sane set of fonts than Ubuntu, which has a pile of international fonts, causing my few Western fonts to get lost.

After that I gave up. I know this is an incremental build and I could get more or perhaps even all things working, but I feel like Debian will need to do more than fix the RC bugs to keep up with the other distros and tempt me.

I know some people will read this and say that I don’t understand Debian, or that Debian shouldn’t be easy to use. Whatever. Linux is growing in double digits, but not Debian. And if you aren’t growing, you are shrinking through attrition. (I wonder if Debian is firing on all cylinders–I’d be interested to know how many of Debian’s developers do less than 4 hours a week of work.) I did a subsequent install of OpenSUSE, and was very impressed with both its stability and polish, so it isn’t just Ubuntu which thinks that the last 5% is important. It should be no surprise that Debian is only #7 on the distrowatch list.

I also didn’t see that Debian is benefiting as much from Ubuntu as I thought it would. Power management features, and many new packages and new versions of code in Ubuntu have not made it upstream. Perhaps this is work Debian has signed up to do but hasn’t done, but it does suggest that things might work better if more work was done directly in Debian; code is flowing downstream much better than upstream right now.

I wish Debian was trying a bit harder for my affection. I’ve boasted that I’ll probably be ready to maintain a Debian box in the Etch timeframe, but now I’m not so sure. Ubuntu has lots of bugs, but I don’t think more than a handful are regressions. It appears that Ubuntu has all the advantage of Debian with no new disadvantages.

Finally, while I do see that Debian has given Ubuntu a great base and that Ubuntu could probably be helping Debian more, it seems fair to say that Ubuntu has taken Debian and polished it up quite a bit and they deserve more props for that than I and many others in the Debian community have given them. Of course, when Canonical adds another 24 people, Ubuntu will really rock…

I have been rude and disrespectful to thee, Ubuntu. Will you take me back? [Eyeing the SUSE CD in the corner–I’ve always had a thing for Germans.]

P.S. Some people are angry that I dare criticize Debian when its not ready, but I want Debian to succeed, this build is the culmination of 12 months of work, with 6 months remaining, and I calls ’em as I sees ’em; that’s why you guys pay me the big bucks!

Written May, 2006, updated May, 2007 in preparation for Debconf 7
Update 2: The slideshow and my thoughts on the subsequent discussion are available here.

Ubuntu is a very exciting distro, gaining critical mass, and demonstrating the possibilities for a powerful and simple Linux desktop. I can’t help spending time handicapping the top Linux distros; it is more fun than the Windows, Mac & OS/2 wars of the past, especially as we are participants rather than merely spectators.

I don’t think that there need be just one distro or family of distros which will own the market, and Linux needn’t become a monoculture to succeed on the desktop. I run Gnome, but can run KDE apps as well, so while we can make this situation better, we don’t need to. Today there are 20 million Linux computers and lots of different distros and I don’t know why there can’t be 1 billion Linux computers with lots of different distros. I’m sure Novell would be ecstatic with only 50 million Linux customers.

For those living inside the Debian or Ubuntu worlds, the issue of their relationship is an old topic, but it it will continue to evolve as they learn from their experience. Ubuntu already exists and is a great distro, and I support the friendly competition and energy that it brings. However, a goal should be to figure out how to best harness each’s efforts. I strongly believe that Debian needs to stay the center of gravity; when a feature is done first in Ubuntu, the center of gravity shifts away. Given the fact that Ubuntu has no “list of features” they’d like Debian to implement, you can say they are on course to become the center of gravity.

As background, I believe that Ubuntu’s existence is an exploitation of a loophole in the GPL: you are allowed to take someone’s code, and do anything you want with it, but the presumption is that you should work together on one codebase. As you make improvements, you should immediately put them back into the standard codebase, and take ownership of any issues with that improvement. When someone makes a fix to the Linux kernel, they don’t use that improvement to make a competing Linux kernel!

While Ubuntu forking Debian goes against the spirit of cooperation embodied in the GPL, there are also practical reasons why it is a bad idea.

Ubuntu’s challenges
While Ubuntu has 2 million (now 8 million) customers, it has lots of challenges as well. The biggest is that the maintainers cannot keep up with the incoming bugs. In fact, if the most popular Linux distro has the smallest engineering team and is the least stable, it threatens the perception of the desktop distro market as a whole. Meanwhile, if Debian had so many new customers and 10,000 (now 30,000) new bugs, I’ll bet they could pick up the pace for the increased workload, especially if it came with a few New Maintainer applications. That mild shock to their system might even be healthy. Excitement brings in new people and makes existing people work harder. Ubuntu is sapping excitement, which is killing Debian.

I think there are 3 main reasons why Ubuntu is buried in bugs:

• Ubuntu is finding bugs which exist in Debian but that Debian hasn’t fixed or doesn’t even know about. This is very worrisome because Ubuntu has 10,000 (now 30,000) active bugs, with 5,000 (now 15,000 ) still unconfirmed. Not only is it a problem that Ubuntu is shipping with so many bugs, Debian is shipping stable releases with potential “release critical” bugs that it does not know about. It therefore questions the merits of Debian even making stable releases.
• Ubuntu is making deep core changes but it doesn’t have the resources to deal with the issues across all the hardware and software because Debian’s expertise is not being fully utilized.
• Ubuntu snapshots bits from Debian-unstable which gives them the latest and greatest Debian code, but it is code which hasn’t been debugged yet.

Here’s a question: of the first 75 bugs in Ubuntu, how many exist and are filed in Debian?

Efficiency by doing work directly in Debian
When Ubuntu did modular X, the packages were provided for Debian but it still needed to be adapted and re-debugged and in the end turned out to be only a “starting point,” according to the Debian engineer who did this work. In other words, even though Debian had Ubuntu’s patches, it was still months of work–not that much better than if they didn’t have the changes at all! It sounds helpful to throw code over the wall, or publish patches on a website, but it takes time to ramp up expertise to understand them. If you have to ramp up expertise in 2 organizations, the global community’s time is not being spent efficiently. Hiring Debian developers, even to work full-time on Ubuntu, does not much help, and actually weakens, Debian.

Note that this inefficiency issue doesn’t happen as much going the other way. Ubuntu doesn’t have Apache maintainers, so they just take the code as-is, and probably file any bugs upstream. In the areas where they both have maintainers: the kernel, X, Gnome, KDE, FireFox, OpenOffice, you will find a ton of duplicative efforts and expertise.

Code is infinitely malleable
Ubuntu wanted modular X, and so they forked Debian and added this feature. However, no one has said why this feature couldn’t have been added directly to Debian. Whether Canonical believes that Debian is missing features or shipping too slowly, it can simply hire Debian engineers to directly fix these problems, without needing to create a separate codebase. The codebase they started with was 100% Debian, so clearly it was possible to add their features to the Debian codebase.

Reasons for a divergent fork no longer apply
Debian and Ubuntu have converged a lot in the last year. Ubuntu’s initial changes were big, however, Debian has caught up with Ubuntu and both want to add Xen and Xgl to Etch/Feisty Fawn. Now that the architectural differences are smaller, the reasons to add features outside Debian become smaller. It only makes sense to add features directly to Debian that Debian wants, but from looking at Ubuntu’s spec list, I’ll bet that Debian wants at least 90% and potentially all, of Ubuntu’s improvements. If there are no features that Ubuntu is adding that Debian doesn’t want, what does that say about the wisdom of separate codebases? Even worse, some significant features that Ubuntu has, Debian is no longer even motivated to add: any user of Debian that has wanted better power management for his laptop is now using Ubuntu, and so Debian 4.0’s power management is several releases behind Ubuntu’s and probably not as well tested.

Ubuntu shouldn’t be doing the big stuff first
Some of Ubuntu’s architectural bets: modular X, 2.6 kernel, new GCC, etc. were safe bets to make. However, core decisions are better left for Debian to decide and whatever decision Debian makes will typically be fine with Ubuntu. Just imagine what a mess would be created if Ubuntu adopts Upstart and Debian something else. Lots of arbitrary differences could cause unnecessary divergence over time.

Building on top of Debian Unstable
Another significant decision Ubuntu made is to ship the Unstable branch. This is an interesting idea that made sense during the Sarge era because Sarge had lots of new code in unstable, but the stable code was ancient. While this suggests that even an unstable branch is a pretty solid base, I’m not sure if Ubuntu can ever be confident in the quality of an LTS release if it has known, and untested and therefore unknown, bugs. Either Debian has ‘release critical’ bugs or it does not.

What is Twobuntu, other than a silly name?

I’m suggesting a few changes to the way Ubuntu develops software:

• When Ubuntu decides to add a feature, the question needs to be asked if it is a core feature that Debian wants. If so, the Ubuntu developer should either do the work directly inside Debian, or takes ownership for it in Debian after implementing it in Ubuntu.
• If Ubuntu adds features in Debian first, it won’t ship with Ubuntu till it hits Debian stable (or unstable) but it will not take much more time to do something in Debian.
• If Ubuntu doesn’t make big changes outside of Debian, it could more easily triage: for example, all X and power management bugs would flow directly upstream. By having all the bugs in one place, it makes it easier to hand them out amongst a unified team of developers, and to analyze the state of a component.

I realize that whether to do a feature inside or outside Debian is a hard question. An exercise for the reader: if Ubuntu decides to adopt a Smart Package Manager, should this be done inside Debian or not?

Because code is infinitely malleable, separate codebases create arbitrary boundaries between ideas and people. There are various ways to work better together, and the goals should be to make development maximally efficient, allow Debian to better feel Ubuntu’s investments, help Ubuntu with its pitched battle against bugs, ensure that bugs and code flow freely, and that Ubuntu and Debian’s architecture, user and developer communities be well connected.

Update: this old proposal is somewhat complicated, but it would allow Ubuntu to exist as a separate entity if they believe that there are enough features they want that Debian doesn’t care about. If it turns out that Debian wants all Ubuntu’s features except for the orange, then it makes sense to think big. I would prefer merging, because I don’t like the idea of disparate communities. In general, the goal should be for Ubuntu and Debian to share a source tree, bug list and community, like Wikipedia and the Linux kernel do. Whether they have Technical Committees, or Community Councils, codes of conduct, etc. doesn’t matter nearly as much.