Blog

  • Twobuntu talk

    I was honored to present my ideas about Ubuntu & Debian at Debconf 7, with a number of Ubuntu and Debian developers present, including Mark (first time I saw him in person) and DPL Sam Hocevar. There was no slide projector, so I had to read my slides, even though everyone says you shouldn’t do that… πŸ™‚

    I was very impressed with the quality of the people at Debconf. I met lots of smart geeks, and even several anthropologists who study Debian as a social organization. You do not ship an operating system by accident, just as you don’t build an airplane by accident.

    Here are my slides.
    Here is a link to all of the Debconf 7 videos.
    Here is a link to my talk, and the subsequent discussion.

    A couple of thoughts from the discussion after the slides:

    One of the big pieces of confusion is that Mark believes that publishing patches in multiple / granular formats is helpful to Debian. Lars Risan, who had a beer with Mark after my talk said that Mark strongly disagreed with my analogy of throwing code over the wall and that Ubuntu is doing everything technically possible to make its changes available to Debian. However, a major point in my talk is that throwing patches over the wall is not helpful because the person on the other side needs to get up to speed on the patch. You can’t just hand someone 100s or 1000s of lines of code because that person will have to take time to learn what is going on (to integrate them, fix any problems, etc.) and this takes a lot of time, perhaps as much time as if the Ubuntu patch doesn’t exist. This is why it took Debian many months to integrate modular X even though it had Ubuntu’s patches. Because it takes so much time, Ubuntu’s patches in practical terms are not helpful to Debian. The format, frequency, etc. of the patches doesn’t matter, it is the time to learn something which is the cost. If two people in different teams are each learning the same thing, they are not “standing on the shoulders of giants” but re-inventing and re-learning from scratch, just like the old, dark proprietary software days. I believe that, even though I spent some minutes on this topic, Mark did not understand this concept. Perhaps he would disagree with the scope of the problem, etc., but he would need to understand it first to disagree with it.

    One of the points in my talk is that Ubuntu is exploiting a loophole in the GPL. You can of course fork code and do with it what you want, but that the spirit of GPL is to cooperate, work in the same codebase, and to fork only under extreme circumstances. Two examples I cited are the Linux kernel and Wikipedia. Jonathan Riddell has responded that Ubuntu is not exploiting a loophole in the GPL because he disagrees that there is only one Wikipedia and one Linux kernel. However, he doesn’t explain what he means. Perhaps he thinks there are multiple Linux kernels because each distro ships their own. Along these lines, Mark at one point responded to my point that there is only one kernel by asking: “Which one is that, the Red Hat kernel?” Only if Ubuntu had at least considered using the Red Hat kernel would his answer have been a substantive rebuttal. The larger point is that the various distro kernels differ from the mainline kernels mostly because they contain backports of fixes from the mainline kernel and other minor tweaks. These changes are very small in nature compared to the chasm of the separate buglist, greatly divergent code, etc. which exist between Ubuntu and Debian. Maybe I should have said that there is “one Linux kernel and Wikipedia team.” However, I see these statements as largely synonymous.

    One of the points I made is that Ubuntu did not show up to Debconf with a list of workitems for Debian: Ubuntu takes whatever they can get from Debian, but any Debian limitations they just fix on their own. Mark responded by saying that the idea that Ubuntu should demand a list of workitems from Debian is a misunderstanding of the way free software works. However, I believe he is nitpicking my choice of the word “workitems.” My point is that much of the design and feature work that happens in Ubuntu is going on in a separate community/world. If the cross-organizational collaboration was healthy, Ubuntu would come to Debconf with a list of things like: “One of Ubuntu’s biggest problems is (x). How can we work together on this problem?” This doesn’t happen because Ubuntu is attacking problems on their own, without the deep and rich consultation if the communities were together. And, another downside of this situation is that the center of gravity on a number of areas is moving away from Debian.

    Mark believes that Ubuntu is good for Debian, because so many people use apt-get. This is a bad way of measuring “good.” What about quantifiable metrics? Here is one: I would measure something as good for Debian if it increases the number of DDs. Anthony Towns told me here that the number of DDs has not dramatically increased in the last 3 years, and the number of new maintainers joining every year is even slightly decreasing. Therefore, this is one metric which demonstrates that Ubuntu has not been good for Debian. Mark says that he thinks it would be great if Debian had 10,000 DDs, but if the Debian userbase is not growing, how is it going to get there?!

    Mark said that: “Ubuntu could have derived from Red Hat, SuSE, Gentoo…” However, Mark did not do this, and instead forked Debian and hired a number of its best developers. Furthermore, I also disagree with the premise that he needed to derive from any codebase at all. A major theme of my talk is that what makes Debian so great is that it supports many hardware platforms and contains many software packages, built to work together. Why could his improvements not be done directly into Debian, just like HP and many other companies and interests have done? Finally, if Mark thinks he could have had so much success deriving from any other distro, Debian is not the special thing he claims it to be.

    A number of Debianites agreed with me on many aspects of these issues, but who have given up trying to convince Mark, etc. Several told me that they were worried about the center of gravity shifting away from Debian, for example. I believe a reasonable compromise to minimize the damage to the community is to let users use Ubuntu, but to encourage all developers to join Debian. I met someone working on MOTU games, and his group by themselves realized that doing their work directly in Debian was a better approach.

    A final thought: if Debian took all of Ubuntu’s patches, which Mark would like Debian to do, and shipped on the same day, how would a user decide which distro to install?

    What do you think? Post your comments below.

  • Debian Etch: Solid, Crufty, Some Assembly Required

    Debian is the the quietest big Linux distro. I see hourly posts on Distrowatch, Slashdot and Digg about the latest builds of Ubuntu and SUSE, and even Mark Shuttleworth’s wearing of a KDE t-shirt is considered news. I presume that things are fine inside Debian and that no gnus is good gnus, but also I believe, as Oscar Wilde said: “What’s worse than being talked about is not being talked about.”

    I’ve written a number of posts about Ubuntu and Debian on this website in the last few weeks, saying that: in principle I should be running Debian, that Ubuntu is too small a team for the size of their customer base and buglist, questioning the size of the fork Ubuntu has made, questioning whether Debian is stagnant, etc. I decided to see if I could test any of these assumptions, so I grabbed the latest Etch bits from the Debian-testing tree and installed it on a 2-year-old Sony VAIO laptop.

    Even though I had grabbed just a daily build, the very nice Debian installer recognized my hardware, it generally installed and ran like a champ. However, my hopes and expectations for Linux in 2006 are greater:

    • The mouse pointer was very slow; it took 15 swipes to move the mouse across the screen and tweaking the speed of the mouse in the Gnome UI didn’t help. There is a bug in Debian’s database which suggested a workaround and that the problem isn’t specific to Debian, but I haven’t seen this problem on the other distros. I plugged in a USB mouse and kept going.
    • My ipw 2100 didn’t work. I know there is a firmware freedom issue, but the drivers ship with the kernel, and they work on other distros. I just plugged in an Ethernet cable, which worked right away, and kept going.
    • Menus: There are simple Ubuntu-style menus, and then a deeply nested set of Debian menus. I’m not sure if this is part a transition, but I do think they should ship with one nice simple default set.
    • There is a a lot of crufty old apps which get installed by default: mutt, sh, tcsh, xeyes, etc., etc. Given that it is so easy to install software, even if you assume that Debian is for experts, why not ship the prettiest and most popular and perhaps put the old command-line stuff into a meta-package?
    • I tried to hibernate and received an error message saying that it didn’t appear software support was compiled into the kernel. Neither was any support compiled into the UI for me to choose hibernate from the logout menu.
    • Sound didn’t work. Totem-gstreamer would crash saying “failed to connect to the d-bus daemon.”
    • Everything is built against gstreamer .08 even though .10 is in the repositories; I presume this is being worked on.
    • Apache is 2.0, latest is 2.2
    • Drupal is 4.5.8, latest is 4.7
    • GCC 4.0.3, latest is 4.1
    • OpenOffice 2.0.1, latest is 2.0.2
    • Tomboy 0.3.3, latest is 0.3.5. No F-Spot or Beagle
    • No Ekiga
    • Tried to install the free ATI 3-d drivers (the proprietary ones I wish I had the freedom to install are nowhere to be found) but it couldn’t install because it complained about missing xserver-xorg-core 1:0.99.0-1, which it said wasn’t installable.
    • I installed the Xfce Window Manager which worked fine, but I didn’t see the application menu for starting any apps, like Xubuntu has.
    • Lots of little things: the “Add to Panel…” dialog box is lamer than the one that ships with Dapper. Sorry I don’t have a screenshot. The dialog to change the desktop background had repaint issues.
    • On the good side, Debian ships with a more sane set of fonts than Ubuntu, which has a pile of international fonts, causing my few Western fonts to get lost.

    After that I gave up. I know this is an incremental build and I could get more or perhaps even all things working, but I feel like Debian will need to do more than fix the RC bugs to keep up with the other distros and tempt me.

    I know some people will read this and say that I don’t understand Debian, or that Debian shouldn’t be easy to use. Whatever. Linux is growing in double digits, but not Debian. And if you aren’t growing, you are shrinking through attrition. (I wonder if Debian is firing on all cylinders–I’d be interested to know how many of Debian’s developers do less than 4 hours a week of work.) I did a subsequent install of OpenSUSE, and was very impressed with both its stability and polish, so it isn’t just Ubuntu which thinks that the last 5% is important. It should be no surprise that Debian is only #7 on the distrowatch list.

    I also didn’t see that Debian is benefiting as much from Ubuntu as I thought it would. Power management features, and many new packages and new versions of code in Ubuntu have not made it upstream. Perhaps this is work Debian has signed up to do but hasn’t done, but it does suggest that things might work better if more work was done directly in Debian; code is flowing downstream much better than upstream right now.

    I wish Debian was trying a bit harder for my affection. I’ve boasted that I’ll probably be ready to maintain a Debian box in the Etch timeframe, but now I’m not so sure. Ubuntu has lots of bugs, but I don’t think more than a handful are regressions. It appears that Ubuntu has all the advantage of Debian with no new disadvantages.

    Finally, while I do see that Debian has given Ubuntu a great base and that Ubuntu could probably be helping Debian more, it seems fair to say that Ubuntu has taken Debian and polished it up quite a bit and they deserve more props for that than I and many others in the Debian community have given them. Of course, when Canonical adds another 24 people, Ubuntu will really rock…

    I have been rude and disrespectful to thee, Ubuntu. Will you take me back? [Eyeing the SUSE CD in the corner–I’ve always had a thing for Germans.]

    P.S. Some people are angry that I dare criticize Debian when its not ready, but I want Debian to succeed, this build is the culmination of 12 months of work, with 6 months remaining, and I calls ’em as I sees ’em; that’s why you guys pay me the big bucks!

  • Ubuntu 2.0 (Twobuntu?!)

    Written May, 2006, updated May, 2007 in preparation for Debconf 7
    Update 2: The slideshow and my thoughts on the subsequent discussion are available here.

    Ubuntu is a very exciting distro, gaining critical mass, and demonstrating the possibilities for a powerful and simple Linux desktop. I can’t help spending time handicapping the top Linux distros; it is more fun than the Windows, Mac & OS/2 wars of the past, especially as we are participants rather than merely spectators.

    I don’t think that there need be just one distro or family of distros which will own the market, and Linux needn’t become a monoculture to succeed on the desktop. I run Gnome, but can run KDE apps as well, so while we can make this situation better, we don’t need to. Today there are 20 million Linux computers and lots of different distros and I don’t know why there can’t be 1 billion Linux computers with lots of different distros. I’m sure Novell would be ecstatic with only 50 million Linux customers.

    For those living inside the Debian or Ubuntu worlds, the issue of their relationship is an old topic, but it it will continue to evolve as they learn from their experience. Ubuntu already exists and is a great distro, and I support the friendly competition and energy that it brings. However, a goal should be to figure out how to best harness each’s efforts. I strongly believe that Debian needs to stay the center of gravity; when a feature is done first in Ubuntu, the center of gravity shifts away. Given the fact that Ubuntu has no “list of features” they’d like Debian to implement, you can say they are on course to become the center of gravity.

    As background, I believe that Ubuntu’s existence is an exploitation of a loophole in the GPL: you are allowed to take someone’s code, and do anything you want with it, but the presumption is that you should work together on one codebase. As you make improvements, you should immediately put them back into the standard codebase, and take ownership of any issues with that improvement. When someone makes a fix to the Linux kernel, they don’t use that improvement to make a competing Linux kernel!

    While Ubuntu forking Debian goes against the spirit of cooperation embodied in the GPL, there are also practical reasons why it is a bad idea.

    Ubuntu’s challenges
    While Ubuntu has 2 million (now 8 million) customers, it has lots of challenges as well. The biggest is that the maintainers cannot keep up with the incoming bugs. In fact, if the most popular Linux distro has the smallest engineering team and is the least stable, it threatens the perception of the desktop distro market as a whole. Meanwhile, if Debian had so many new customers and 10,000 (now 30,000) new bugs, I’ll bet they could pick up the pace for the increased workload, especially if it came with a few New Maintainer applications. That mild shock to their system might even be healthy. Excitement brings in new people and makes existing people work harder. Ubuntu is sapping excitement, which is killing Debian.

    I think there are 3 main reasons why Ubuntu is buried in bugs:

    • Ubuntu is finding bugs which exist in Debian but that Debian hasn’t fixed or doesn’t even know about. This is very worrisome because Ubuntu has 10,000 (now 30,000) active bugs, with 5,000 (now 15,000 ) still unconfirmed. Not only is it a problem that Ubuntu is shipping with so many bugs, Debian is shipping stable releases with potential “release critical” bugs that it does not know about. It therefore questions the merits of Debian even making stable releases.
    • Ubuntu is making deep core changes but it doesn’t have the resources to deal with the issues across all the hardware and software because Debian’s expertise is not being fully utilized.
    • Ubuntu snapshots bits from Debian-unstable which gives them the latest and greatest Debian code, but it is code which hasn’t been debugged yet.

    Here’s a question: of the first 75 bugs in Ubuntu, how many exist and are filed in Debian?

    Efficiency by doing work directly in Debian
    When Ubuntu did modular X, the packages were provided for Debian but it still needed to be adapted and re-debugged and in the end turned out to be only a β€œstarting point,” according to the Debian engineer who did this work. In other words, even though Debian had Ubuntu’s patches, it was still months of work–not that much better than if they didn’t have the changes at all! It sounds helpful to throw code over the wall, or publish patches on a website, but it takes time to ramp up expertise to understand them. If you have to ramp up expertise in 2 organizations, the global community’s time is not being spent efficiently. Hiring Debian developers, even to work full-time on Ubuntu, does not much help, and actually weakens, Debian.

    Note that this inefficiency issue doesn’t happen as much going the other way. Ubuntu doesn’t have Apache maintainers, so they just take the code as-is, and probably file any bugs upstream. In the areas where they both have maintainers: the kernel, X, Gnome, KDE, FireFox, OpenOffice, you will find a ton of duplicative efforts and expertise.

    Code is infinitely malleable
    Ubuntu wanted modular X, and so they forked Debian and added this feature. However, no one has said why this feature couldn’t have been added directly to Debian. Whether Canonical believes that Debian is missing features or shipping too slowly, it can simply hire Debian engineers to directly fix these problems, without needing to create a separate codebase. The codebase they started with was 100% Debian, so clearly it was possible to add their features to the Debian codebase.

    Reasons for a divergent fork no longer apply
    Debian and Ubuntu have converged a lot in the last year. Ubuntu’s initial changes were big, however, Debian has caught up with Ubuntu and both want to add Xen and Xgl to Etch/Feisty Fawn. Now that the architectural differences are smaller, the reasons to add features outside Debian become smaller. It only makes sense to add features directly to Debian that Debian wants, but from looking at Ubuntu’s spec list, I’ll bet that Debian wants at least 90% and potentially all, of Ubuntu’s improvements. If there are no features that Ubuntu is adding that Debian doesn’t want, what does that say about the wisdom of separate codebases? Even worse, some significant features that Ubuntu has, Debian is no longer even motivated to add: any user of Debian that has wanted better power management for his laptop is now using Ubuntu, and so Debian 4.0’s power management is several releases behind Ubuntu’s and probably not as well tested.

    Ubuntu shouldn’t be doing the big stuff first
    Some of Ubuntu’s architectural bets: modular X, 2.6 kernel, new GCC, etc. were safe bets to make. However, core decisions are better left for Debian to decide and whatever decision Debian makes will typically be fine with Ubuntu. Just imagine what a mess would be created if Ubuntu adopts Upstart and Debian something else. Lots of arbitrary differences could cause unnecessary divergence over time.

    Building on top of Debian Unstable
    Another significant decision Ubuntu made is to ship the Unstable branch. This is an interesting idea that made sense during the Sarge era because Sarge had lots of new code in unstable, but the stable code was ancient. While this suggests that even an unstable branch is a pretty solid base, I’m not sure if Ubuntu can ever be confident in the quality of an LTS release if it has known, and untested and therefore unknown, bugs. Either Debian has ‘release critical’ bugs or it does not.

    What is Twobuntu, other than a silly name?

    I’m suggesting a few changes to the way Ubuntu develops software:

    • When Ubuntu decides to add a feature, the question needs to be asked if it is a core feature that Debian wants. If so, the Ubuntu developer should either do the work directly inside Debian, or takes ownership for it in Debian after implementing it in Ubuntu.
    • If Ubuntu adds features in Debian first, it won’t ship with Ubuntu till it hits Debian stable (or unstable) but it will not take much more time to do something in Debian.
    • If Ubuntu doesn’t make big changes outside of Debian, it could more easily triage: for example, all X and power management bugs would flow directly upstream. By having all the bugs in one place, it makes it easier to hand them out amongst a unified team of developers, and to analyze the state of a component.

    I realize that whether to do a feature inside or outside Debian is a hard question. An exercise for the reader: if Ubuntu decides to adopt a Smart Package Manager, should this be done inside Debian or not?

    Because code is infinitely malleable, separate codebases create arbitrary boundaries between ideas and people. There are various ways to work better together, and the goals should be to make development maximally efficient, allow Debian to better feel Ubuntu’s investments, help Ubuntu with its pitched battle against bugs, ensure that bugs and code flow freely, and that Ubuntu and Debian’s architecture, user and developer communities be well connected.

    Update: this old proposal is somewhat complicated, but it would allow Ubuntu to exist as a separate entity if they believe that there are enough features they want that Debian doesn’t care about. If it turns out that Debian wants all Ubuntu’s features except for the orange, then it makes sense to think big. I would prefer merging, because I don’t like the idea of disparate communities. In general, the goal should be for Ubuntu and Debian to share a source tree, bug list and community, like Wikipedia and the Linux kernel do. Whether they have Technical Committees, or Community Councils, codes of conduct, etc. doesn’t matter nearly as much.

  • Open Letter to Ubuntu’s Benevolent Dictator

    Dear Mark Shuttleworth aka Sabdfl,

    I don’t know if you saw my post “10,000 bugs away from World Domination” which made my day by getting Digged and was even read by one of your engineers Scott James Remnant, but the thesis is that the challenge to widespread use of Linux is not that it doesn’t have wobbly windows and Xen support, though these are exciting features I look forward to in Eft, the problem is that it has bugs.

    PC Hardware sucks

    Few non-geeks fully realize how building software for PCs is staggeringly complicated. The work involved in getting every driver ready to handle sleep & hibernate is enormous, and that’s just one row in the hardware matrix. Brian Valentine, a Windows VP, once described Windows as a testing organization, and that is one of the reasons why it is so successful and why it ships so slowly. Microsoft software has its faults in addition to being buggy, but it is still pretty thoroughly debugged and they have buildings full of hardware and the electricity bills to prove it. (Think Steve Jobs could stomach buying a WinBook XLi to test their ‘fantastic‘ little platform on? Support for the Intel processor doesn’t get Mac OS X anywhere close to support for the PC platform.)

    I think Linux needs some serious love in a couple of usability scenarios such as external monitors and printing, but from researching your bug list and doing anecdotal analysis of installs on 6 different machines I believe the current biggest challenge for humans with Ubuntu is bugs which only show up on certain hardware: install race conditions, power management issues, video card crashes, screen resolutions which can’t be set in the UI, hangs on startup, soundcards and bluetooth devices unrecognized, etc.

    Ubuntu might be 10,000 bugs Away From World Domination, but a tremendous milestone is the 5,000 bugs to reach World Installation.

    It is possible to work around almost any bug: hack your xorg.conf, disable power management, try the install again, swap hardware, etc. but these problems hurt the perception of Linux, waste even experts’ time, and are structural impediments to its widespread use. Whether Linux has the right set of applications and is ready for phase 3 (profit) of World Domination is mostly irrelevant until the system is up and running.

    Ubuntu’s Maturity vs. SUSE & Fedora

    Bugfixing is time-consuming and not particularly rewarding. What’s worse, from looking at your incoming workload, I believe that even if Ubuntu had 10x the number of devs, you would all still be running at 110%. (If you did grow that much, you’d then equal the resources of the big dogs SUSE and Redhat.)

    Another challenge for Ubuntu is that in addition to its backlog of bugs, it is less mature than Redhat and SUSE. I recently saw an IBM laptop just like mine running SUSE, and hibernation worked with visual status of what was happening. I can only dream of those days on Ubuntu; hibernate has started working literally hours ago, though the flashing brown lines do not fill me with confidence. I know this is just one anecdote, but I have others. πŸ™‚

    The good news for desktop Linux is that Redhat and SUSE are learning from Ubuntu that they have done a bad job at evangelism, polish, freedom, and frequent releases. They also still have some catching up to some important Debian innovations:

    • install that works, including wireless
    • resizing NTFS on install [UPDATE: SUSE does this, but not Redhat]
    • an excellent package manager, with an easy to use UI (how’d Debian stoop to that?)
    • a free update service that works and which even supports in-place OS upgrades
    • a distribution which can scale down to embedded devices
    • a mature, stable, big, coherent platform
    • Other good stuff? I barely know what I am talking about here.

    Debian’s excellent qualities, and those which Ubuntu bring, make Debian/Ubuntu an excellent platform, but Ubuntu might very well be the OS that shows us the promise of Linux on the desktop, but not necessarily the one that keeps us.

    Where to harvest eyeballs?

    If I had your cards and credit at a Vegas blackjack table, I’d double-down. πŸ™‚ Meanwhile, there should be ways to better use Debian’s resources for your common aims. Debian is a team of 1600 maintainers who has provided you a base set of 18,000 packages and while they aren’t necessarily interested in re-coloring their swirl orange, they presumably would like to have the improvements in desktop scenarios and the work on the larger variety of hardware that Ubuntu’s efforts and userbase bring to the table. It would be interesting to know how many Ubuntu bugs exist in Debian or would exist in Debian if they were using the same verson of the upstream code. In other words, if few of your bugs are Ubuntu specific, then we know for certain that their 3200 eyeballs would be helpful.

    Perhaps you could consider radical things like aligning every 2nd or 3rd release with Debian, and therefore share compilers, the desktop kernel and more of the workload. I know this was not easy in the past, but both groups have been through a lot of changes in the last 2 years and you are now more closely aligned, sharing GCC 4.0, Kernel 2.6, modular X, etc. I also see Debian plans to ship this year and has done work on Xen and Xgl which aligns it well with Eft. Finally, these would be great releases to provide long-term support.

    OSS Rulez

    Once the bugs are fixed, it is then a good time to talk about Linux’s structural advantages. One of the biggest is the many packages. The Linux kernel isn’t what people will appreciate, it is the applications on top. When I talk to people about Linux, I am often asked whether there are any apps to run on it!! Behind the “Applications – Add/Remove…” menu item in Ubuntu is a pretty dialog box with a incredibly comprehensive set of applications from a Logo VM for Slashdotters in diapers, to many games equal to the task of multislacking that Windows provided with Solitaire, to all kinds of productivity and multimedia apps in between. To buy all that stuff in Windows would cost a fortune and in Linux it can be installed in just one click with no questions or nagging. (Of course, OSS also has other great communities of software, such as the one surrounding PHP, which has helped make Apache so popular.) Linux apps could all use a coat of polish, but with just a couple of doublings of desktop marketshare, plenty of geek resources and money will become available, and Linux will double 6 more times to reach 1 billion PCs.

    Its a fun time to be running Linux on the desktop. It is close to being ready and the pace is incremental, but steady. The best news is that you can see the pace quickening. The 2.6.8 (800KB) kernel had ~550 contributors, while 2.6.14 (2.1MB) had 844, a growth of 50% in a timespan of 14 months. If the OSS dev efforts everywhere were to grow at that pace, life would be very good, and Microsoft’s position will finally be accepted as unsustainable. Now that Scott McNealy is gone, someone should tell the new CEO of Sun that a Linux desktop has more C# apps than Java apps, a stunning failure given Java’s enormous investments in the ostensibly open JCP and a 7-year head start. Let us hope that at least, the J2SE and smaller VMs become OSS. If I could free one chunk of code, it would be that.

    Having just come in from the cold, I have great respect for Dapper, Debian, GNU/Linux, and to the free software movement as a whole. As the neocons say about Iran, “Faster, please!”

    -Keith

    P.S. I wrote an Open Letter to the DPL recently, saying that in theory I should be running Debian rather than Ubuntu, but also that Debian seems stagnant. I do think some of my ideas about backports and proprietary software were wrong, but I still believe that installing new apps post-ship, and proprietary apps and drivers present tough technical and philosophical challenges.

    P.P.S. Last October, I did an interview with Bradley C. Edwards, one of world’s experts on the Space Elevator, in which we cover a bunch of topics. Perhaps the next time you go to space will be on a Space Elevator.

    1000 readers but only 8 Diggs? I’m going to cry

    UPDATE: Thank you for making it down here. Vote in the poll!

  • Open Letter to the new Debian Project Leader

    Dear DPL Anthony Towns,

    I read your post-election comments about your mission as DPL to be increasing the tempo of Debian, and I think that is an excellent theme. This is the 21st century: Google existed only on paper 8 years ago, China and India are just coming online and will eventually contribute to OSS in a manner commensurate with their size, and maybe even in the Middle East and Africa they’ll quit killing each other or dying of pandemics and become productive members of the modern world. In 20 years, Linux will be running on 1 billion devices smaller than a PC and 1 billion PCs and Nicholas Negroponte will be giving us $100 to take one of his laptops. 1 billion PCs running Linux requires 20 years of 22% annual growth starting from ~20 million today. It needn’t happen that steadily and slowly as FireFox made it to 100 million in 1 year.

    Linux, and the millions of lines of OSS on top and underneath it, is at an interesting inflection point: without fanfare, steadily making progress in the embedded, server, supercomputer, space, etc. space, but with little desktop marketshare and almost even smaller mindshare in the outside world. The lack of mindshare on the desktop is headwind for the server and embedded market and still allows for doubt of the legitimacy of the OSS development process.

    OSS Picking up Steam
    My initial experience with Debian was inauspicious. I was running Fedora Core 4, happily at the time, when I randomly bought a Linux magazine bundled with a Sarge (Debian 3.1) DVD, read a nice review of it, saw that it shipped with the 2.4 kernel, and promptly tossed it into the trash. As it turns out, 2.6 is supported in that release so my facts were wrong, but I stand by my sentiment. I have since discovered that 2.4 was only the default for Sarge and that now Debian is even transitioning away from the 2.4 kernel, and that is good. A Debian based on 2.4 might be a good reason for a derivative distribution, assuming there is still enough interest.

    For comparison, FireFox is shipping once per year, Gnome is shipping twice per year and the Linux kernel is shipping 4 times per year. The Open Source movement is not only moving quickly but actually gaining momentum. The only constant is not simply change, but an increasing rate of change. This is in contrast to Debian, in which the last release took 3 years and the one before that took 2 years. I believe there is no technical reason why you cannot ship whenever you’d like. The Linux kernel merges in changes whenever they are ready without anything holding up the train. Your codebase is much bigger and your changes are more frequently more intrusive, but I believe there has to be a way to work things out even if it simply requires more resources.

    Ubuntu & Debian
    My second experience with Debian, running Ubuntu, has lasted much longer, and I am a happy customer and so is my father, who can’t really tell that he’s not running Windows anymore, though he does understand the geopolitical significance of it.

    I have nothing against Ubuntu, and I love my operating system very much, but I believe that in principle, I should be running Debian. The Ubuntu guys took the efforts of your decade-old project, now containing 1,600 developers, added 10-30 of their own, threw in some new artwork, and have created a huge amount of excitement. Their success is a testament to their efforts, but to Debian’s even more. Of course, Debian’s success is a testament to many other people and so on, but Debian has built a large organization, focused itself on one big noble, common, goal and has built a superb system upon which there are 130 derivatives–one of which now threatens to dwarf Debian itself.

    Everyone agrees that Ubuntu couldn’t exist without Debian, but I also believe that Debian is better setup to take Ubuntu where it needs to go. There are hints that the Ubuntu team feels like they brought a pork chop suit to a lion’s den. Ubuntu’s user base and development team is growing exponentially, but I believe they could get there much faster with more of Debian’s help. Its a simple fact of math that their many new bugs can be more easily divided between 1600 people than 10-30.

    I am not a zero-sum person, and while I do believe that every time someone installs Ubuntu, it is good for Debian, I also see a downside. When he files bugs, the knowledge gained by the dev fixing the bug will not necessarily flow upstream to you guys. When someone writes software for Ubuntu, it may not run on a Debian system. I think benefits flow downstream better than upstream and I’m not sure if Canonical’s new tools will fix that.

    Ubuntu is a relative and therefore a friend, but they want both desktop and server customers and therefore threaten the center of gravity of Debian and its 130 derivatives. And if Debian developers think the excitement is elsewhere, they might not necessarily move to Ubuntu. I wonder whether it would be better for Ubuntu to send most interested developers upstream to you guys, or whether you could share work on the desktop kernel which still needs a lot of work for laptops.

    What can Debian learn from Ubuntu?
    I believe an important lesson to learn from Ubuntu is that excitement is generated when you ship fast. Also, it is important to focus on the desktop and ease of use. Developers are users of your software, and users become future developers. We all start out as noobs. Eric S. Raymond wrote a great memo about the difficulty of getting printing working in Linux and if software is too hard for him to use, then it is just not completely ready for the outside world yet. All the code is there: the drivers, the network protocols and the management UI; they just require spit and polish. You can build a powerful system which is also easy to use and apt and Debian-installer are superb examples of that.

    Is Debian Stagnant?
    While Ubuntu doesn’t have the large, experienced team that you have, they do have excitement and momentum. When I look through the Debian website, I see some worrying metrics:

    • It is taking hundreds of days for new packages to be added, or for requests for help to be fulfilled.
    • There are 189 bugs assigned to the x86 kernel, and the average age of those bugs is 1 year
    • The Debian-apache e-mail alias is deathly quiet, other than spam (some cleverly containing bug numbers in the subject!), and mails wondering about Apache 2.2 go unanswered.
    • You’ve got churn on your bug database at the rate of 12,000 entries per month or 600 per workday. If half of those are by the dev and a full-time dev can work on 5 bugs per day, then you’ve got the equivalent of 60 full-time devs working on bug fixing, or that your 1,600 devs are doing 1.5 hours of bugfixing per week.
      I’m not going to judge anyone else’s volunteer contributions, but it is interesting to have some high-level measurements of the excitement of the team. I’d be curious to know what others think of these swags and how they compare to the time people spend hacking in other projects, maintaining their free blogs, myspace pages or other hobbies.
    • As Debian matures, your current set of volunteers might graduate from college, become satisfied or bored or otherwise move on.

    A lot of what your devs do on a day to day base is hacking–integrating new versions of code, debugging it and keeping the system running. It is important stuff, but it is different from writing code and I believe it needs a constant stream of fresh blood to keep up with this somewhat tedious, labor-intensive task, and to allow your experienced people to solve the harder problems. You should be able to grow the team of contributors proportionally with the userbase if you want that. A lot of people would be excited to work on Debian and you should harness that.

    What are the Big Issues?
    In addition to a focus on desktop and polish, I think it is important to think big, long-term, and strategically. Things like apt and debian-installer play a big part in the definition of Debian. What future investments should your team be making? You could create a process whereby the community identifies the best big issues and then the DPL would provide the leadership to see that these efforts get off the ground. This can all happen in a meritocratic, volunteer process if there is buy-in. Here are a couple to think about:

    • Allow users to run new versions of code without upgrading the OS. It is an unsupported scenario on both Ubuntu and Debian to run FireFox 1.5 on your released OSes. Ironically, my mom would be okay with FireFox 1.0 for years on end, but your most passionate customers are not. There are workarounds, but they are time-consuming. The current policy is to only support fixes for security bugs, but FireFox 1.5 contains many security enhancements, some of which must not been backported, and I’ll bet the FireFox team considers FireFox 1.5 to be better in every way, including security. There is a backports mechanism, but it is not a first-class mechanism.
      It should even be possible to test pre-releases of FireFox and so be ready to sim-ship with the Mozilla folks–that would be exciting! Likewise, OpenOffice.org is going to start shipping every 3 months. Upgrading apps after install is not only a problem for the big apps, but also the small ones: GtkPod is my latest discovery of an app which requires many steps to install if not using the ‘official bits’. [Update: Backports being ‘unsupported’ may be a bigger problem in Ubuntu than Debian.]
    • Are there challenges which would limit your growth? If you had 10x the users, could you scale out your download mirrors and other infrastructure to handle it? I personally think a peer-to-peer APT would be cool and I know such a tool will get written if it makes sense, but developers don’t pay the bandwidth bills.
    • We can hate closed source, but we cannot wish it away. Closed source video, network, modem and biometric drivers, apps like Flash, Real, Skype and Adobe Reader, codecs, and Sun’s Java 5 VM are examples of closed code that are an important part of a modern computer and need to be supported, grudgingly.

    I don’t even know if those are good problems, I’m just proposing that you guys find some and start chipping away at them. There are many challenges Linus will never get to, unless he gets pissed off and creates a ‘Linus Gnu/Linux’ distro.

    Debian as Man of The Year
    Many people in the outside world still do not get even the basics of why Open Source is viable and actually superior to closed source and your organization needs to continue to make explaining this part of the public message.

    I also find it interesting that Debian has very little name recognition compared to MySQL, Redhat and IBM, yet the fact that Debian is such a large effort is newsworthy and should get you at the seat at the table with anyone you’d like. Google made it on the cover of Time, and Debian could win ‘Man of The Year.’

    Developer Productivity
    By far, the biggest thing which is slowing down both open and closed source software development is the use of old tools. C and C++ are fundamentally 30 year-old programming languages and they don’t have garbage collection, metadata, language support for multi-threading, modern class libraries, do not agree on what a string is or how to allocate memory, are not portable, and the compilers are huge, ugly and slow. Line for line, a Java app might be 20% slower than a C app, but algorithms define the speed of code and as Anders Hejlsberg said of .Net, a VM is the best use of Moore’s to come around in a long time. There are many huge benefits to modern languages but the biggest is that a developer is up to 10x more productive. I won’t argue that kernel mode code should be written in Java–yet–but the kernel is only a small portion of a distro and much of the other code could easily be.

    I think Scott McNealy should be fired by Sun’s Board of Directors for his poor stewardship of Java (I run more C# apps than Java apps on Ubuntu!) but in the meanwhile, you should encourage code to be written in any managed language: Perl, Python, PHP, Java, C#, etc. Less VMs is definitely better, but the toothpaste is already out of the tube and multiple VMs aren’t a serious cost.

    Fini
    Some of this may be silly, obvious or other, but I offer it humbly as food for thought to an organization who has built something big and important, all of which prepares it well for the many exciting developments ahead.

    Update: I also wrote an Open Letter to Ubuntu’s leader Mark Shuttleworth, talking to him about Ubuntu’s challenges.