Really, I’m not out to destroy Microsoft. That will just be a completely unintentional side effect.
Linus Torvalds, 2003

The Linux mascot, Tux, created by Larry Ewing

The kernel of an operating system (OS) is the central nervous system of a computer. It is the first piece of software that the computer executes, and it manages and mediates access to the hardware. Every piece of hardware needs a corresponding kernel device driver, and you need all of your drivers working before you can run any of your software. The kernel is the center of gravity of a software community, and the battle between free software and Windows is at its lowest level a battle between the Linux and Windows kernels. Microsoft has said that it has bet the company on Windows, and this is not an understatement! If the Windows kernel loses to Linux, then Windows, and Microsoft, is also lost.¹

The Linux kernel is not popular on desktops yet, but it is widely used on servers and embedded devices because it supports thousands of devices and is reliable, clean, and fast. Those qualities are even more impressive when you consider its size: printing out the Linux kernel’s 8,000,000 lines of code would create a stack of paper 30 feet tall! The Linux kernel represents 4,000 man-years of engineering and 80 different companies, and 3,000 programmers have contributed to Linux over just the last couple of years.

That 30-foot stack of code is just the basic kernel. If you include a media player, web browser, word processor, etc., the amount of free software on a computer running Linux might be 10 times the kernel, requiring 40,000 man-years and a printout as tall as a 30-story building.

This 40 man-millennia even ignores the work of users reporting bugs, writing documentation, creating artwork, translating strings, and performing other non-coding tasks. The resulting Linux-based free software stack is an effort that is comparable in complexity to the Space Shuttle. We can argue about whether there are any motivations to write free software, but we can’t argue it already exists!

One of the primary reasons I joined Microsoft was I believed their Windows NT (New Technology) kernel, which is still alive in Windows Vista today, was going to dominate the brains of computers, and eventually even robots. One of Bill Gates’ greatest coups was recognizing that the original Microsoft DOS kernel, the source of most of its profits, and which became the Windows 9x kernel, was not a noteworthy engineering effort. In 1988, Gates recruited David Cutler from Digital Equipment Corporation, a veteran of ten operating systems, to design the product and lead the team to build the Windows NT kernel, that was released as I joined in 1993.

The kernel Cutler and his team developed looks like this:

Windows NT kernel architecture block diagram. Cutler had a Windows 95 doormat outside his office; you were encouraged to wipe your feet thoroughly before entering.

Unfortunately for Microsoft, the original kernel lived on through Windows 95, Windows 98, and into Windows Me. (Microsoft also had Windows CE, a small kernel for embedded devices. Microsoft had three separate kernels for most of my tenure, whereas the same Linux kernel is used on small and big devices.)

Windows has become somewhat popular for servers and devices, but it never achieved the dominance it did on desktop PCs. Perhaps the biggest reason is that its code wasn’t available for others to extend and improve upon. The Linux kernel took off because there are people all over the world, from Sony to Cray, who tweaked it to get it to run on their hardware. If Windows NT had been free from the beginning, there would have been no reason to create Linux. However, now that there is the free and powerful Linux kernel, there is no longer any reason but inertia to use a proprietary kernel.

There are a number of reasons for the superiority of the Linux kernel. But first, I want to describe the software development process. When you understand how the Linux kernel is built, its technical achievements are both more impressive and completely logical.

Distributed Development

In Linux we reject lots of code, and that’s the only way to create a quality kernel. It’s a bit like evolutionary selection: breathtakingly wasteful and incredibly efficient at the same time.
—Ingo Molnar, Linux kernel developer

A portion of the speaker list for the 2006 Linux Kernel Symposium the author attended. Linux kernel development is a distributed effort, which greatly enhances its perspective.

Every 20^th century management book I’ve read assumes that team members work in the same building and speak the same language. Microsoft’s corporate culture was based on the theory that software development was a collaborative effort that should be centralized so that people could work together. As a result, most Microsofties, especially the programmers, were based in Redmond because that was where all the other engineers were located.

Microsoft had a very open development model inside the company: developers would periodically switch teams, collaborate freely together on unreleased code, and join e-mail discussion groups with engineers in similar product roles. These resources of collaboration are some of the many which are unavailable to those outside.

A software program is basically a bunch of human-readable text files, compiled together into a machine-specific binary. When just one person is working on the codebase, things are easy to manage, but when you have multiple people, you need three more tools:

A source control system which records all changes, analogous to the History tab that exists at the top of every Wikipedia page. This provides many capabilities such as going back to find out when and why a code change was made.

A bug or issue tracker keeps a to-do list. This keeps track of the problems over time, and as people leave.

A communications mechanism (in-person, email, chat or forums) where programmers can discuss how to implement features, and work together to troubleshoot problems.

Larger teams have developed more formal processes to, for example, get consensus on when someone is given permission to make changes to source code, but the basic development process in the free software world is little different than what takes place at Microsoft.

The Internet, which was born when Microsoft was a mature company, has changed countless aspects of our lives, including the way software can be developed. Without the Internet, free software could never exist because the developers wouldn’t be able to work together. (Microsoft uses the Internet to do development in other places, although it is still primarily in Redmond.) Microsoft grew up before the birth of the web, and thus has yet to fully embrace this distributed process. For example, Microsoft’s bug databases aren’t available on the Internet. Microsoft isn’t taking the maximum benefit from the knowledge gained by its users because it doesn’t have as many feedback loops.

Linux has achieved enormous gains against Windows, even while, from a 20^th century manager’s perspective, organizing the Linux kernel is a worst-case example for building a productive, prosperous organization. Where are the team-building exercises? The three-year planning retreats? It should be amazing that Linux releases anything at all, let alone dominates the supercomputer business.

Linux succeeds in spite of the cost of working remotely for several reasons. First, the distance between programmers forces people to formalize things, which helps quality: people get together at conferences like the Linux Kernel Symposium and present papers and get consensus and feedback on ideas from a wide variety of people. In addition, the ability to have anyone from around the world contribute is a greater benefit than the cost of putting people together. You couldn’t create a building large enough to host the millions of Wikipedia contributors, let alone the thousands of Linux kernel contributors.

Ingo Molnar’s quote above appears contradictory at first glance, but it is not. Linux receives lots of different ideas; many get rejected, but what remains incorporates the best of all the ideas. The Internet allows you to quickly evolve towards optimal solutions with feedback loops of discussions, tests, and bug reports. (Tests are very important because they give you objective numbers. If you want to see if the new disk cache code is faster, you can compile the kernel, which is a very disk-intensive task, and time the result.)

In the free software movement, the battles aren’t between empires, but rather between engineers fighting over technical details — testosterone-laden VIPs are irrelevant. The Linux kernel community has taken the idea of a meritocracy to the next level. All changes to the official Linux kernel must go through Linus, and his Lieutenant, Andrew Morton, and then the relevant subsystem maintainer — but first, the proposed change has to go through everyone else! All Linux changes are posted to a mailing list where anyone can comment and give opinions. Linus wrote:

The contributors for any given project are self-selected. Someone pointed out that contributions are received not from a random sample, but from people who are interested enough to use the software, learn about how it works, attempt to find solutions to problems they encounter, and actually produce an apparently reasonable fix. Anyone who passes all these filters is highly likely to have something useful to contribute.

Linus’ primary job is to provide technical expertise. He once said his job was to keep out bad code, and that might be enough. Let the varied users of Linux take it to new places, while he will make sure no one is screwing up the existing code along the way.

Linus was asked whether the unpolished nature of a large group of programmers, with disparate backgrounds, created a dreary situation that made him want to go back and work in private, and he said:

I actually like arguing (sometimes a bit too much), so the occasional flame-fest really does nothing but get me pumped up.

At the same time, I’m actually pretty good at just “letting it go”, once I’ve argued enough and am bored with the argument. Part of that is also having to occasionally just admit that you were wrong, and have the ability to send out a “mea culpa” e-mail just saying so.

I tend to care much more about improving the general development model than about the details of some particular subsystem. So that tends to make it easier for me to “let go.” I’ll state my opinions, but even if I’m convinced I’m right, if I’m not actually willing to write the code, in the end I’ll happily be overridden by the people who do write the code.

This is obviously very much a matter of personality.

There are things that I tend to worry about, and that can be really painful, but they are pretty rare. The classic example is the old “Linus doesn’t scale” argument, where it ended up being the case that I really had to fundamentally change the tools I use and how I work. And that was a lot more painful than worrying about the actual code.

Linux Kernel Superiority

Here are the reasons Linux is superior to the Windows kernel:

Refactored Code (Reliability)

Here is a diagram of the Linux kernel:

Layers of the Linux kernel “onion”. The Linux kernel is 50% device drivers, and 25% CPU-specific code. The two inner layers are very generic.

Notice that it is built as an onion and is comprised of many discrete components. The outermost layer of the diagram is device drivers, which is 50% of the code, and more than 75% of its code is hardware-specific. The Microsoft Windows NT kernel diagram, shown several pages back, puts all the device drivers into a little box in the lower left-hand corner, illustrating the difference between theory and reality. In fact, if Microsoft had drawn the kernel mode drivers box as 50% of the Windows NT diagram, they might have understood how a kernel is mostly hardware-specific code, and reconsidered whether it was a business they wanted to get into.

Refactoring (smoothing, refining, simplifying, polishing) is done continuously in Linux. If many drivers have similar tasks, duplicate logic can be pulled out and put into a new subsystem that can then be used by all drivers. In many cases, it isn’t clear until a lot of code is written, that this new subsystem is even worthwhile. There are a number of components in the Linux kernel that evolved out of duplicate logic in multiple places. This flexible but practical approach to writing software has led Linus Torvalds to describe Linux as “Evolution, not Intelligent Design.”

One could argue that evolution is a sign of bad design, but evolution of Linux only happens when there is a need unmet by the current software. Linux initially supported only the Intel 80386 processor because that was what Linus owned. Linux evolved, via the work of many programmers, to support additional processors —more than Windows, and more than any other operating system ever has.

There is also a virtuous cycle here: the more code gets refactored, the less likely it is that a code change will cause a regression; the more code changes don’t cause regressions, the more code can be refactored. You can think about this virtuous cycle two different ways: clean code will lead to even cleaner code, and the cleaner the code, the easier it is for the system to evolve, yet still be stable. Andrew Morton has said that the Linux codebase is steadily improving in quality, even as it has tripled in size.

Greg Kroah-Hartman, maintainer of the USB subsystem in Linux, has told me that as USB hardware design has evolved from version 1.0 to 1.1 to 2.0 over the last decade, the device drivers and internal kernel architecture have also dramatically changed. Because all of the drivers live within the kernel, when the architecture is altered to support the new hardware requirements, the drivers can be adjusted at the same time.

Microsoft doesn’t have a single tree with all the device drivers. Because many hardware companies have their own drivers floating around, Microsoft is obligated to keep the old architecture around so that old code will still run. This increases the size and complexity of the Windows kernel, slows down its development, and in some cases reveals bugs or design flaws that can’t even be fixed. These backward compatibility constraints are one of the biggest reasons Windows takes years to ship. The problem exists not just at the driver layer, but up the entire software stack. When code isn’t freely available and in one place, it makes it hard to evolve. Microsoft has accumulated so much baggage over the years that it could be an airline.

While the internal logic of Linux has evolved a lot in the last ten years, the external programmer interfaces have remained constant. The key to a stable interface is incorporating the right abstractions. One of the best abstractions that Linux adopted from Unix is the file abstraction. In order to perform almost any function on a Linux computer, from reading a web page on a remote website to downloading a picture from a camera, it is necessary to simply use the standard file commands: open and close, read and write.

On my computer, in order to read the temperature of the CPU, I just need to open the (virtual) text file “/proc/acpi/thermal_zone/THM0/temperature” and the data I request is inside:²

temperature: 49 C

In essence, the Linux kernel is a bundle of device drivers that communicate with hardware and reveal themselves as a file system. As new features, security issues, hardware requirements and scenarios confront the Linux kernel, the internal design evolves and improves, but the file system abstraction allows code outside the kernel to remain unchanged over longer periods of time.

Here is a random sample of the change log of the Linux kernel from 2.6.14. As you can see, it is filled with all kinds of cleanup and bugfix work:

spinlock consolidation

fix numa caused compile warnings

ntfs build fix

i8042 – use kzalloc instead of kcalloc

clean up whitespace and formatting in drivers/char/keyboard.c

s3c2410_wdt.c-state_warning.patch

[SCSI] Fix SCSI module removal/device add race

[SCSI] qla2xxx: use wwn_to_u64() transport helper

[SPARC64]: Fix mask formation in tomatillo_wsync_handler()

[ARCNET]: Fix return value from arcnet_send_packet().

Many of the Linux kernel’s code changes are polish and cleanup. Clean code is more reliable and maintainable, and reflects the pride of the free software community.

If you look at the code changes required to make a bugfix, in the vast majority of cases all that is needed is a revision of a few lines of code in a small number of files. A general guideline Linux has for bugfixes is this: if you can’t look at the code change and prove to yourself that it fixes the problem, then perhaps the underlying code is confused, and this fix shouldn’t be added near the end of a release cycle.

According to Stanford University researchers
, the Linux kernel has 0.17 bugs per 1,000 lines of code, 150 times less than average commercial code containing 20-30 bugs per 1,000 lines.³
Microsoft’s Windows bug databases aren’t available on the Internet so it is impossible to make comparisons, but even if Linux isn’t more reliable already, it is setup to become so because the code is simple, well-factored, and all in one place.

Within the free software community, different teams are disparate entities, and so the idea of arbitrarily moving code from one part of the system to another can’t easily happen. Inside Microsoft there are no boundaries, and so code is moved around for short-term performance gains at the cost of extra complexity.

Here is a graph of all the function calls into the OS required to return a simple web request. These pictures demonstrate a visual difference in complexity that often exists between free and proprietary software:

System call graph in Microsoft’s proprietary web server, IIS.

System call graph to return a picture in the free web server Apache.

Uniform Codebase (Reliability, Maintainability, and Usability)

My job is to say “no”, to some extent. If you keep things clean enough and have some kind of requirement for what code can look like, you’re already ahead of the game. —Linus Torvalds

Linux engineers have found a way to run the same codebase on a wide variety of processors, on devices from cellphones to supercomputers, an unprecedented achievement. Linux has been tuned to first, run correctly, and then run efficiently on two, four, eight, and now 1,000 processor machines. Software has infinite malleability, so such a universal kernel has always been possible — it just took a bunch of different hardware companies working together to make it happen.

Putting everything into one codebase helps reliability. Running the Linux kernel on a 32-processor computer shakes out multi-threaded bugs far more quickly than on a two-processor laptop. Running on low-end machines keeps the code small and simple, which makes it run faster on desktops. Features that first appear on laptops and tablets eventually trickle their way down to even smaller devices where the code undergoes even more testing and enhancement. The many hardware and server developers who want extreme reliability ensure that the kernel on my PC is as reliable as Linux’s most demanding customer.

Linux is more flexible than the Windows NT kernel, though both are very clean and flexible. For example, the National Security Agency (NSA) has created a free software component called SELinux (Security Enhancements to Linux) that adds a strong security enforcement mechanism known as Mandatory Access Control.⁴
Making these mechanisms public helps ensure there are no back doors to the NSA’s computers. I will discuss in a later chapter why governments can adopt free software, even for high-security scenarios, but it appears the NSA already understands this concept.

Throughout the Linux world, one has many more choices for just the right tool to do the job. Some might argue that too much choice is a bad thing, but creating many components forces clear boundaries, and survival of the fittest whittles down the inferior choices over time.

Frequent Ship Cycles (Maintainability and Usability)

Microsoft had a motto: “Ship early, ship often.” This philosophy is wise for software development because it forces teams to maintain a high-quality product every day, and the earlier you release, the sooner you can receive and incorporate feedback.

However, this philosophy only works when adopted. Unfortunately, Microsoft’s two biggest products, Windows and Office, do not follow this philosophy. Of course, paying $240 every year for the latest upgrade to Windows “Ultimate” wouldn’t be acceptable either!

The Linux kernel ships every three months. For a product of its size and complexity, Linux’s rate of shipment is unprecedented. This pace has allowed the Linux kernel to ship drivers before Windows, and even before the hardware itself has shipped. Linux supported USB 3.0 before Microsoft, which did not include it in Windows 7. Because Linux is constantly near shipping, you can take any random build from Linus’ computer, put it into a rocket and feel quite confident that it won’t crash.

A big part of the Department of Justice lawsuit against Microsoft focused on the company’s bundling of many software components with their operating system. The government accused Microsoft of excluding third-party software developers and stifling competition. But Microsoft’s tying has been both a blessing and a curse.

The blessing is that by having the pieces work together, they can reuse code and be more integrated. The curse is that Microsoft has created a situation whereby it revises and adds new features to all of its interdependent components at the same time. As a consequence, its components take years to stabilize, and you can’t ship until the last component is ready.⁵

By contrast, in a free OS, software components depend only on released versions. Every team doesn’t try to ship on the same day, so the OS contains the latest versions of all released components.⁶ Separate development organizations have enforced boundaries that have simplified dependencies, and allows everyone to move ahead at whatever pace they choose.

Many users wonder whether free software will ever be as good as proprietary software because presumably the free software guys can’t afford things like usability studies. Having watched usability studies at Microsoft, grabbing people off the street to give their bits of feedback in a room with two-way mirrors is not necessary. The Internet, and all its communications mechanisms, provides a continuous and richer feedback mechanism than you can get with any usability study. In addition, usability studies don’t matter if you can’t incorporate the changes quickly and easily. With clean codebases and frequent ship cycles, usability will happen automatically. I’ve spent a lot of time using Linux and find many applications are perfectly usable.

Shipping a new platform every five years in theory provides Microsoft’s partners a stable platform upon which to build. However, in reality, its results have flaws. For instance, I wasn’t able to install an HP Photodesk 7960 printer driver on Windows Server 2003, although the driver installed perfectly on XP. The subtle and unintentionally undocumented differences between those operating systems, which shipped two years apart, has created compatibility headaches even on Microsoft’s supposedly uniform platform.

Lower Development Costs (Maintainability)

It is much less expensive for hardware vendors to support Linux. If you want to build a device driver, a great place to start is by looking at existing shipping device drivers, an opportunity that Linux offers to everyone. A proprietary “Device Driver Toolkit” with its sample code is never as good as production code. Those expensive kits contain documentation, but not source code — so you sometimes have to guess at what is happening down below.

We find in Windows today that hardware manufacturers have duplicated a bunch of the functionality Windows provides but doesn’t quite fit their needs. For example, IBM includes its own applet and status icon for wireless Internet, so Windows XP on IBM hardware has two. Presumably they weren’t satisfied with the features Windows provided, and weren’t able to fix them. And so they had to build new applets from scratch! This is also what gives Windows a feeling of a jumble of components slapped together.

Here are five of the 100 applets IBM adds to Windows:

Windows XP with 5 of IBM’s 100 extra applets. Notice the large number of status icons on this almost-virgin installation.

Building all of these applets, designing multilingual user interfaces, providing the means to install and configure, etc. is ten times more work than merely writing the device driver, leveraging other shipping drivers, and uploading it to the official codebase.

My Photodesk 7960 printer worked on Windows XP, but didn’t work on Windows Server 2003 because the installation code crashed — which HP shouldn’t be forced to bother with in the first place.

Security (Reliability and Maintainability)

To mess up a Linux box, you need to work at it; to mess up your Windows box, you just need to work on it. —Scott Granneman

Attempting to compare the security of operating systems is a complicated endeavor because there are an infinite number of risks. It is like asking whether one car is safer than another. Against what: Getting hit from the front? Rolling over? Having the gas tank pierced? Its ability to brake?

Furthermore, neither Windows nor Linux are perfectly secure operating systems; both are prone to buffer-overrun viruses, an issue I will discuss in the tools chapter. Furthermore, new threats appear over time, and so each nefarious advancement requires new logic to defend against it. Given these caveats, it is still possible to make some comparisons.

Some of the previous advantages of Linux, such as its simplicity, modularity, support for SELinux, etc. all help with its security. In addition, the nature of how Linux is developed also helps. A document commissioned by the US Military said:

Open source software is potentially subject to scrutiny by many eyes Therefore bugs, security flaws, and poor design cannot hide for long, at least when the software has a community of programmers to support it. And since fixing the code doesn’t depend on a single vendor, patches are often distributed much more rapidly than patches to closed source software.
Can increase code quality and security With closed source software, it’s often difficult to evaluate the quality and security of the code. In addition, closed source software companies have an incentive to delay announcing security flaws or bugs in their product. Often this means that their customers don’t learn of security flaws until weeks or months after the security exploit was known internally. —Open Technology Development Roadmap

Another big difference between Linux and Windows is that Linux was adapted from Unix, which had a multiuser design right from the beginning. In Windows, users have historically had full Administrator access to the machine, including the ability to overwrite system files. When the computer is attacked by a virus, the virus can gain the same capabilities as the user and thereby hide itself inside system files, which make it very difficult to remove. On Linux, I can write only to my own files and have read-only access to all others. Linux’s multiuser nature, and therefore its focus on minimal permissions, minimizes any damage.

One study found that there are about 60,000 known viruses for Windows, and only about 40 for Linux. Another study by Evans Data showed that 8% of Linux developers say their machines have been infected by malicious code, compared to 60% of Windows machines.

Brian Krebs of the Washington Post found that code to exploit flaws in Internet Explorer 6 existed for 284 days in 2006, while Firefox was at risk for just 9. Computer security expert Bruce Schneier recommended in December 2004 that people not run Internet Explorer. Some argue that Linux and Firefox have fewer viruses because they have fewer users, but Apache is well-respected for its security.

For most of my tenure at Microsoft, we worried about features, reliability, and performance, not security. Microsoft’s Chief Research and Strategy Officer, Craig Mundie, said in 2002:

Many of the products we designed in the past have been less secure than they could have been because we were designing with features in mind rather than security.

Microsoft has greatly increased its focus on security in the past few years, and I am sure the security of every product improves with every release, but the baggage of their codebases serve as an ongoing impediment.

Having browsed through the sources to a number of Linux applications, one can say the free codebases are typically cleaner than their Windows counterparts, which makes them easier to secure. The default server-oriented Debian 4.0 Linux distribution requires a mere 170MB of disk space, whereas Windows Server 2003 requires 3 GB. All other things being equal, the fact that Debian is 17 times smaller means it will be more secure. The free database MySQL is a mere 26MB download; the code is clean and small, and therefore much more likely to be reliable and secure.

Another advantage of Linux is that all the applications in a Linux operating system receive security updates. In the Microsoft world, only Microsoft’s code is protected by Windows Update.

While Linux is free to acquire, it can also be cheaper to run and maintain than Windows because of its better security. The city of Manchester in England spent $2 million in 2009 to remove the Conficker worm from their computers.

Linux has learned from Windows

While the Windows NT kernel was state of the art at the time it was released in 1993, most of its good ideas have been learned well and absorbed, in spite of the fact that the code has never been released.

For example, the Linux kernel supports asynchronous I/O (input/output), an innovative way to do reads and writes without tying up “thread” resources. This was an innovation first made widespread in Windows NT.

The ability to load code dynamically is another important feature the Linux kernel adopted from NT and others. Plug and play and suspend and hibernate was a collaboration between Microsoft and hardware companies in the 1990s, and Linux now supports this feature.

Throughout the free software stack, developers have incorporated good ideas from the outside world. There is no Not Invented Here syndrome in free software; a good idea is a good idea, and existing code is even better. In software today, the biggest impediment to sharing ideas is not ego, but license agreements.

The Linux kernel has even learned from Microsoft’s mistakes. For example, one feature added to the Windows NT 4.0 kernel was to put the code that draws widgets into the kernel itself. While this can improve graphics performance, it also means that a bug in the code of a button has the capacity to crash the entire system. The best way to keep a system secure and reliable is to keep as much code as possible in user mode above the kernel, and Linux follows this strategy.⁷

The Feature Race

One of Microsoft’s best arguments against free software over the years has been to tout their new features and use that as “proof” that free software will always be behind proprietary software. The moment a Microsoft product comes out, it always offers features that no one else has. However, most of the features are in fact ones that others do have, and Microsoft is catching up. Nearly every new feature is an evolutionary step forward in a place that Microsoft felt needed work.

But like Microsoft, each team in the free software community is improving their code every day. In fact, because Microsoft takes so long to ship, the free software community has often added features before Microsoft. The website http://kernelnewbies.org displays the latest list of the Linux features added since the previous release 3-4 months before, and it is typically 15 pages long! For example, here is just the list of driver features added to the 2.6.26 version of the Linux kernel, which had a 3-month dev cycle.

Don’t worry if you don’t understand what these things mean as I don’t either. It is just important to understand that even the hardware of computers are too big and complicated for one company to oversee the development of.

This is just a portion of the code changes for that version of Linux, and doesn’t include work in the file systems, networking, performance, architecture-specific work, and so forth.

Free software has added countless features before Microsoft. However, many of these features are so obscure that your eyes would glaze over reading the list. Anyone who focuses on the highly-touted features in Microsoft’s latest release is missing the larger picture: even when these features are compelling, they often make you more deeply tied into the proprietary world and make it harder to migrate away.

Microsoft’s new software always has new features, but usually only one piece of code for a scenario. Linux supports 60 file systems, including several clustering solutions, that allow a seamless scale-out of disk storage amongst a group of servers. Microsoft supports just one, which is tied very tightly to its system, which in turn limits its flexibility. Free software also approaches solutions from different directions. There are several ways to attack virtualization and Linux is working on them in parallel and finding what is common between them.⁸

Linux is Inexorably Winning

The fact that the Linux kernel has these many advantages over Windows means two things. First, the argument that free software engineers are unable to innovate and only copy the work of others is not true.

The head of Windows Server recently said
that free software, “by its very nature, does not allow intellectual property to be built.” This statement is factually incorrect: the Linux kernel is doing many things no kernel has ever done before, just as Wikipedia is doing things no encyclopedia has ever done before.

Second, even if Microsoft gave away the source code to the Windows kernel, the existing free software community would dismiss it. A worldwide community has pounded Linux into the shape it should be. Linus calls Windows “witchcraft”, so why work on an inferior codebase whose details up till now have been opaque?

Charging for an OS

A Linux operating system is an entirely different beast compared to a Microsoft operating system. Microsoft was constantly torn about how much value to invest in Windows, and how much to set aside for extra licensing revenue in other products. Windows Vista has five different versions, (originally they announced eight!), each with basically the same code, but with dramatically different prices:

Microsoft charges $85 to $350 for Windows Vista, but the code in each version is 99% the same.

Microsoft’s natural goal is to put as many new and compelling features into high-end, high-profit versions even though the prices are not correlated to the work required to build the various Vista versions.

Creating multiple versions is tricky because if third-party applications depend on features that aren’t in a particular version of Windows, then the applications won’t run, and the Windows brand is weakened. Therefore, Microsoft would sometimes add logic to cripple the performance of high-end features on low-end versions. In the free software world, no one deliberately cripples their own creation.

When I worked at Microsoft, there were numerous turf wars. For example, the Word team fought with the Windows team over whether WordPad, a tiny Windows applet, should have the ability to read Word’s DOC files. The Windows team wanted to create an operating system that had the ability to display the hundreds of millions of DOC files, but the Word team didn’t want to create a reason for people not to purchase Word. There were also running battles between the teams responsible for Outlook and Outlook Express, Exchange and SQL, Works and Word, FoxPro and Access, Access and VB, SQL Server and Access, PC and XBox. Each team was paranoid about another team adding functionality that would discourage someone from buying their product.

Microsoft also was torn between bundling important features in the operating system, like they did with web browsers, instant messaging and multimedia, and leaving features out in order to reap more money later. Windows doesn’t ship with a dictionary because Office has one. Windows doesn’t come with development tools because those are part of Microsoft’s Visual Studio business. With Linux, anything free is welcomed into the operating system.

In addition to Microsoft’s strategic decisions to exclude certain features, there are also cases of benign neglect. The Sound Recorder in Windows XP lets you only record for 1 minute, a limitation that exists from the days of 16-bit Windows that no one bothered to fix. Microsoft’s official solution is for the customer to purchase Office OneNote.

Applets, command-line tools, and many other important but unsexy parts of an operating system were always allocated very limited resources. Ask Steve Ballmer for resources for the unimportant Sound Recorder, and you would receive a dirty, polystyrene fork hurtling in your direction. In the free software model, anyone, on their time frame, can improve a piece of code that is useful or interesting to them, whether “strategic” or not. Eric Raymond calls this phenomenon a developer “scratching their own itch.” Wikipedia is built almost entirely from this mechanism, and only free software can capture every little advancement.

Complexity of License Agreements

It would be possible to fund the construction of all roads with tolls. This would entail having toll booths at all street corners. Such a system would provide a great incentive to improve roads. It would also have the virtue of causing the users of any given road to pay for that road. However, a toll booth is an artificial obstruction to smooth driving—artificial, because it is not a consequence of how roads or cars work.
—Richard Stallman

Even if you believe it is perfectly okay to charge for software, it is hard to know what to charge consumers for software. Microsoft used to charge a fixed amount for a server product. However, one day they realized that customers who had 10 users connected to a server should be paying less than those who had 100. This observation resulted in the creation of client access licenses (CALs); a concept that required payment for each individual user of server software, but which is an additional burden on the customer. If you have 1,000 employees accessing 1,000 servers, you need to fill out a lot of paperwork.

This was all before the invention of the Internet whereby the number of users of a server could easily be in the thousands and which made many usages of CALs expensive and unsustainable. Therefore, Microsoft moved towards a model where the cost was based on the number of processors in the computer so that little boxes would cost less than big boxes.

This model worked until Intel introduced the concept of “hyperthreading”, which fools a computer into thinking there are two processors inside the computer, but which adds only 15-30% more performance. Microsoft’s customers would of course be unhappy at the thought of purchasing a bunch of new licenses for such a small performance improvement, so Microsoft ended up giving free licenses for hyperthreaded processors.

Then, virtualization was created:

Intel virtualization marketing diagram: Virtualization allows the ability to run multiple operating systems on one computer: each instance thinks it controls the machine. The best uses for virtualization are web hosting, inside corporate data centers, and for software developers.

Virtualization allows different applications to be fully isolated from each other inside different complete instances of an operating system, but at the same time share the CPU and other hardware resources. Isolation is important because even within a single company’s data center, different departments don’t want to run their code on the same machine. If Hotmail went down, they didn’t want it to be the fault of the Microsoft Bob web page.

I was told that inside the cavernous datacenters of a Fortune 500 company, the computers used on average only 15% of their CPU’s capacity. The different departments all maintained their own hardware, which they’d built out to handle the maximum possible load; this is an idea as silly as utilizing only 15% of an office building.

Virtualization gives you software isolation but allows you to share hardware. However, when you install more proprietary software, licensing issues arise. If I put three copies of a database server in separate virtualization instances on a four-processor computer, under many of Microsoft’s licensing models I would have to purchase enough licenses for 12 processors — even though the computer only has four. Like hyperthreading, virtualization is another technology that wasn’t conceived of when Microsoft created their per-processor licensing model.

In a free software environment, you can add new hardware and add or remove applications without paying or keeping track of anything. I will write more about economic aspects later, but for now it is only important to understand that free software sidesteps these hassles, which are often very onerous for enterprises.

An interesting moral dilemma for proprietary software vendors is whether to allow pirated copies to receive security updates.

Free Software Only Costs PCs

Creating a unified industrial software base built around free software will not only make PCs more powerful, but it will also allow us to push their intelligence everywhere, from cars to medical equipment. Free software will be built because it is so valuable to businesses; everyone else will come along for the ride and get it for free.

The proprietary software model has hurt many of the smaller software markets. In fact, except for gaming business which is as much about artistry as software, Microsoft is nearly the last proprietary software company standing. When I was at Microsoft, our biggest competitors were companies such as Borland, WordPerfect, Corel, Lotus, Netscape and Sybase — names you never hear anymore.

There is no Microsoft of educational software selling products used in every school. There are several companies selling proprietary products, perhaps receiving enough revenue to stay in business, but the proprietary software model has prevented them from achieving critical mass.

Recently, I saw an ad for a trivial software applet capable of converting a DVD to the iPod video format. Microsoft managed to convince everybody that each little software product is worth selling. Now you see why Stallman’s analogy of proprietary software as tollbooths is apt — they serve as permanent, pervasive obstacles holding up progress for software that was written years earlier. Free software flows with much less friction. In fact, in the free software world, the definition of a PC operating system completely changes.

A Free Operating System

Our most potent operating system competitor is Linux and the phenomena around open source and free software. The same phenomena fuels competitors to all of our products. The ease of picking up Linux to learn it or to modify some piece of it is very attractive. The academic community, start-up companies, foreign governments and many other constituencies are putting their best work into Linux.
—Bill Gates

The One Laptop Per Child has as much CPU power as a workstation of 1991, but it would be just a shiny box without free software and content.

The biggest difference between Windows and Linux is that free software contains thousands of applications, installable with one click, and managed as one set. A Linux operating system includes all the obvious stuff like a spreadsheet, web browser, and instant messaging, but it also includes tools for making pictures and music, children’s applications, server software, the Bible, development tools, and much more.

Audacity is the most popular free audio editor on Linux. It doesn’t have a talking paper clip: “It looks like you’re trying to add echo. Would you like some help?” But it does provide a well-rounded set of features, and has many effects for the manipulation of sound.

A rich and reliable free audio manipulation tool

Audacity, of course, doesn’t have a one-minute limitation on its ability to record as Windows XP does. Its most important feature is its plugins for import, export, and effects — extensibility is one of free software’s best features.

Developers of free software applications tend to build extensibility plugins as a fundamental way of writing their software because they know their tool will never by itself be able to do all the things people will want. A plugin provides a boundary between things that manage data, and things that manipulate it. The most popular plugins eventually become a part of the base system, but by being built separately, they have forced clean boundaries and modularity.⁹

Every application that Linux has that Windows doesn’t is a feature Windows is missing:

Richard Stallman’s free software vision realized: A free Linux operating system contains an entire store of free applications available with one click, and built to work together. Having so many tools at your disposal makes computers more personal, powerful, productive, and enjoyable. Your computing experience becomes limited only by your creativity.

A free operating system is where Metcalfe’s law meets software: the more people using free software, the more applications will be built for it. Just one piece of free software isn’t useful, but with an entire stack, we can enter a shining age.

Going from today’s 20 million Linux users to the anticipated one billion means the potential for 50 times more resources. The free software stack has several challenges I will discuss throughout this book, but it is important to mention here that few of the applications in the dialog box are as polished or reliable as Firefox. However, many are very powerful, and more than good enough to depend on in a business.

While Linux still needs work, Windows is no day at the beach. Here is an email from Bill Gates’ describing his experience of installing Microsoft MovieMaker on Windows:

From: Bill Gates
Sent: Wednesday, January 15, 2003 10:05 AM
To: Jim Allchin
Cc: Chris Jones (WINDOWS); Bharat Shah (NT); Joe Peterson; Will Poole; Brian Valentine; Anoop Gupta (RESEARCH)

Subject: Windows Usability degradation flame

I am quite disappointed at how Windows Usability has been going backwards and the program management groups don’t drive usability issues.

Let me give you my experience from yesterday.

I decided to download (Moviemaker) and buy the Digital Plus pack … so I went to Microsoft.com. They have a download place so I went there.

The first 5 times I used the site it timed out while trying to bring up the download page. Then after an 8 second delay I got it to come up.

This site is so slow it is unusable.

It wasn’t in the top 5 so I expanded the other 45. These 45 names are totally confusing. These names make stuff like: C:\Documents and Settings\billg\My Documents\My Pictures seem clear.

They are not filtered by the system … and so many of the things are strange. I tried scoping to Media stuff. Still no moviemaker. I typed in movie. Nothing. I typed in movie maker. Nothing.

So I gave up and sent mail to Amir saying – where is this Moviemaker download? Does it exist? So they told me that using the download page to download something was not something they anticipated. They told me to go to the main page search button and type movie maker (not moviemaker!)

I tried that. The site was pathetically slow but after 6 seconds of waiting up it came. I thought for sure now I would see a button to just go do the download. In fact it is more like a puzzle that you get to solve. It told me to go to Windows Update and do a bunch of incantations.

This struck me as completely odd. Why should I have to go somewhere else and do a scan to download moviemaker? So I went to Windows update. Windows Update decides I need to download a bunch of controls. (Not) just once but multiple times where I get to see weird dialog boxes. Doesn’t Windows update know some key to talk to Windows?

Then I did the scan. This took quite some time and I was told it was critical for me to download 17megs of stuff. This is after I was told we were doing delta patches to things but instead just to get 6 things that are labeled in the SCARIEST possible way I had to download 17meg. So I did the download. That part was fast. Then it wanted to do an install. This took 6 minutes and the machine was so slow I couldn’t use it for anything else during this time. What the heck is going on during those 6 minutes? That is crazy. This is after the download was finished.

Then it told me to reboot my machine. Why should I do that? I reboot every night — why should I reboot at that time? So I did the reboot because it INSISTED on it. Of course that meant completely getting rid of all my Outlook state.

So I got back up and running and went to Windows Update again. I forgot why I was in Windows Update at all since all I wanted was to get Moviemaker. So I went back to Microsoft.com and looked at the instructions. I have to click on a folder called WindowsXP. Why should I do that? Windows Update knows I am on Windows XP. What does it mean to have to click on that folder?

So I get a bunch of confusing stuff but sure enough one of them is Moviemaker. So I do the download. The download is fast but the Install takes many minutes. Amazing how slow this thing is. At some point I get told I need to go get Windows Media Series 9 to download.

So I decide I will go do that. This time I get dialogs saying things like “Open” or “Save”. No guidance in the instructions which to do. I have no clue which to do.

The download is fast and the install takes 7 minutes for this thing.

So now I think I am going to have Moviemaker. I go to my add/remove programs place to make sure it is there. It is not there.

What is there? The following garbage is there. Microsoft Autoupdate Exclusive test package, Microsoft Autoupdate Reboot test package, Microsoft Autoupdate testpackage1. Microsoft AUtoupdate testpackage2, Microsoft Autoupdate Test package3.

Someone decided to trash the one part of Windows that was usable? The file system is no longer usable. The registry is not usable. This program listing was one sane place but now it is all crapped up.

But that is just the start of the crap. Later I have listed things like Windows XP Hotfix see Q329048 for more information. What is Q329048? Why are these series of patches listed here? Some of the patches just things like Q810655 instead of saying see Q329048 for more information.

What an absolute mess. Moviemaker is just not there at all. So I give up on Moviemaker and decide to download the Digital Plus Package.

I get told I need to go enter a bunch of information about myself.

I enter it all in and because it decides I have mistyped something I have to try again. Of course it has cleared out most of what I typed.

I try (typing) the right stuff in 5 times and it just keeps clearing things out for me to type them in again.

So after more than an hour of craziness and making my programs list garbage and being scared and seeing that Microsoft.com is a terrible website I haven’t run Moviemaker and I haven’t got the plus package.

The lack of attention to usability represented by these experiences blows my mind. I thought we had reached a low with Windows Network places or the messages I get when I try to use 802.11. (don’t you just love that root certificate message?) When I really get to use the stuff I am sure I will have more feedback.

Linux Distributions

With Linux, each OS distribution carves out a niche to meet its users’ needs. There are specialized versions of Linux containing educational software, tools for musicians, versions dedicated to embedded or low-end hardware, and regional versions of Linux produced in places like Spain and China.

The various distributions have much in common, including the Linux kernel, but use different free software and installation mechanisms. One distribution called Gentoo downloads only one binary, a bootstrapping compiler. The rest of its deliverables are the source code to the components they offer. This gives the user the ability to build a system highly optimized for his hardware.

Some distributions are optimized to run well on old hardware and fit on CDs the size of a credit card:

Damn Small Linux is the most popular Linux for old computers and ships on 80×60 mm CDs.

Linux is very popular on servers, which require an additional focus on performance, reliability, and security. One of the simplest ways to decrease bloat and security risks is to remove the graphical interface:

top – 12:54:17 up 62 days, 20:14, 2 users, load average: 0.16, 0.42, 0.43
Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2% us, 0.3% sy, 0.0% ni, 98.5% id, 0.2% wa, 0.7% hi, 0.2% si
Mem: 514248k total, 489360k used, 24888k free, 79128k buffers
Swap: 1020088k total, 18416k used, 1001672k free, 177528k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2041 root 10 -5 0 0 0 S 0 0.0 0:11.74 usb-storage
1 root 16 0 1564 528 460 S 0 0.1 0:01.09 init
2 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:14.63 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
Screenshot of a text-based Linux process viewer. Just because a computer displays only text doesn’t mean there isn’t a modern, rock-solid operating system underneath.

It is as hard to write a graphical user interface (GUI) for a server as it is to write the server itself: every new feature in the server needs a corresponding improvement in the administration interface. If your server can generate 1,000 records per second, then your user interface had better be able to handle that without choking.¹⁰

While most devices do not need a GUI, desktop users do, and like everything in the free software world today, there are several good choices.

The following map shows the Debian branch of the Linux distribution family tree; they derive from each other, just like in a biological ecosystem:

A portion of the Linux family tree showing the Debian branch, the biggest free software distribution.

Here is a chart showing the relative popularity of Linux distros:

Linux distributions, sorted by popularity. The line shows the divide between both halves of the popularity curve.

What you see here is an almost perfectly smooth curve that illustrates a relatively new idea called the “Long Tail.” One way to think about this idea is to look at the English language. Words like “the” are used with great frequency, but many more words like “teabag” are used infrequently. There is a long tail of infrequently used English words, and to just ignore them would be to throw away much of what makes our language so special.

The lesson of the long tail in business is the importance of catering to customers with special interests. The long tail of Linux distributions means that the creation of a free software ecosystem doesn’t mean the end of the free market, or of competition.

Wikipedia and the Linux kernel are two of the best examples of the fact that free software and the free exchange of ideas can create superior products without licensing fees. The mere existence of these premier products, without a gigantic company behind them, is proof that the proprietary development model is doomed.