Free download:
Soundtrack for the book:
Software Wars, the Movie
The best book explaining free market economics:
The best explanation of space elevator engineering challenges:
I have worked for years on this book. If you enjoyed the free download, a donation of the cost of a newspaper would be appreciated.
I've got the cat, so now I'm just a million dollars and a monocle away from being a James Bond villain. But in this movie, we all get the technology. I am formulating plans...
|
Here are some great quotes by Milton Friedman which describe some of the biggest problems in modern society.
The greatest advances of civilization, whether in architecture or painting, in science and literature, in industry or agriculture, have never come from centralized government.
The most important single central fact about a free market is that no exchange takes place unless both parties benefit.
The only way that has ever been discovered to have a lot of people cooperate together voluntarily is through the free market. And that’s why it’s so essential to preserving individual freedom.
“If you put the federal government in charge of the Sahara Desert, in 5 years there’d be a shortage of sand.”
Is it really true that political self-interest is nobler somehow than economic self-interest?
Many people want the government to protect the consumer. A much more urgent problem is to protect the consumer from the government.
Most economic fallacies derive from the tendency to assume that there is a fixed pie, that one party can gain only at the expense of another.
So that the record of history is absolutely crystal clear. That there is no alternative way, so far discovered, of improving the lot of the ordinary people that can hold a candle to the productive activities that are unleashed by a free enterprise system.
The black market was a way of getting around government controls. It was a way of enabling the free market to work. It was a way of opening up, enabling people.
The Great Depression, like most other periods of severe unemployment, was produced by government mismanagement rather than by any inherent instability of the private economy.
I think of my friends in the UK.
Many Americans feel their country is crumbling too. There have been demonstrations in the US against the current government for 2 years now. They are known as the tea party. They protest for economic improvement like people are demanding in your country.
The official storyline in the US is that the tea party protesters are: “racist, uncompromising, potentially violent, and stupid”. The movement is mentioned regularly in the US news, and on comedy shows, and it is almost exclusively in a negative light. (Here is one example.) This is in spite of the fact that they have not flipped over even one car yet. This is done mostly to slow the growth of the movement, and it has had great success. Almost everyone who has not seen or been to a tea party rally are fully convinced they are diabolical.
The good news is that the solution to economic stagnation in our countries is simple: we need to unwind the bankrupt welfare state.
Maybe you could create a tea party in the UK? We’ll combine efforts! We can send you Sarah Palin and you send us Daniel Hannan.
Another huge problem is single motherhood. Incidence of this in the UK is the highest in Europe. Single-motherhood is a consequence of the welfare state. The social and economic are thoroughly mixed!
Open letter to an Apache mailing list regarding the OpenOffice incubation proposal.
Having written code in Office, why this fork is bad is intuitively obvious to me. I wish everyone had the same instincts. In the days since I first saw the announcements, I’ve been reading and learning a number of additional facts that helped me understand all the reasons why the fork is a bad idea in this situation.
Here I will summarize the “no” vote reasons. While reading, I gained respect for Apache the foundation and OO the brand. There is love people attach to these names. Apache could offer poop on a stick and it would have downloads and people curious about how to contribute back.
Many of us want all of these good ideas and energies to be channeled. The LibreOffice team is not a raging success yet and they’ve just climbed some big hills. Yet no one inside is complaining of a reason to fork.
Given all I have read, this is my (unfinished) list for the arguments against:
——-
- This is mostly a code dump, not the set of 50(?) full-time engineers who have created / been maintaining this code.
- This technology is massive. It is about the same size as the Linux kernel (10 million lines). This is a world that thousands could get lost in. Because the codebase is so large, it makes the cost of the fork that much greater.
- IBM’s priority will be on Symphony and Notes (which build on top) more than the core code.
- Much of the expertise is at Sun / Oracle, and IBM is not bringing any of them over.
- IBM’s biggest investment here is “open source” evangelists. They believe the community will build everything for them.
- This is mostly a cabal of IBM / Sun evangelists trying to confuse Apache into letting them fork a community.
- The Apache foundation has a lot of experience, but none with this technology. Therefore, their help will be limited. It is like asking a hospital to fix your car.
- The code dump is missing a lot (filters, images, translations, etc.)
- OpenOffice is now primarily a brand to be preserved.
- This brand is in jeopardy now.
- There is nothing to incubate. LibreOffice has just built everything you need.
- The Apache License (AL2) is not within the spirit of the tradition of this codebase.
- There is no company waiting to build some amazing technology like Watson in LibreOffice but hold back because of the license. This is just a fantasy IBM can promise and a rhetorical trick because anything about the future is impossible to disprove.
- A proprietary Watson could be built in LO today. LGPL is the ideal license for this codebase. It is already a compromise between GPL and Apache. LibreOffice should not compromise anymore.
- This proposal is considered to have a practical license agreement, but grabbing code changes from LibreOffice is said to be impractical. This is considered a good thing.
- Linus said that making Linux the GPL was the “best thing he ever did”.
- Copyleft is compelling to small LibreOffice contributors. “Do you really want to write Apache 2 (AL2) so that IBM can sell it?”
- AL2 will make ongoing code sharing from LO impossible.
- The move from Java towards Python in LO will add more barriers.
- There is a lot to be done: polish, services, plugins, mobile, etc. We don’t have people to waste.
- LO has just recruited many of the most passionate and experienced volunteers and other unaffiliated third-parties.
- People will show up here because of the Apache, OpenOffice, and IBM name. They will not understand what is going on.
- There were 87 people on the list, but many are just curious.
- The leaders create bad plans, find curious and naive people who “support” it, and use the numbers as proof that their plan has merit.
- The community of contributors to this podling is artificially inflated with “advocates” and not many people with expertise in the OO codebase.
- Microsoft would root for Apache to fork the LibreOffice community.
- LibreOffice is a young community, so some are easily confused and frightened. Many barely know this name “LibreOffice”. Meanwhile LO needs and would love to have more people.
- The OO brand was given up by Oracle primarily because of the success of LibreOffice.
- The OpenOffice brand would be very valuable to LO today.
- These “open source” evangelists from IBM, Sun, etc. should hand over the OpenOffice trademark before they hurt someone. LibreOffice can maximize the value and carry it on best right now.
- They need all kinds of help. They are not turning down one contribution, and no one inside is threatening to fork because of problems.
- The hardware / bandwidth costs are not very expensive. It is the human costs.
- It is not just a question of if you fail, but what is the damage in that failed experiment.
- There is also the opportunity cost of doing something better than failure.
- If this podling fails, it could hurt the value of the OpenOffice brand, LibreOffice, waste resources (these emails are just the start), hurt Apache’s reputation, etc.
- Forks are one of the biggest reasons why free software has struggled in places. For an example in another industry, look at Blu-Ray vs. HD-DVD.
- People at IBM responsible for Notes / Symphony may get bad reviews for building on top of a dying fork and when internal customers complain the product isn’t as good as what comes with Linux. These “open source” evangelists are supposed to have their finger on the pulse of the community, not their finger in the face of the community. I stole that from someone
- No major revisions have been proposed.
- A “no” vote on current idea is fail-fast and the potential for a better plan.
- LO see this as a danger. They received more cash donations since this announcement.
- It will only be a trickle of volunteers. If more show up, LibreOffice can recruit in bulk.
- With the interested people already occupied, this will never catch up with LibreOffice.
- Wise people I have consulted with in LibreOffice believe this will fail.
- Some are not even worried anymore, but I am less confident.
- Some believe the Apache foundation is being used to legitimize a premature idea.
- I believe the result will be the same no matter the vote unless the plan is changed. This plan is like an animal chewing off its own paw even though it wasn’t stuck in a trap.
- Once you have written failure into your plans, like to build a house out of sand instead of concrete, these “friendly” meetings to resolve differences cannot achieve much.
———
I am an un-affiliated observer rooting for Linux on the desktop and Python over Java. I have spent years surveying and writing about Linux so I’ve come to respect the Apache server very much. Any rude bits in my mails were directed at IBM I believe they should know better than to attempt to fork a community. I think the Apache foundation has been caught in the cross-fire of the language and license battles. I feel sorry for these political battles. There are also actual proprietary competitors to fight as well! Isn’t that the most important?
Even if this is born, and fails, the community will pick up the pieces. It has many times before. The LO opinion of the plan is close to unanimous and strongly-felt. My feelings are more mixed.
Perhaps this can help serve as impetus for the vote. Many are curious to its result and are anxiously awaiting.
Hope this work is helpful.
Warm regards,
-Keith
Update: June 13: The podling was created. Oh well, Rome wasn’t built in a day. It is tragic that they’ve designed failure into their plans. Not only that, the opportunity cost of combining resources immediately. Reading their plan, some sentences are reasonable, and some stand out like daggers.
A better plan would have helped instead of hurt. These emails were a huge waste of time, and just the start. Many “open source” evangelists are actually hindrances. Some think forks are okay because they’ve happened before. This is like advocating for slavery because slavery has happened before.
The worst part is that the supporters of this plan made no changes to address the major flaws found by the community since the proposal was first announced. It is rude not to retract and make a new plan when relevant people have big objections to your idea. If you want to get code into the Linux kernel, you need to fix all the problems people find. Plans can be refined and improved, but this did not happen. The cost of the flaws only gets more expensive if this goes forward. Therefore, it was cheapest to vote this down now, so we can make a new plan. The new plan would involve something like making LibreOffice and its license the mainline, and merging TDF and Apache. Who knows where it goes from here.
Post is here.
Start:
I wrote two responses, which eventually allowed me to boil it down to
one sentence: You can do anything you want in addition to perfection
(i.e. zero bugs).
Long answer:
I was baptized into the zero bugs religion about 20 years ago. This
was before the web and time-based releases, but these only add
complications, and are yet compatible with zero bugs all the time. As
a beginner, you can cheat such as zero bugs older than 6 months / 2
releases, or give yourself 1,000. Over time, you can pick new goals
and improve your numbers. Reaching enlightenment may require multiple
steps. You don't have to meet your goals, but you will become better
for trying.
Original link here.
Copied here:
Many interesting ideas on version numbering schemes. I like 2.11.X because it maps to years easily in people’s mind, but I look forward to seeing what is chosen. You guys break many of the rules for software development, so why not going backwards in version numbers
While you are talking about arbitrary numbers and new goals, I want to offer that you could consider a push towards zero bugs. In general, as long as your reliability monotonically increases (no regressions) that is an acceptable minimum approach because it means that you will never have a customer go from being happy to unhappy.
However, it is common in companies to make an effort to get towards zero bugs. Zero bugs is impossible, and that is a philosophical discussion. If you look through your current list of bugs, nearly every one looks scary to me and important to someone. You currently have 2,800 active bugs (http://bit.ly/LinuxBugs) The last time I looked, I found the median age was 10 months. In general, bugs should be fixed in the next release and so therefore 3 months.
Zero bug bounces is hard for the others because they don’t have sufficient resources. However, I believe you easily do. I can’t say that anything magical technically will happen if you work on your bugs faster, but I can say that people I respect as much as you taught me this. My salary was based on my ability to promptly respond to my bugs, and zero was everyone’s goal. Hitting zero, even for a minute, could be a newsworthy event, as another way Linux is better than the others. It also shows leadership to user mode. I sometimes get the feeling that many in the FOSS community look at bugs as something they could work on when they get bored of adding new features, instead of: “Holy poop, there is someone unhappy out there.”
Warm regards,
-Keith
http://keithcu.com/
I have decided to restart OpenRacing: (Note, we may rename it to PyTorcs or pySpeed-Dreams[1]) Note, I’ve decided to call it PyTorcs for now. See you in Github.
PyTorcs is based on the Torcs codebase, which is widely considered the best FOSS racing game and which already has autonomous cars. However, the codebase contains a lot of cruft.
So, it has been methodically re-written into C# Python, ported to leverage the graphics engine Ogre, the physics engine ODE, OIS, standard widget APIs, OpenAL, and extended with a more general track model.
Then, there will be a clean and 6x smaller codebase, with the heritage of Torcs, for simulating autonomous vehicles which can handle the complexities of urban scenarios, and can eventually navigate via the use of a vision recognition engine and simulated sensors like radar and GPS.
There is nothing to announce yet but a plan. Here it is so far:
Python Port
- Do mechanical port of C# code (http://codeconverter.sharpdevelop.net/SnippetConverter.aspx)
- Port the Swig wrapping tools to generate Python
- Debug
Steps after which can go independently:
- Extra data attached to map model
- Define map APIs for the robots to use. ([[1]], [[2]],[[3]]) In the short term, it seems will need three spatial indices inside the map, physics, and graphics engines.
- Get basic autocar driving (waypoints, lanes)
- Grab more features from [Simplix]
- Networking
- Windows & Mac port (remaining code is very portable)
- Make map more pleasant to look at.
- Simulate Lidar (Can we put the source of light behind the camera?)
- Faster than realtime simulation runs (turn off graphics, optimize physics)
- Better weather
- Port over a new and prettier car model from Speed-Dreams and discuss the current set of difficulties in using their data with Ogre
- Find / modify a big urban map with highway exit
- OpenStreetMap augmented-reality visual annotations
- OpenStreetMap auto-generate mesh to drive through
- Smart objects like street lights
- Assisted-driving features (prevent crashes)
- Port over automatic transmission, wheels, etc. from Torcs / Speed-Dreams
- Plug into vision engine
- Define, record, and replay simulations
- Joystick, better keyboard
- Port simuv3 to Python (low priority, isolated task)
- Etc.
If you are interested in working in a Python driving simulator, please contact me or put in some information below. There are plenty of big and little tasks. A handful of people, let alone 10, could accomplish a lot, chipping away in any of those areas. I’ll make another post here when the Github repository is ready, C# is gone (Github here) Python is running.
[1] Just kidding about the last one. I am looking for a good readable simple name, but I never got to choose names of the codebases I worked on so I’m not practiced at it.
This is a rant I posted to the Ableton forum.
Open letter to Ableton;
I was very annoyed about Ableton / Linux support, so I decided to come here and complain and I found a thread — of course!
If an application supports Windows and Mac, supporting Linux is not much work. Somehow, there are very many products that work on all 3 platforms. If you only supported Windows, you would be in much worse shape. I’ll bet a sandwich Ableton doesn’t have even have one person working on Linux. 3-5 could have a solid native port in a few months.
The Linux audio stack is getting mature now. What is required now is a realization by you that your customers want Linux support. Note, the WINE support for Ableton Live is getting solid today, but it does have problems. On the latest Ubuntu, it installs and runs, which is a big milestone, but it has some perf glitches (some things are very slow), and the audio doesn’t work. With Ableton supporting Linux directly, or via Wine, ideally both, these problems could easily and quickly get fixed.
A free / GPL Ableton would be very nice, but the proprietary version of Ableton on Linux enables users to run a free OS, which is even better. Not supporting Linux is damaging to the freedom of Ableton’s customers. Microsoft continues to win because of the lack of vision or laziness of others.
I don’t recommend rioting in the streets, but I do encourage customers to loudly remind every software vendor that the freedom to choose your own OS is very important, and companies should respect their customers’ hardware and software preferences.
You might think it isn’t worth it to build a Linux version today, but how can you know the demand of a product you don’t have? Linux marketshare is growing every year and studies show that worldwide usage is comparable to the Macintosh today. It is true that not much music-making is done on Linux now, but that is partially your fault! Are you waiting for Linux to be dominant in music-making before you enter the market? Any businessman will tell you that is exactly backwards.
People may not use a product if it doesn’t run on all platforms: PDF, Flash, Firefox, Wikipedia, etc., etc. are popular because they work on all platforms. Not having a Linux version puts the entire company at risk.
I know you are busy, but that you can afford it. It is not a matter of development being at capacity (as if people ever sit around), it is a matter of prioritizing. When you say you don’t have the resources, you just are saying it doesn’t seem important yet. You actually could make a major shift in priorities quickly if you wanted to. Requirements often show up up mid-way through every development cycle that need to be incorporated, and it gets done. Ableton says that they aren’t going to support Linux because they can’t be “all things to all people”. That is equating one feature with all features.
You either embrace the future or your competitors do it for you. I don’t care who builds it, but music-making software is one of the top challenges for the Linux desktop. Many people run 1 or 2 proprietary apps on Linux. Several of Ableton’s employees are long-time users of Debian Linux. It is sad that Linux has so many users who are not supporters. Supporting Linux can mean many things, I just ask you to start with creating a version of Ableton that runs on at least Debian. If you feel very busy, I can recommend moving away from C++ towards 99% Python. That will help help speed the Linux port and every other feature.
Regards,
-Keith
http://keithcu.com/SoftwareWars/
P.S. Here is a quote:
Sometimes the real hurdle to renewal is not a lack of options, but a lack of flexibility in resource allocation. All too often, legacy projects get richly funded year after year while new initiatives go begging. This, more than anything, is why companies regularly forfeit the future — they over invest in “what is” at the expense of “what could be.”
New projects are deemed “untested”, “risky”, or a “diversion of resources.” Thus while senior execs may happily fund a billion-dollar acquisition, someone a few levels down who attempts to “borrow” a half-dozen talented individuals for a new project, or carve a few thousand dollars out of a legacy budget, is likely to find the task on par with a dental extraction.
The resource allocation model is typically biased against new ideas, since it demands a level of certainty about volumes, costs, timelines, and profits that simply can’t be satisfied when an ideal is truly novel. While it’s easy to predict the returns on a project that is a linear extension of an existing business, the payback on an unconventional idea will be harder to calculate.
Managers running established businesses seldom have to defend the strategic risk they take when they pour good money into a slowly decaying business model, or overfund an activity that is already producing diminishing returns.
How do you accelerate the redeployment of resources from legacy programs to future-focused initiatives?
—Gary Hamel, The Future of Management
Feb-15-2011
Science doesn’t always proceed at the speed of thought. It often proceeds at sociological or even demographic speed. — John Tooby
Open Letter to the Linux Kernel Mailing List (LKML);
If we were already talking to our computers, etc. as we should be, I wouldn’t feel a need to write this to you. Given current rates of adoption, Linux still seems a generation away from being the priceless piece of free software useful to every child and PhD. This army your kernel enables has millions of people, but they often lose to smaller proprietary armies, because they are working inefficiently. My mail one year ago (http://keithcu.com/wordpress/?p=272) listed the biggest workitems, but I realize now I should have focused on one. In a sentence, I have discovered that we need garbage-collected (GC) lingua franca(s). (http://www.merriam-webster.com/dictionary/lingua%20franca)
Every Linux success builds momentum, but the desktop serves as a powerful daily reminder of the scientific tradition. Many software PhDs publish papers but not source, like Microsoft. I attended a human genomics conference and found that the biotech world is filled with proprietary software. IBM’s Jeopardy-playing Watson is proprietary, like Deep Blue was. This topic is not discussed in any of the news articles, as if the license does not matter. I find widespread fear of having ideas stolen in the software industry, and proprietary licenses encourage this. We need to get these paranoid programmers, hunched in the shadows, scribbled secrets clutched in their fists, working together, for any of them to succeed. Windows is not the biggest problem, it is the proprietary licensing model that has infected computing, and science. Desktop world domination is not necessary, but it is sufficient to get robotic chaffeurs and butlers.
There is, unsurprisingly, a consensus among kernel programmers that usermode is “a mess” today, which suggests there is a flaw in the Linux desktop programming paradigm. Consider the vast cosmic expanse of XML libraries in a Linux distribution. Like computer vision (http://www.cs.cmu.edu/~cil/v-source.html), there are not yet clear places for knowledge to accumulate. It is a shame that the kernel is so far ahead of most of the rest of user mode.
The most popular free computer vision codebase is OpenCV, but it is time-consuming to integrate because it defines an entire world in C++ down to the matrix class. Because C/C++ didn’t define a matrix, nor provide code, countless groups have created their own. It is easier to build your own computer vision library using standard classes that do math, I/O, and graphics, than to integrate OpenCV. Getting productive in that codebase is months of work and people want to see results before then. Building it is a chore, and they have lost users because of that. Progress in the OpenCV core is very slow because the barriers to entry are high. OpenCV has some machine learning code, but they would be better delegating that out to others. They are now doing CUDA optimizations they could get from elsewhere. They also have 3 Python wrappers and several other wrappers as well; many groups spend more time working on wrappers than the underlying code. Using wrappers is fine if you only want to call the software, but if you want to improve the underlying code, then the programming environment instantly becomes radically different and more complicated.
There is a team working on Strong AI called OpenCog, a C++ codebase created in 2001. They are evolving slowly as they do not have a constant stream of demos. They don’t consider their codebase is a small amount of world-changing ideas buried in engineering baggage like STL. Their GC language for small pieces is Scheme, an unpopular GC language in the FOSS community. Some in their group recommend Erlang. The OpenCog team looks at their core of C++, and over to OpenCV’s core of C++, and concludes the situation is fine. One of the biggest features of the ROS (Robot OS), according to its documentation, is a re-implementation of RPC in C++, not what robotics was missing. I’ve emailed various groups and all know of GC, but they are afraid of any decrease in performance, and they do not think they will ever save time. The transition from brooms to vacuum cleaners was disruptive, but we managed.
C/C++ makes it harder to share code amongst disparate scientists than a GC language. It doesn’t matter if there are lots of XML parsers or RSS readers, but it does matter if we don’t have an official computer vision codebase. This is not against any codebase or language, only for free software lingua franca(s) in certain places to enable faster knowledge accumulation. Even language researchers can improve and create variants of a common language, and tools can output it from other domains like math. Agreeing on a standard still gives us an uncountably infinite number of things to disagree over.
Because the kernel is written in C, you’ve strongly influenced the rest of community. C is fully acceptable for a mature kernel like Linux, but many concepts aren’t so clear in user mode. What is the UI of OpenOffice when speech input is the primary means of control? Many scientists don’t understand the difference between the stack and the heap. Software isn’t buildable if those with the necessary expertise can’t use the tools they are given.
C is a flawed language for user mode because it is missing GC, invented a decade earlier, and C++ added as much as it took away as each feature came with an added cost of complexity. C++ compilers converting to C was a good idea, but being a superset was not. C/C++ never died in user mode because there are now so many GC replacements, it created a situation paralyzing many to inaction, as there seems no clear place to go. Microsoft doesn’t have this confusion as their language, as of 2001, is C#. Microsoft is steadily moving to C#, but it is 10x easier to port a codebase like MySQL than SQL Server, which has an operating system inside. C# is taking over at the edges first, where innovation happens anyway. There is a competitive aspect to this.
Lots of free software technologies have multiple C/C++ implementations, because it is often easier to re-write than share, and an implementation in each GC language. We all might not agree on the solution, so let’s start by agreeing on the problem. A good example for GC is how a Mac port can go from weeks to hours. GC also prevents code from being able to use memory after freeing, free twice, etc. and therefore that user code is less likely to corrupt your memory hardware. If everyone in user mode were still writing in assembly language, you would obviously be concerned. If Git had been built in 98% Python and 2% C, it would have become easier to use faster, found ways to speed up Python, and set a good example. It doesn’t matter now, but it was an opportunity in 2005.
You can “leak” memory in GC, but that just means that you are still holding a reference. GC requires the system to have a fuller understanding of the code, which enables features like reflection. It is helpful to consider that GC is a step-up for programming like C was to assembly language. In Lisp, first GC language, the binary was the source code — Lisp is free by default. The Baby Boomer generation didn’t bring the tradition of science to computers, and the biggest legacy of this generation is if we remember it. Boomers gave us proprietary software, C, C++, Java, and the bankrupt welfare state. Lisp and GC were created / discovered by John McCarthy, a mathematician of the WW II greatest generation. He wrote that computers of 1974 were fast enough to do Strong AI. There were plenty of people working on it back then, but not in a group big enough to achieve critical mass. If they had, we’d know their names. If our scientists had been working together in free software and Lisp in 1959, the technology we would have developed by today would seem magical to us. The good news is that we have more scientists than we need.
There are a number of good languages, and it doesn’t matter too much what one is chosen, but it seems the Python family (Cython / PyPy) require the least amount of work to get what we need as it has the most extensive libraries: http://scipy.org/Topical_Software. I don’t argue the Python language and implementation is perfect, only good enough, like how the shape of the letters of the English language are good enough. Choosing and agreeing on a lingua franca will increase the results for the same amount of effort. No one has to understand the big picture, they just have to do their work in a place where knowledge can easily accumulate. A GC lingua franca isn’t a silver bullet, but it is the bottom piece of a solid science foundation and a powerful form of social engineering.
The most important thing is to get lingua franca(s) in key fields like computer vision and Strong AI. However, we should also consider a lingua franca for the Linux desktop. This will help, but not solve, the situation of the mass of Linux apps feeling dis-integrated. The Linux desktop is a lot harder because code here is 100x bigger than computer vision, and there is a lot of C/C++ in FOSS user mode today. In fact it seems hopeless to me, and I’m an optimist. It doesn’t matter; every team can move at a different pace. Many groups might not be able to finish a port for 5 years, but agreeing on a goal is more than half of the battle. The little groups can adopt it most quickly.
There are a lot of lurkers around codebases who want to contribute but don’t want to spend months getting up to speed on countless tedious things like learning a new error handling scheme. They would be happy to jump into a port as a way to get into a codebase. Unfortunately, many groups don’t encourage these efforts as they feel so busy. Many think today’s hardware is too slow, and that running any slower would doom the effort; they do not appreciate the steady doublings and forget that algorithm performance matters most. A GC system may add a one-time cost of 5-20%, but it has the potential to be faster, and it gives people more time to work on performance. There are also real-time, incremental, and NUMA-aware collectors. The ultimate in performance is taking advantage of parallelism in specialized hardware like GPUs, and a GC language can handle that because it supports arbitrary bitfields.
Science moves at demographic speed when knowledge is not being reused among the existing scientists. A lingua franca makes more sense as more adopt it. That is why I send this message to the main address of the free software mothership. The kernel provides code and leadership, you have influence and the responsibility to lead the rest, who are like wandering ants. If I were Linus, I would threaten to quit Linux and get people going on AI There are many things you could do. I mostly want to bring this to your attention. Thank you for reading this.
I am posting a copy of this open letter on my blog as well (http://keithcu.com/wordpress/?p=1691). Reading the LKML for more than one week could be classified as torture under the Geneva conventions.
In liberty,
-Keith
Here is some light New Year’s Eve reading for you. Original link (Translation per Shardul Pandey)
——
United States should elect Keith Curtis as President
I don’t know Keith Curtis. I have read his book, After the Software Wars and after decades I have encountered an intelligent American like him. He is so very intelligent that Republicans should choose him as their next President. I have discussed his wisdom to my political friends. We are giving him complete importance.
He is a friend of Shardul and supports his project in the public interest. He had discovered a few drawbacks in our work and we will improve it further. The United States must focus on Keith’s issues, as it is good both for US and world. I assure that India is focusing. We were thinking like Keith Curtis for a long time. We had thousands of hours in serious political discussion and one day when I was leaving the room, Mr. L.K. Advani spontaneously asked me that either someone from Microsoft or Google has written something like that somewhere? I thought that the answer was evidently negative. But a few days later I found a book in the British Library and once again my thoughts got transformed for America.
– Indian political veteran Rajendra Kumar who has been a close confidante of many prime ministers in India.
The article about Google working on driverless cars has been making the rounds. It is interesting that Google can generate news about a side project of a side project, and yet the car companies are unable to demonstrate anything newsworthy in this area.
I’ve emailed Sebastian Thrun and a few others before and so I am sending them an open letter in response to his latest news article.
——
Hi all;
I saw your recent article in the NYT that got sent all over the world and I thought to write you again.
I have three suggestions for you:
1. License your software as GPL v3. I came from Microsoft so I understand that you call your code “secret sauce”. I wrote a book about this topic that you might find an interesting summary of the issues; the thing to remember is it will take more than just your little team to finish this. The reason millions around the world know the name Linus Torvalds is because his code was licensed as GPL v2 and therefore lots of other people joined in — and were forced to contribute back (unlike the lax BSD license.)
2. Move to Python + assembly language. I spend two chapters on this topic, so I will say here that Python is a productive, reliable, rich and free language. Any language with garbage collection is good, but Python has the richest set of libraries with things like SciPy (http://scipy.org/Topical_Software)
3. Use a free simulator like Rigs of Rods. The only way you can ever build confidence in such a system is to create crazy scenarios and test the code end to end. It is only when you’ve got every possible scenario tested that you can say to the world that this is ready. Your real-world test runs can be used to make your simulations better. You all might not be interested in this aspect right now, but there will need to be some people who are! I could imagine a big testing team working exclusively on simulators.
I’m happy to discuss any of this further. I appreciate your time.
Kind regards,
-Keith
If the WW II generation was The Greatest Generation, the baby boomers were The Worst. My former boss Bill Gates is a baby boomer. While he has the potential to do a lot for the world by giving away his money to other people, after studying Wikipedia and Linux, I see that the proprietary development model he adopted has greatly stifled the progress of technology his generation should have provided to us. I start a book with the statement that we should already have cars that drive us around as we have had video cameras and powerful computers for decades. The reason we don’t have robot-driven cars is that proprietary software became the dominant model:
Free versus proprietary software is similar to the divide between science and alchemy. Before science, there was alchemy, where people guarded their ideas because they wanted to corner the market on the means to convert lead into gold. The downside of this “strategy” is that everyone would have to learn for themselves that drinking mercury is a bad idea! The end of the Dark Ages arrived when man started to share advancements in math and science for others to use and improve upon. Computers are an advancement comparable to the invention of movable type and while they have already changed many aspects of our lives, we still live in the dark ages of technology.
We can blame the baby boomers for proprietary software. (We can also blame them for C++ and Java, and I write two chapters detailing why they have been a total disaster for the industry. I recommend everyone use Python today.) We can also blame boomers for outlawing nuclear power, never drilling in ANWR despite decades of discussion, never fixing Social Security, destroying the K-12 education system, leading us to our bankrupt welfare state, and numerous of the other long-term problems that have existed in this country for decades, that they did not fix, and the new ones they created. Linus Torvalds is a Generation X-er, having been born in 1969. It is this generation that is coming into its own now that will invent the future, as we incorporate more free software, cooperation, and free markets into society. The boomer generation got the collectivism part, but they failed on the freedom aspect.
My book describes why free software is critical to faster technological development, and it ends with some pages on why our generation can build a space elevator in less than 7 years. I believe that in addition to driverless cars, understanding DNA, getting going on nanotechnology, and terraforming Mars, are also in reach. Wikipedia surpassed Encyclopedia Britannica in 2.5 years which is strong evidence that the problems in our world are as much social as technical. Let’s step up, work together, and make it happen!
This is some article about how Beijing is spending $740 on green energy.
This is a big headfake, or stupid on China’s fault. E=mc^2 says that energy is basically infinite. So why shouldn’t it cost almost nothing by now? We need cheap energy to have a manufacturing sector. We can’t be a country full of lawyers.
We’ve been talking about battery-powered cars for 100 years. The problem has always been the batteries. Liquid fuel is a dense form of energy. We should do biofuels, not expensive, toxic batteries.
Windmills are another stupid idea. It would take windmills over the entirety of the UK to provide for all of its energy needs. The wind power is up high. If we are going to do wind, we should be doing kites. When the government gets involved, it screws it all up. All these programs do is distract the free markets and make them invest in the wrong technologies. The government is terrible about choosing winners. That is why R&D tax credits are a good idea.
As for electric, there is no point building bad batteries today and hoping that teaches you how to build better ones. No, the problem is the chemistry and that should be solved first. The Chevy Volt will not succeed as it is using conventional batteries. The government is just investing tons of money in the current crappy stuff.
Boehner was on the defensive for his nuking an ant statement.
He should have said something like this:
I’m sorry for mispeaking. My teleprompter was broken that day, and so I was speaking with my mind and heart as my guide. And I ended up using a metaphor that only half-conveyed the idea I was trying express. Which is that this reform is a spreader bar on our financial sector when our economy is already bound and shackled and gagged. As esteemed Richard Posner has written,
“The most sensible legislative response to the financial collapse of September 2008 would have been to do nothing until the causes of the collapse were fully understood.
There is no urgency about legislating financial regulatory reform. The existing regulatory agencies have virtually total authority over the financial industry. And because they were asleep at the switch when disaster struck, they are now hyper- alert to prevent a repetition of it. Indeed, bank examiners have become so fearful of condoning risky practices that they are making it difficult for banks to lend to small businesses and consumers and thus are retarding the economic recovery.”
So I didn’t mean to suggest that the ongoing financial crises he is presiding over are small. Far from it. He never does talk to me, but if he had called, I would have been happy to explain in any amount of detail my concerns.
So instead, he loaded onto his teleprompter a political attack that is based on a mischaracterization of my position. I just want to know at one point Obama realized this mistake. Or whether he knew it was a mistake when he said it.
Of course, this demonstrates that Obama isn’t the post-partisan unifier he claimed to be in his campaign commercials. Who is the real Obama? The one who claims to believe in the free market, or the one who pretends his stimulus bill is a success?
There are 12 Cuban programmers each translating a chunk of my book into Spanish. Here is my latest mail to them:
Hi;
An idea for an interesting project occurred to me, and you are the perfect candidate country to build it!
I have come to understand that rolling releases are what Linux should become. I have installed Ubuntu on friends computers, and years later, they are running the exact same software. Free software improves every day, and it is a shame that people are not always running the latest.
Therefore, you should put rolling-releases software on every desktop machine in Cuba. You should put something on that automatically upgrades to the latest version without any intervention by a local user.
Now, this can create a problem. If you put out a fix and every machine crashes and won’t boot, you are screwed.
Before I get to the solution, there are many ways to mitigate this problem. For example, you can mirror Debian-Testing, but create a different schedule for how the software deploys. Some machines will grab the latest builds. Some machines will grab Debian Testing + another 20 days. If you can categorize machines, and setup different dates for how long things sit in -Testing, you can catch problems early.
There is no software that does this yet and I you guys should build it. It allows people to connect up to Debian-Testing, but not be on the front lines. I’m not sure how to implement it. But it is a big enough project that it would require a bunch of you.
You also need to work on what to do when things go bad. For example, if a machine crashes, it should reboot into a special state to see if there are any new packages. It could be that a fix is waiting for it.
Furthermore, if the machine is still hosed, then you could have it decide to do a reinstall. You should setup every machine to have a separate user and home partition. This gives you the possibility to wipe machines.
Every machine that is running this Linux is connected to a very powerful force. Once it is installed, it can all be run remotely. Year after year, the software in every school and such will automatically get better. And hey, even if it breaks, well that’s just a reason to hire some Debian experts in every school.
And it would be useful to many other people. It could be one of the biggest reasons to move people back to Debian. In fact, you could end up unifying a lot of the other forks by Debian implements a better Debian-Testing. The early people who joined Ubuntu came from Debian and used it mostly because it shipped more frequently.
So you need to figure out how to implement it, and how to handle bad situations. The key is to have multiple types of correction mechanisms. One thing you can do is look at past Debian-Testing breaks and see if your solution handles them.
What do you think?
-Keith
Here is a tutorial (not the easiest thing to read) of her views.
http://cnsnews.com/news/article/65720
Does anything in it stand out as possibly scary?
- Focusing on government motives. She argues that campaign finance laws as bad because she divines that their motives were bad. They were just trying to stem the “excess” flow of money into politics! The problem was the unintended consequences. But as long as she thinks she finds people with good motives, even total idiots, what they are doing is fine.
- Inciting others to commit harm is not allowed, but legislation banning flag-burning is legal because she is sure it will never incite others to commit harm.
- If there is an overabundance of an idea, “action disfavoring that idea [would] un-skew public discourse.” As there are too many right wingers on Talk Radio, so we should “un-skew” the number of them.
- Restrictions should focus on whether the person’s ideas are causing “public harm”. Harm is surely very narrowly defined.
- Government should treat all people creating public harm equally. If we are going to round up one-right winger, we should be fair and round up them all.
Using this sample of her writings, are you convinced yet that she is worthy of being on the Supreme Court?
Here we are 5+ years in, and these basic problems still exist.
The problem is Mark doesn’t see the point of doing work in Debian, because if he did, then why should he have created Ubuntu? Mark thinks contributing to Debian makes as much sense as contributing to Red Hat: nice in some ideal world, but not worth investing in. Do you think he talked to Dell or the Amazon cloud folks about supporting Ubuntu and Debian?
Either:
Work doesn’t go into Debian, which is bad for Debian,
Or work is done in Debian in which case Ubuntu has little reason to exist.
Update: It looks like the Ubuntu people are going to put their stuff into Debian. That is good to hear. So we’re going from the first scenario to the second — which is an improvement.
Hello,
Ubuntu is developing more and more software of its own and often Debian
reintegrates the software but later on. I wonder why you are not
integrating new software immediately in Debian:
- you would benefit from the feedback of the debian community sooner
and avoid some packaging churn later on [1]
- even when you have constraint of integration with other software in
debian and that you’re blocked, you can have a ubuntu-specific packaging
thanks to the dpkg-vendor framework and still share the source package
between both distributions
- you would have some explicit responsibility in maintaining the software
that you create
[1] I’m referring to http://bugs.debian.org/571929 for desktopcouch, the
initial packaging is sub-optimal and the upstream developers (all
@canonical.com) have been asked for their feedback and insight on the
reasons of some dependencies but they have never responded… this
sucks hard. It would be nice if this small fingerpointing would lead
to someone reacting…
Cheers,
–
Raphaël Hertzog
I just wrote this to a friend in facebook and thought I’d post it here. He wanted to know what I didn’t like most about the healthcare bill:
It is so hard to say what I don’t like most because it is a big bill but also because I am upset about the process. This bill was rammed through the Congress. The only bipartisanship was with the opposition. A Republican in the liberal state of Mass won Ted Kennedy’s seat and yet still they kept pushing this through in spite of this and other the messages from the electorate. This bill has been a part of the debate for a year, and yet it has been firmly rejected by the American people.
It was sold on a pile of lies.
Barack Obama said as a Senator that the 60-vote requirement is the biggest reason for having the two separate legislative bodies. The Senate is where legislation like a hot tea, goes to cool down. It was always understand that bad government laws can very easily make any situation worse. Yet as prez, he doesn’t care the Senate anymore and has endorsed the 51-vote reconciliation process. “Nobody cares about the process”, he tell us. “people without insurance are dying every day.” As if he’s going to stop death. He’s supposed to be a constitutional law processor.
We are told this bill will decrease the deficit — as if a program like this can ever do that. This was negotiated between special interests in secret, as if they have all the answers. As the process went on, the legislation got worse. If a worker showed up to your house, and every day he worked he made it worse, you’d surely fire him.
We are told we can keep our doctor and insurance, but the bill says that insurance now needs to be approved by DC. My catastrophic insurance will be outlawed — they want everyone on full-service plans. This bill also doesn’t fix the tax equalization issue that your employer gets a tax deduction for healthcare, but you don’t. This has caused people to tie healthcare to employment, which is very bad.
So it is as much what is not in the bill as what is in it.
But anyway, as to what it contains. The US already has programs to help the poor and the elderly. This bill doesn’t even attempt to shore them up or reform or improve them. It actually makes those programs in even more fiscal trouble. Every American is putting money into these programs. We can talk about giving people things, but if the money isn’t there, it is just a promise. These entitlements always end up costing more than the estimates. By now they should know better!
The biggest issue is that it doesn’t push towards healthcare as more of a free market. Why can everyone afford cellphones today? Not because the government stepped in. The problem with healthcare is the “third-party payee” system. If you want going to the doctor to be like visting Apple, removing all the regulations. That, plus tort reform, buying insurance across state lines, and a few other things would totally change things.
Price and cost controls and regulations stifle progress in the free market. That is the old economic thinking in that they don’t consider the unintended consequences of their stuff. If the US government got into the software business as a way to alleviate the problem with Microsoft’s anti-competitive behavior, would you think that was a good remedy?
It also took a very long time to make this bill happen because it was so terrible that many Democrats didn’t like it and so they had weeks of back room deals. Obama should have scrapped the bill in January. We could have passed something smaller and better by now. They are arrogant.
The good news is that Obama is the best community organizer the Republicans have ever had. They now know what they stand for — basically the opposite of what Obama is doing. Even McCain understands how tax cuts can decrease unemployment. Obama believes the ends justify the means, and is clueless. This is a terrible combination. He was educated by white liberals in Columbia and Harvard so his skin color plays zero part in my analysis. Karl Marx and John Maynard Keynes were white people, but so were Milton Friedman and Ronald Reagan. I believe that half of America would be a better president — polls show that a majority would not have pushed through this healthcare bill.
William Buckley wrote this in 1963: “I would rather be governed by the first 2,000 names in the Boston telephone directory than by the 2,000 members of the Harvard faculty.” If the Republican party had nominated a constitutional law professor, and a resume that would fit on the back of his driver’s license, he would have gotten hammered.
This is a funny video about the h/c law.
Behind the scenes, the people who created this new law have in mind the idea that in the long-term the best way to save money on healthcare is to push the “scarce” resources towards the young people. It is like them looking at the high price of cellphones 20 years ago and deciding that the best thing the government should do is to step in and redistribute them to the “productive” people.
And in their anger, they lash out at their critics. Sarah Palin gets criticized for months for bringing up death panels, when we all know that every healthcare system has limits today, and this legislation has it.
Every alleged misbehavior by someone angry at DC today gets amplified in the media. Congressmen who wanted to vote “no” on healthcare reform get harassed, but the only stories we ever hear about are of the angry electorate calling up those who voted “yes”. This follows their “tea-partiers are evil” storyline. The establishment doesn’t care about any threats to those who are not following along with their agenda. They want those who don’t follow along to have it difficult.
A wheelchair-stricken black man gets beaten up by big SEIU men, but we only ever hear about the racist tea-baggers — that sliver of Americans that oppose Obama’s policies and constitutes half of the electorate.
Many Democrats cheer Obama’s “achievement”. They have absolutely no idea what they are cheering for.
Version 1.03
Dear Mark,
Thank you for your feedback on my book excerpt about Ubuntu / Debian, and I appreciate your time.
The main concern I have about the current situation is this: when an Ubuntu person does some coding work and posts the diff to a website, it is now a workitem for someone else in Debian to dig into and understand. Of course, having that fix is helpful, but the time to do something is mostly the time to learn how to do it.
If you pick up a violin, it will be clumsy to you unless you know how to play one. It is the same with a patch if you haven’t seen it before. So therefore, whenever someone learns something, they should make sure that they do their work in both Debian and Ubuntu to save someone else time.
Neither you nor anyone else has ever acknowledged this problem or attempted to explain how this won’t happen all the time in the current situation.
With separate trees and buglists and such, people no longer work together simply as a matter of course of the tools they use. As a DD told me: this is social engineering. There are ways to mitigate this, but it is a problem that wouldn’t exist if it weren’t for the social engineering that preceded it.
Forks
In essence, I think forking is like war: it means that the discussions and diplomacy have failed. Any feature can be done in software, it is just a matter of technical and social work. So when you created Ubuntu, you were saying (before trying) that what you wanted to do couldn’t be done in Debian. This is an ironic statement given that you were starting with Debian and 10 of its best developers.
Note that forking is different from branching which happens all the time in DVCS. There are many branches of the Linux kernel out there, but the team is very unified. In the Ubuntu / Debian world you have separate documentation, incompatible packages, separate conference schedules, email lists, bug-tracking systems, and most importantly: separate teams maintaining a number of the same packages.
I know that some here might bring up a few good reasons for a fork:
- Non-free code. That affects only a few packages, and Debian already has mechanisms to handle non-free code. Furthermore, it is possible to create two versions of a Live CD with and without the questionable stuff. So for simplicity here, let’s leave out freeness issues and assume you agree with Debian policies.
- Having 6-month ship cycles. I have spent hours thinking about it and I concluded that this is doable with a separate tree in the Debian infrastructure. I’d rather not get into that here.
- You state that Debian’s greater generality is something you aren’t interested in. You brought up Debian’s larger number of supported architectures as an example. I address that in footnote 8, but I’d add that Ubuntu supports more architectures in every release so it seems even this is getting smaller. Furthermore, writing portable code is not that hard, especially if you write in a garbage collected programming language. In nearly all GC languages, you use the same “binary” on all platforms.
So when I subtract out those difference, I run out of reasons for the separation.
I realize that achieving consensus on things is work, especially in the distributed nature of a volunteer movement. But this is not your responsibility, it is the responsibility of the people doing the work. You say that you don’t think you could have steered Debian in the direction that you want, but every contributor to Debian steers it. You still have to achieve a consensus within Ubuntu, and as it gets bigger, it will also get somewhat harder. And as for whether it is hard to achieve consensus in Debian, I would just point out that there are hundreds of DDs and millions of users, so they’ve managed to make it pretty far.
Money
You assert that mixing money with volunteers would have been a problem for you. Of course, we can’t know. You reference Dunc-tank; this is something I do not address in my book because I felt it was too “inside baseball”. But it seemed like there were a number of problems with it, for example, the secrecy with the way the program was developed.
The failure of Dunc-tank is not proof that you can’t mix paid folks and volunteers. And isn’t that exactly what is happening in the Ubuntu community?! We can also look to the HP example. HP pays programmers to work on Debian, and makes money supporting Debian. So this situation, and others, have not turned into a problem. Likewise, Linus gets paid to work on the Linux kernel and yet this has not caused problems.
People should contribute to free software only when they want to. If you do something because you want to, it doesn’t matter what other motivations other people have to do what ever they are doing. I would love to get paid for taking pictures, and the fact that some do doesn’t make me enjoy taking pictures less.
You can also think about the money like this: someone is working on Debian their few hours a week, and if someone else shows up and starts working 40-50 hours a week, that is great. The more people the better! I presume you’d like to hire as many of the great Debian people as you could. The world needs more full-time programmers working on free software. Many of the other Debian volunteers are in school or using it in their company so they can’t work on Debian full-time.
I realize that mixing volunteer and paid work is a tricky issue. But the solution is not just to assume that humans are flawed and so create new institutions which has the same potential problem! Note that there is also tension in what you are doing today if you hire Debian people to do work that never makes it into Debian.
Fixes
I agree that there are people looking for conspiracy and malice. Part of your challenge comes with the territory of being a suddenly very important part of a movement filled with passionate geeks. I also think that you guys were slow to adopt the GPL / AGPL for the code that you write, for example. The urge to hoard is strong and maybe some in Ubuntu don’t understand yet that it was free software, and Debian, that made this all possible. Free software and all that it implies almost requires a bit of faith.
From my perspective, the toothpaste is out of the tube. I don’t know exactly what could be done, but I also don’t consider all the possibilities because I am far away. I feel that the relationship will be uneasy as long as it continues as it is today.
You say that many people in the Ubuntu community recognize the importance of Debian now. That is good to hear. I didn’t get that sense as much in 2007. At the Debconf you suggested that people should just work in whatever codebase they wanted to, and not necessarily both.
I think that a merge is not the big task it sounds because the entire Debian and Ubuntu would join in. But even today, I think there more things than can be done. Here is a thought experiment. Suppose you decided that Bzr is duplicate code and therefore the Launchpad repositories should be converted to Mercurial. This thought experiment doesn’t get Ubuntu much closer to Debian, but it is the kind of “out of the box” thinking one could consider. Maybe there is a way to fast-track Ubuntu people into the Debian team? What about cancelling MOTU and having them all join the Debian team? How can Debian’s debbugs and Alioth work better with Launchpad? What about having UDS at the next Debconf? What about sharing documentation and wikis? I feel like one could come up with a long list of things big and small that would squeak greater efficiency and improve the situation.
You state that you don’t think Ubuntu can ever be loved by Debian developers. I hope you and your org don’t quit trying. No situation need ever be stagnant and the OS is such a critical piece of the free software movement.
One of Linux’s top challenges
I wrote this book for two reasons: explaining why free software is superior and what things can be done for faster world domination. (You can find a short version of the to-do list here.) It will not be easy to beat Microsoft.
Many, even in our movement, believe that free software will never achieve world domination on the desktop, and so I focused on that. One of the reasons why Unix never was much competition for Windows was because there were separate teams re-implementing each other’s features. This Ubuntu / Debian inefficiency is one of the big things I came up with. I think what you have done has been amazing for Linux, but Ubuntu is buggier and moving more slowly than it could be.
I presented a version of these ideas in Debconf 2007 to you and others, and listened carefully to all the feedback, but I didn’t feel that my main point — about the violin — was understood. So I’ve left it in for now. In fact, I met a number of people at that Debconf who agreed with me and gave me evidence of other inefficiencies. One DD was surprised that I understood the Debian and Ubuntu situation so well even though I am not really a part of either! And my Microsoft friends I’ve talked this over are easily convinced that it is an example of the FOSS community shooting itself in the foot.
This is not an issue I push much anymore, so I consider it mostly as food for thought as a case study on forks, and because if Ubuntu’s developers understand the importance of joining the Debian community, then the inefficiency can be minimized. It is cheaper for one person to do work twice than for two people to do the same thing.
Respectfully,
-Keith
There are a lot of people worried about “open data”, like Matt Asay and Tim O’Reilly.
Of course, they call free software “open source” and they think the lack of open data it is a big deal with companies like Google threatening to control what links we click on and companies like Facebook using free software but not letting you easily migrate your data to Myspace, etc.
But I disagree. The biggest problem with Google is their lack of support for free software. They’ve only started releasing stuff in the last few years, and a lot of it is the plumbing code, not cool the stuff that AI researchers of all kinds would be digging over. Perhaps it is because people have been talking about “open data” instead of free software. Even if you could export data from Google’s search servers, you wouldn’t necessarily be able to make sense of it if you didn’t have the accompanying software.
The free software battle is much more important than the open data battle. It is not really that hard to get your data out of Facebook or out of Amazon EC2. It is your data and you put it in there. And this can anyway be solved by a few standards or even some free import / export code.
The open data movement should be a (small) part of the free software movement.
Get your popcorn!
Microsoft is reviving itself right now. It has the money and the expertise. Windows 7 is a major improvement over Vista, even though it demonstrated the holes in Microsoft’s aging technology stack.
Windows 7′s additional 3 years of debugging, plus a new Windows Phone 7, Silverlight, Windows Azure, IE 9, Bing, Natal, Office 2010, etc. show how Microsoft is shoring up its weaknesses and taking the initiative. That is great as it makes the battle fun to watch.
But the seeds of Microsoft demise have already been sown. Microsoft will lose because it doesn’t understand that it is an “encyclopedia” of free software for every task that you can imagine that is the true power of the computer. Wikipedia is just one thing you can build using free software, and it is the million-man free software army that is working on this problem and gaining critical mass.
Hardware First
The first thing the free software community needs to do is convince every hardware company to fully support Linux: feature-complete GPL drivers in the Linux tree before the device ships. This should be done for all components, not just the ones inside PCs that ship with Linux today.
Hardware companies already know that it is in their self-interest to support free software, but they just aren’t executing. But as governments and companies and users continue to request Linux, things will improve. Note, it doesn’t matter that Windows 7 is much better than Vista because large organizations are now making the switch, and pushing PC vendors to better support Linux over time.
The Long Tail of Software
Perfect hardware support is something that Microsoft already promises, so this by itself will not be what convinces people to switch to Linux — it just makes this possible. What we need is to keep improving the various software programs for every task you can imagine in art and science. We also need to push people to write it in Python rather than C/C++. We have enough programmers today, but too many big and old codebases that need a coat of polish.
This was originally posted to Linus and the rest of the Linux kernel mailing list, and then to Tom’s Hardware.
“The future is open source everything.”
—Linus Torvalds
Dear LKML;
I have written a book that makes the case for Linux world domination. I find it interesting that the idea of Linux on the desktop is responded to by either yawns or derision. I think it depends on whether you see Linux as a powerful operating system built by a million-man army, or one filled with bugs and missing the cool stuff like speech recognition.
The points I wrote should be obvious to you all, but there are some pages on how to have Linux succeed faster I thought I would summarize here. Given this is such a high volume list, I figured it cannot decrease the signal to noise ratio very much! I didn’t see such emails are disallowed by the LKML FAQ.
I’ve been using Linux since mid-2005, and considering how much better things everywhere are now compared to then, it surely is an interesting time to be involved with free software. From no longer having to compile my Intel wireless driver or hack the xorg.conf, to the 3-D desktop, to better Flash and WMV support, to the countless kernel enhancements like OSS -> ALSA and better suspend/resume, things are moving along nicely. But this is a constant battle as there must be 10,000 devices, with new ones arriving constantly, that all need to just work. Being better overall is not sufficient, every barrier needs to be worked on (http://www.joelonsoftware.com/articles/fog0000000052.html).
The Linux kernel:
The lack of iPod & iTunes support on Linux is not a bug solved by the kernel alone, but Step 1 of Linux World Domination is World Installation. Software incompatibilities will be better solved as soon as the hardware incompatibilities become better solved. The only problem you can’t work around is a hardware problem.
If you hit a kernel bug, it is quite possible the rest of the free software stack cannot be used. That is generally not the case for other software. Fixing kernel bugs faster will increase the pace of Linux desktop adoption, as each bug is a potential barrier. If you assume 50M users running Linux and each bug typically affects .1% of those users, that is 10s of thousands of people. Currently, the Linux kernel has 1,700 active bugs (http://tinyurl.com/LinuxBugs). Ubuntu has 76,371 bugs (https://launchpad.net/ubuntu/+bugs). I think bug count goals of some kind would be good.
In general, Linux hardware support for the desktop is good, but it could get better faster. From Intel, to Dell, to IBM and Lenovo, to all of their suppliers, the ways in which they are all over-investing in the past at the expense of the future should be clear; the Linux newswires document them in detail on a daily basis. I was told by an Intel kernel engineer that his company invests 1% of the resources into Linux as it does to Windows. It is only because writing Linux drivers is so much easier that Intel is seen as a quite credible supporter of it. The few laptops by Dell that even ship with Linux still contain proprietary drivers, drivers that aren’t in the kernel, and so forth.
Peter Drucker wrote: “Management is doing things right, leadership is doing the right things.” Free software is better for hardware companies because it allow for more money to go into their pocket. Are they waiting for it to hit 10% marketshare first? I recommend senior IBM employees be forced to watch their own 2003 Linux “Prodigy” video (http://www.youtube.com/watch?v=EwL0G9wK8j4) over and over like in Clockwork Orange until they promise free, feature-complete drivers for every piece of hardware in the kernel tree before the device ships. How hard can it be to get companies to commit to that minuscule technical goal? In fact, it is hard to imagine you can be happy with a device without having a production Linux driver to test it with.
It is amazing that it all works as well as it does right now given this, and this is a testament to the general high standard of many parts of the free software stack, but every hardware company could double their Linux kernel investment without breaking a sweat. The interesting thing is that PC vendors that don’t even offer Linux on their computers have no idea how many of its customers are actually running it. It might already be at the point that it would make sense for them to invest more, or simply push their suppliers to invest more.
There are more steps beyond Step 1, but we can work on all of them in parallel.
And to the outside community:
* Garbage collection is necessary but insufficient for reliable code. We should move away from C/C++ for user-mode code. For new efforts, I recommend Mono or Python. Moving to modern languages and fewer runtimes will increase the amount of code sharing and the pace of progress. There is a large bias against Python in the free software community because of performance, but it is overblown because it has multiple workarounds. There is a large bias against Mono that is also overblown.
* The research community has not adopted free software and shared codebases sufficiently. I believe there are enough PhDs today working on computer vision, but there are 200+ different codebases (http://www.cs.cmu.edu/~cil/v-source.html) plus countless proprietary ones. I think scientists should use SciPy.
* I don’t think IBM would have contributed back all of its enhancements to the kernel if it weren’t also a legal requirement. This is a good argument for GPL over BSD.
* Free software is better for the free market than proprietary software.
* The idea of Google dominating strong AI is scarier than Microsoft’s dominance with Windows and Office. It might be true that Microsoft doesn’t get free software, but neither does Google, Apple and many others. Hadoop is good evidence of this.
* The split between Ubuntu and Debian is inefficient as you have separate teams maintaining the same packages, and no unified effort on the bug list. (http://keithcu.com/wordpress/?page_id=558)
* The Linux desktop can revive the idea of rich applications. HTML and Ajax improve, but the web defines the limits of what you can do, and I don’t think we want to live in a world of HTML and Javascript.
* Wine is an important transitional tool that needs lots of work (http://bit.ly/fT3pXr)
* OpenOffice is underfunded. You wonder whether Sun ever thought they could beat Microsoft if they only put 30 developers on it, which is tiny by MS standards. Web + OpenOffice + a desktop is the minimum, but the long tail of applications which demonstrate the power of free software, all need a coat of polish. Modern tools, more attention to detail, and another doubling of users will help. But for the big apps like OpenOffice, it will take paid programmers to work on those important beasts.
There are other topics, but these are the biggest ones (http://keithcu.com/wordpress/?page_id=407). I’ve talked to a number of kernel and other hackers while researching this and it was enjoyable and interesting. I cite Linus a fair amount because he is quotable and has the most credibility with the outside world Although, Bill Gates has said some nice things about Linux as well.
If you want to respond off-list, you can comment here http://keithcu.com/wordpress/?p=272.
Thank you for your time.
Keep at it! Very warm regards,
-Keith
I’ve tried for a while to figure out why computer vision is mostly still in research labs in spite of the fact that there are many thousands of people and different algorithms and codebases for doing computer vision. One analogy that occurs to me is image compression.
There are an infinite number of ways of compressing an image, and each one gives a different result. In principle, we could have 1,000s of people around the world working by themselves on this very hard problem. But, it would be better to take a combination of the best ideas, and have everyone use that.
While codecs and computer vision seem quite different, they share an important similarity: in the pipeline of computer vision, from pre-processing to feature extraction, each step produces a smaller amount of data. At the end of the analysis you might be left with the data that this is an image of your house, which is just a few bytes. This compression is also precisely what a codec does.
Another similarity is that decoding is much simpler than encoding. Decompressing an image is faster than compressing it, and the encoders can typically get smarter while the decoder doesn’t even realize it. Likewise, we have plenty of software today that can generate a photo-realistic image of a house. The computer is doing the reverse process of what happens in our eyes.
So perhaps it could be that we have 1000s of computer vision people around the world taking an image and extracting the data, but it is some combination is the best. To be fair, this doesn’t tell us how hard problem is. Will it take the best ideas of 3 or 50 people?
To answer that involves look at each piece. Note that there is plenty of good free code for image processing, which is an important piece of computer vision. When it gets to lines and edges, it seems like that is less well decided. But I suspect that there are many ways to do this, but we should just pick some robust way and move on. [More here]
I’ve discovered the best codebase for people who want to work on computer vision is http://stefanv.github.com/scikits.image/index.html. It’s got Python-powered SciPy and DVCS.
So let’s get going.
I have had this idea for a Virtual Darpa Grand Challenge for a couple of years now and I’m shopping around this slide deck to angel / VC people. I don’t know many, but I am looking and learning.
But I thought I’d also put this out there to the Linux community and see what they think of it. I’ve never done anything like this before so I’m not even sure if I should take on this idea, but I’d be interested in hearing what people think of it and any advice on how to make it happen.
(The first few slides are background because I can’t assume someone knows about the benefits of free software.)
Thanks!
-Keith
Here is the latest version of the PDF you can also download.
|
|
