OSA Alert Industry Interview

Open Source: The Only Viable Business Model

Known as a thought leader and open source pioneer, Michael Tiemann today works as Vice President, Open Source Affairs at Red Hat. He also currently serves as President of the Open Source Initiative and the GNOME Foundation. In 1989, he co-founded the first profitable open source company, Cygnus Solutions. In a recent speech, he gave detailed, annotated support for the superiority of the open source development model, and ended up highlighting all five development-oriented benefits of open source on OSA's top ten list.

OSA Alert: I've heard a lot of great things about “Exonovation – Leveraging Innovation from the Edge,” which you gave as a keynote at the ETE2009 conference. Can you give me a rundown?

Michael Tiemann: Sure. The term “exonovation” corrects a misfeature of the English language. The term “innovation” makes one think of things that happen within the walls … inside the organization. Indeed, a lot of organizations think of innovation as a core capability. However, in a book that was published in 2005 by John Hagel and John Seely Brown called “The Only Sustainable Edge,” they argued – I think very effectively – that the real excitement in the new world economy comes from the relationships that company build at the edges. The real B2B relationships. The term “exonovation” makes it clear that the interesting new comes from outside, not inside an organization.

Let me address the reason why we must be so mindful about how we treat the new in the decaying context of the old. As we look at what companies are challenged with to succeed in the new economy, people talk about the importance of innovation, and how important it is to use technology to cut costs and reach new markets and improve customer service et cetera. Yet when we look at the macro perspective – and I mean the super macro perspective – multiple industry analysts have estimated the global spend for Information and Communication Technology (ICT) at 3.4 trillion US dollars per year. A lot of money.

Yes, indeed.

The shocking part of that number is that it can be argued that one trillion US dollars a year are being wasted either on applications that are abandoned before they ever go into production, or spent on applications that are missing key features or are late to the point of interrupting operational capability or both. Back in 2006 when I started putting these numbers together, one of the challenges I faced was that a lot of people were not all that concerned about the one-third of every dollar that was being wasted on bad technology spend because they were so happy with how fast the two-thirds was growing. That dynamic is completely gone in 2009. The idea of spending a dollar on ICT and getting back – at best – 66 cents in value is unconscionable, especially today when you consider how important nowadays every trillion dollars is to the global economy.

Isn't that fairly common? Aren't most industries wasteful?

If true, that's a terrible state of affairs, and one that could be subject to remedy. In fact, the rest of my presentation talks about specifically how the open source model dramatically remedies this gap. We look at the cost of the proprietary software model in terms of duplication of effort, in terms of average software quality, in terms of the tactics some of the top software vendors use to destroy their competition, even harming themselves in order to do damage to other people. This is not consonant with the idea of building value.

To give you an example, in 2004, 2005 and 2006 a company called Coverity, which does software defect detection, analyzed the average number of defects per thousand lines of code in proprietary software and compared it to open source software. The average, which has remained consistent, is that proprietary software contains 20-30 defects per thousand lines of code. When they first measured the Linux kernel back in 2004, the kernel consisted of 5.3 million lines of source code. (For comparison, Windows XP is reported to weigh in at around 35 million lines of code.) If Linux were industry average, one would expect between 100,000 and 150,000 bugs or defects across 5.3 million lines of code, and yet the Coverity scanners were only able to find 985. The Linux community got the report and published a message saying, “Hey, there's only 985 bugs to look at, let's clean this up!” Within six months, every single defect that the community felt was legitimate was corrected. By contrast, if Microsoft wrote industry average code, their 35 MLOC code base would contain more than 7 million defects! You can see the problem.

Why is there such a big difference in error rates?

Let me begin with Fred Brooks, who wrote a book called the “Mythical Man Month” about his experiences managing the IBM OS360 Project, which was – at the time-- the most expensive single industrial project ever (it even cost more than the government spent on developing the atomic bomb). The book was a belated explanation to his boss – Thomas Watson, Jr. – about why having so many of the smartest people in the world and working with as much money as IBM could feed into that channel, that the operating system was so late and didn't have so many of the features that were expected. Fred Brooks didn't have an answer for that question in 1964 when he was asked it, but by 1975 he had developed a comprehensive answer. The title of his book is based on the analogy that the bearing of a child takes a woman nine months, no matter how many women are assigned the task. He argued that the development of software is an organic process that's not subject to industrial logic. He observed that adding more people to a late project only makes it later, and that assigning more people to fix bugs that are out of control actually adds more bugs than it removes.

Now look at the stark contrast between being able to manage 985 bugs down to none, versus the situation we see with Windows XP, which ultimately became Vista, and we all know how much difficulty Microsoft had in managing that project. Vista shipped horribly late, and it shipped without a single feature that was advertised for enterprises. The only substantial feature it shipped with was one that no enterprise wanted – the Hollywood-mandated media control protocols and trusted computing platform. Basically, they made sure that Hollywood can trust your computer even if you cannot.

And it was full of bugs.

Exactly. Proof that adding more people to a buggy system only makes the system worse.

In 2005, Coverity remeasured the Linux kernel, which had grown to 5.7 million lines of code, and found that the defect density had actually decreased. So some people wondered if maybe the Linux kernel was not a fair test. Maybe something in the water makes the Linux kernel people off-the-charts smart. So they looked at 32 other open source applications, ranging from the PHP scripting language to the GNU C compiler, and what they found was that the worst of these 32 open source programs was only 50 times better than the average proprietary software.

What is it about the open source development model that generates so much innovation and so few bugs?

In my presentation, I showed a 2001 study by James Herbsleb, Roy Fielding and Audris Mockus about the Apache Web Server. (“Two case studies of open source software development: Apache and Mozilla,” published in TOSEM, July 2002.) They compared some development metrics between proprietary software – a few companies volunteered access to their code repositories – and open source code repositories. They measured – for each developer working on the project – the number of lines they added, the number they deleted, the number they changed, and how many bug reports they responded to, and so on. They ranked them by activity where the Developer 1 did the most. In both the proprietary and the open source cases, Developer 1 did approximately 20% of everything. If it were true that software worked according to industrial logic, any of the programs could have been produced by five developers. But of course we know that's not true. When you look at the graph, five developers get you to about 40% of everything. When you get out to 10-15 developers, you climb up to about 75-80% of everything.

It turns out that in the proprietary world, project size tended to be about 25-35 developers, and projects exhibited what we call a “short tail.” You see this relatively linear graph (on logarithmic paper) from Developer 1 up through Developer 25. Well, with Apache back in 2001, they had 388 developers. You do see the characteristics of the short tail up through about 10-15 developers, but after that you get this incredibly long tail out to Developer 388. What the paper argues is that this “long tail of participation” leads to more features, implemented sooner with fewer bugs.

How does that play out in the real world?

After seeing that result presented by Professor Herbsleb in 2003, I have developed a new insight into the study. Imagine that you are Developer 388, credited with changing a single line of code. What is the real experience of that developer? Are they like the person playing the triangle for the symphony, waiting the whole night to strike a single note, and then getting paid full union scale for their troubles? Or do we believe – as I do – that Developer 388 does not even conceive of themselves as an Apache developer? Perhaps they are working on another system that interacts with Apache and in the course of it, something unexpected happened. After going through every line of code that they wrote, they realize that the problem is with Apache. They go to the Apache code and find exactly what they expect – that one line of code that needs to be modified … a condition has been reversed in an IF statement or a test has been omitted, or whatever. They fix the line and send a patch off to Apache (which, by the way, is where the name comes from … “a patchy server”) and now they have a server which works so they can continue developing their own project.

Now, let's look at the world of proprietary software. Say you are a developer working on a project that uses a proprietary database … Oracle, Microsoft SQL Server, take your pick. You conclude that there must be a bug in the database software, so you call up the vendor and say, “I've found an error in your database.” You can just imagine the reaction. After the laughter subsides, the vendor might say, “Well, maybe you have, but we're not going to fix it.” Or maybe they will fix it, or have already fixed it, but you can't have it until the next release. Well, your problem is today, so you are motivated to put a workaround into your code to deal with the problem, something completely false and unnecessary. For example, you might put in a line that says, “When I ask the database what 2+2 equals and it says 5, change the 5 to a 4.” It is extremely difficult to maintain a lie, but this is exactly what happens because they don't have access to the code. Over time, the hundreds of vendors who ultimately comprise a proprietary software stack all find themselves putting workarounds in that are unmaintainable and untrue. I believe this explains one reason why proprietary software becomes so brittle, so difficult to maintain, so soon after its implementation. Conversely, it explains why open source software is so flexible. It is not full of such garbage.

Everything that's there is there for a good reason.

Exactly. If we go back to the fact that we're wasting a trillion dollars a year, and look at the fact that the proprietary development model is causing every proprietary developer to put themselves into a special class of hell by being forced to put falsehoods into their own code rather than dealing directly with the subject, I think it explains a lot of what has been observed by open source software customers like the New York Stock Exchange and the National Security Agency. Namely, that open source software is better, faster and cheaper because at the core it works and can be fixed. What you hear in the proprietary world is, “You can't fix it. Maybe we'll fix it, maybe we won't. Depends on how we feel.” In the open source world, the mantra is, “If anybody can fix it, somebody will.” What is fantastic about the open source community is that “anybody” does not mean “anybody at Red Hat” but anybody in the community of multiple millions of developers. When you look at the quality comparisons, it is obvious that the network effect has come into common use.

What drives that? Why would it be that way?

The proprietary system is based on a highly asymmetrical model. Some people have information, while most people have none. Monopoly power has gone to people's heads. The companies who have ordered the world in such a way that they sit in a seat of individual power where they can destabilize their weak competitors and just print money, why should they go back to the messy business of competition? It would mean taking a risk that they might not win every contest they enter. Why not sit in a position where you can prevent the competition from happening in the first place?

To give you an example, let me share an example I heard from our new CEO, Jim Whitehurst. While he was COO for Delta Airlines, he helped take the company out of bankruptcy. When they were slashing every cost they could, the one cost they couldn't cut – in fact it grew – was information technology. After he joined Red Hat, he said, “Now I get the joke.” The vendors who had locked Delta into a proprietary decision matrix and architecture were absolutely not going to let that client escape given that those vendors held all the cards. It's amazing how different the competitive landscape looks when you can postulate an open source alternative.

Open source sort of turns the traditional competitive model on it's head then, doesn't it?

It turns on its head the notion that property is the universal, all powerful, perfect way to organize the world. It replaces it with the concept of dynamic competition. What's exciting about the 21^st Century are the innovations that happen as a result of the new relationships that can be constructed between companies. Those relationships are foreclosed or prevented by a status quo or a singular vision a propriety rights that allocates monopolies to a small number of players. That world of stasis and control is not the right world for the future. When people ask the theoretical question of why monopolies are bad, they are bad because they preclude competition. When we look at the results of Vista, we can say “Wow, the only possible way to explain this result from a $6 billion a year R&D budget for 10 years is that they don't have their eye on the ball.”

They have become an institution that serves itself.

Exactly. Open source says, let's break down all these barriers. Let's make it so every single day, the dollars go to the people who deliver the best value. It's a scary thought for someone who has grown accustomed to a promised place at the top of the food chain. Well, Red Hat just released it's results and they show that we can be profitable and infinitely competitive at the same time. In fact, some of the biggest companies in the world have downloaded our software for free from our website so they can compete directly with us. They say, “Hey, why go to Red Hat when you can come to us for half the price.”

You sell more than just a product.

Exactly. It's the value of the comprehensive package. I'll tell you what. The day that Red Hat stops being the primary source of value for the software product is the day people start buying from somebody else, including the “buy versus build” decision. Red Hat's half billion dollar plus revenues speak to the fact that 500 million times last year people said spending a dollar with Red Hat is better than spending it on internal development or proprietary technologies.

The capitalistic model is based on creative destruction. New, more agile players replace the old companies, and there's a lot of waste.

I'm all for creative destruction when there are generational shifts. For example, in the 1970s, when people woke up for the first time that maybe gasoline wouldn't be cheap for ever, there was an expectation that there would be an inventory of gas guzzlers that would go by the wayside. What a real tragedy it would have been if Toyota and Honda – after getting the public hooked on economy cars – had started building crappy, or worse, deadly dangerous cars. The point is that it's one thing when a winner replaces a loser, but that's not the case with proprietary software. The one trillion dollars in waste is not caused by great new technologies replacing crusty old technologies. It's about a model that has failed.

We all know how successful the World Wide Web is, and Tim Berners-Lee is well known today. You can find evidence of the World Wide Web as far back as a 1945 article in The Atlantic Monthly by Vannevar Bush entitled “As We May Think.” It is uncanny how much the article seems to describe the Internet and Google. Years later, academia and industry started developing prototypes of networked, associative, hypertext-linked systems – more than four dozen attempts. The difference with Tim Berners-Lee is that his was the first system to use an open source license and to invite universal participation on a level playing field of open standards.

I think it teaches us that we are living in a world where we do the same thing day in and day out, and so when somebody does something truly different, you get a revolutionary result. The same thing is going on with enterprise infrastructure software. I consistently see tremendous cost advantage, tremendous performance improvements, tremendous quality improvements. Open source is the revolution that promises – through results – to put a dent in this one trillion dollar problem.

Does open source represent a whole new economic model?

Some of the really important aspects of open source – such as letting go of control and building a community – are what make it work. I can't tell you how many times I've heard a CIO say, “Wow, I'm so impressed with what I see, we're going to build an internal open source community.” The paradox is that they don't see that it's the ability of the whole world to tell Red Hat what it's doing wrong that creates the results. Creating an internal system and slapping the open source label on it will not work.

Can the open source collaboration model work at the general business level? Everyone seems to agree that business collaboration is a good idea, but when it comes down to sharing resources or sales leads or competitive information, things often fall apart.

I can provide some backhanded encouragement. I've been doing this for 20 years. There are a lot of people who don't have the patience to do anything for 20 years. Yet, there's a lot to be said for taking the 20-year view. Look at banks. They started thinking of their mortgages like commodities and look what happened. I think we live in a context today where people have completely lost sight of the longer term horizon. We have a terrible problem in that we've been developing software the wrong way for 40 years. One out of three dollars are wasted, and yet can you imagine a manufacturing company discarding 33% of their products?

Going back to your car analogy, look at the example of Saturn. GM had to create it outside of its normal management structure to make cars that could compete with Toyota.

Well, we all know the story of W. Edwards Deming. The company that rejected his teaching before he went to Japan was General Motors. Alfred P. Sloan heard about this whiz kid who was doing great things in the U.S. Army and called him in. Deming accepted the job in good faith and did a top to bottom review. At the end, he told them that the problem with GM was that it's factory system was destroying the hope of the workers. By treating them as cogs in a machine instead of as valuable tradespeople, every opportunity for efficiency was being lost on the factory floor. He told them to break down the barriers, bring workers into the decision making process, and value them as human beings. Of course, the response from GM management at the time was, “How can you be so stupid? Labor is the enemy! Get the hell out.” When you look at Deming's admonitions and his concept of transformation from back in the 1950s, it's all about human empowerment and valuing the self. When I look at what he wrote and I compare it to the success factors in the open source world, it's uncanny. Today, the IT world is in crisis. Our top revenue companies are doing everything wrong. They profit from the disasters they visit upon their customers. That's not sustainable. They should be profiting from the value they create for their customers.

Haven't we seen this before with the change in thinking regarding quality and six sigma defects rates, etc. back in the 1980s?

Yes. In fact, ironically, Deming says that quality is a byproduct. It's proof that you're doing it right. Too many are focused on the short-term model of “How do we get over this bar to pay ourselves more bonuses?”

I mention it because it was the tolerance of American industry for defect and obsolescence that created the opening for Japan Inc.

Yep. This gets us right back to Developer 388. The final point in Deming's 14 points from his book Out of the Crisis is that “Transformation is everybody's job.” Open source provides a model where “everybody” can be interpreted to mean literally everybody. Not everybody on the shift or everybody at that company. The whole world. Developer 388 sees a singular problem with Apache and fixes it because he or she saw it. When you integrate that over everybody, you get a system that is asymptotically approaching perfection with respect to defect rates. Our rates aren't 20 to 30 per thousand lines of code. Open source rates don't even round up to one. When I give this talk to the Department of Defense, I ask “Can you imagine a commander asking for another battalion, and getting the answer 'I've got good news and bad news. The good news is there are 1,000 soldiers in the battalion. The bad news is, 20 to 30 of them are insurgents and we don't know which ones.' If you take the open source battalion, you are putting a full fighting force on the field.”

Finally, I understand you used game theory in your presentation.

Yes, I do reference a fascinating paper in that talk. I'll send you a link, as well as some other information to share with your readers.

EDITOR'S NOTE: The relevant references are explained and linked in this excerpt from Michael Tiemann's blog post of May 9, 2007:

There are three academic authors that I believe explain at least 90% of the success I have observed among both open source companies and the open source community. (I need to figure out how to properly integrate their work into our research page.) In summary: