Notes on Glyn Moody's Book
Rebel Code, Linux and the Open Source Revolution

Pablo Nogueira

20 June 2003

Disclaimer

These type-written notes are personal reminders written to myself, not reviews, nor abridgements, nor literary criticism. They may not make sense unless you've read the book.

 

[Excerpt from page 4]: "Thanks to the advent of relatively low-cost but powerful PCs and the global wiring of the Net, the new hackers are immeasurable more numerous, more productive, and more united than their forebears. They are linked by a common goal—writing great software—and a common code: that such software should be freely available to all. Hackers rebel against the idea that the underlying source code should be withheld. For them, these special texts are a new kind of literature that forms part of the common heritage of humanity: to be published, read, studied, and even added to, not chained to desks of inaccessible monastic libraries for a few authorized adepts to handle reverently."

Free Software vs Open Source. The meaning of `Free'

We have the following categories for software:

These concepts are actually being pushed into other fields like documentation (eg. LDP) and journalism (eg. Slashdot.org) [p301-302].

For Richard M. Stallman (RMS) there aren't just two kinds of software, "good" or "bad", but rather "free" or "proprietary" [p259-260]. Read his Definition of Free Software. RMS plays a crucial role in the free software community. Read also the GNU manifesto at www.gnu.org. In short, programmers get paid, but not that much. Nevertheless, proprietary software has gotten into GNU/Linux world, using Linux as base OS, something Stallman considers immoral. It has backfired on him.

For Michael Tiemann, being pro open source makes sense not as an ideology but as a brand strategy [p241].

There is a tension between Purists (pro free) vs Pragmatists (may include proprietary). On page 265 the incident regarding the Qt graphical library is described. It run like this: pragmatists put proprietary stuff to fulfil a need promptly, the purists complain and either force proprietary software to drop, to come open, or they start their own clone (eg. GNOME). Another example is Eric Allman's Sendmail Inc, a company founded by the creator of sendmail that develops commercial versions of the popular mail daemon. The core material, the innovative stuff, is kept free [p243].

There are tangled relationships between trade secrets, free software, copyrights, and freedom [p312]. Moreover, GNU GPL might not stand in court! [p312]

The separation of open source from free software took place on the Freeware Summit organised by O'Reilly on 7th April 1998 in Palo Alto [p166ff], which allowed licenses other than GNU GPL (a move that upset Stallman). An Open Source software license guarantees at least the following:

Thus, Open Source is now related to a development methodology rather than a (more radical) non-profit endeavour. This has upset RMS but convinced small and big companies to take the plunge. It has been considered as a good move in public relationships.

More on ideology

RMS aims at providing software free to use and share with others that can also be extended to our own purposes. New additions must be free too and one should only pay for physical delivery. Close source implies domination: "I don't let you understand how this works" so that you depend on me. RMS considers this immoral. At the end of the day, this is only an instance of a wider credo of liberty in general (for him all freedoms matter and he can contribute in the field of software). Isn't scientific knowledge free? Shouldn't music be free?

He has devoted his life (time, money, etc.) to this endeavour. He seems to epitomise the concept of down-sizing: he does not have a car, does not own a house, has no wife, no children, spends little money ("if you spend a lot of money then you're the slave of having to make money. Then money jerks you around, controls your life... There's only one way I could have made that money, and that is by doing what I am ashamed of", that is, writing proprietary software. Is RMS asking us to be ascetics? He does not compromise his principles and defends them stubbornly, even if at times that impairs the widespread use of its GNU software [p28-29].

To ask qualified programmers to develop software for free in their spare time seems weak. Society does not seem to work this way: why not call for freedom of service (don't pay for consultancy), freedom of housing (don't pay for accommodation), free music, etc. It's not only a matter of freedom but also of technical innovation. Also Copyrighting or patenting can be pushed to the extreme.

Eric S. Raymond describes himself as a libertarian. He believes in a small government, free market eschewed of (government) regulation (he calls it coercion), right to bear arms, market anarchism (abolish government altogether) and, of course, free software. The root of such ideals being his powerless feeling and dependence as a child due to the mild spastic palsy of his left leg [p144-153]. I cannot help but see a contradiction here. Without government regulation there would be no case against Microsoft. For a start, sometimes the market does not self-regulate fairly due to some "dirty" practises. ESR has been criticised for "promoting" himself too much [p153]. Well, geeky hackers like to talk a lot about themselves, they aren't that shy (and imho there is nothing wrong with it)

The following metaphor by Moody is illustrative [p154]:

"There are precedents for gifted individuals who conceive and work on huge projects, often with little hope not only for recompense but of recognition. In other spheres, these people are called artists, and much of the history of art is the story of those who create masterworks out of an inner necessity to do so, over and above what they may have been paid to produce."
Alongside artistic satisfaction ESR points another key factor: "the reputation game".

Donald Knuth is one of advocates of software as art, just as any human endeavour like writing poetry or doing mathematics. His chief concern is to teach how to write beautiful programs [p156]. Art can only be appreciated when is open to scrutiny.

Larry Wall puts it succinctly: "people really do help people for the sake of helping people" [p161]. There is a sort of Marxist cloud lurking in the air: "from engineerings according to their ability to engineers according to their needs" [p224].

What creates such an enthusiasm around GNU/Linux and Open Source is not only that the software is better (faster, reliable, flexible, etc) and cheap, but also that users are in control of the technology, they no longer depend on a proprietary vendor to solve their problems, they can fix them or contribute or get help from someone at little cost. There should be no fear of sharing code. At the end of the day it will be only altered by other programmers interested in the same area, so one should be able to still make money, but not as much (as RMS put it). Not everybody will be interested or have the time to tinker with our software (see the Linux Kernel example), so it will be under control.

The development model

The development model is based on distributed hordes of hackers (enthusiastic programmers) cooperating together with some team of experts in control. Andrew Tanenbaum was the first critic of a purely distributed model [p78]: one person (or a team) must be in charge. And despite Linus's answer against centralisation that is what's happened with kernel development to date.

Having one boss yields a bottleneck effect [p177ff]. However, McVoy comments: "Two leaders, doesn't work" [p178].

Responsibility is granted by means of reputation (reputation game) [p84]: the programmer is already working on a problem, has a past record of "good choices", and has credibility in the eyes of other hackers (and perhaps, even if not mentioned, the blessings of the chief leader). This stresses the fact that the leader must behave as a benevolent dictator, just as Linus Torvalds did. The keyword is meritocracy, a natural selection inside the community. But the personality of one man seems to have played a big role [read on p119 Brian Sparks's words about Linus. More detail on the development model can be found in The Cathedral and the Bazaar on ESR's home page.]

There is the risk of Incomprehensible Programs. The term comes from Joseph Weizembaum's book Computer Power and Human Reason where he used it to describe very large AI software projects whose rationale is no longer understood by the current team of developers. In the case of open source it refers to the fact that only a few hackers know about the internals of the software in which they work, but if the developing team unfolds into distributed and loosely-coupled clusters then anarchy will be the outcome. At the end of the day, software is dependent on a scarce commodity: qualified and motivated developers [p248].

There are two distinct schools of thought within the Linux community [cf. networking fork incident, p77]:

  1. The pragmatic approach: make it work first, then make it better.
  2. The perfectionist approach: make it better first.
Linus Torvalds adheres to the first one and gave it impetus [p80]. His model: release early and often, correct every patch as soon as possible and make a new release. This posed a problem for the distributions [p96]. He adapted to that problem by releasing a stable and an experimental version of the kernel [p111].

Being innovative is not the issue, as proved by the disastrous consequences of innovation taken by Caldera [p229]. It has to be ripe and arrive at the right time, and take into consideration user needs, public relationships, etc.

After the hype of releasing Mozilla (Netscape's open source move), Jamie W. Zawinsky (JWZ) steps out [p202-203] because there is no pressure to deliver on time. He dislikes the "short bursts", the long periods between major releases and the lack of deadlines of most open source projects. John Ousterhout comments that in an open source development model some things don't get done: "All of the additional kinds of services and value-adds that you need if the company is going to be using this for mission-critical applications — development tools, higher-level kinds of facilities" [p254] The stuff hackers don't like to do? According to Brian Behlendorf [p249] there are unfinished tasks in an open source project, things that no one wants to do because they are difficult or because they require specialised knowledge. Recall that hackers usually work on open source projects in their spare time. Nevertheless, the rate of development of open source seems amazing: in 1991 a Finnish student criticises Minix and starts his own OS for the PC. In 2001 that project has evolved into a full-scale OS with millions of users and many distributions and companies around it.

JWZ on Netscape's attempt to revive its browser by going open source:

"Open Source does work, but it is most definitely not a panacea [...] you can't take a dying project [Netscape], sprinkle it with the magic pixie dust of `open source', and have everything magically work out. Software is hard. The issues aren't that simple." [p203]
As a result we find the following two models: the rewarded auction bazaar or the hacker for hire sites, where open source projects are posted and (hopefully qualified) programmers bid. Examples are SourceXchange and Colab.Net. These are projects of substantial size (incomprehensible for a single programmer) and have a license agreed on negotiation [p249].

There is also the issue of redundancy. Many people work on similar projects at different stages and related ideas. How many window managers are there currently? [p159]

Forking

A fork (a new branch of the same product developed by different people with a different rationale) is deem disastrous in hacker terms [p75]: there is a duplication of effort into two separate strands of development of the same original product, the user base decreases and gets confused, there are fights among developers arguing about which version or feature is better, etc. But forks can be healed thanks to the open, free nature of the development. Changes can be taken from one strand to the other, and they can even be merged, though that would be a lucky coincidence for most cases, since forking occurs due to severely different design issues, not minor local disagreements. Examples of forks are: Unix vendor fragmentation, XEmacs vs Emacs, Van Kempen development of the networking stuff that was forked by Alan Cox (ended up in drastic healing: one project was dropped due to Linus Torvalds's choice). KDE and GNOME are not forks, applications run in both GUIs [p266] (sure?)

ESR explains why forks rarely occur [p154]. What makes a project and people stick together is "peer esteem... where prestige is measured not by what you have, but by what you give away". This would also explain why these hackers devote so much time and effort to such complex tasks with no material reward whatsoever.

Selling services, not the product

Services are still necessary as not everyone is an experienced programmer/sysadmin. The user should have the freedom to choose his service provider. With open source more service providers arise (?) Applications, support, consultancy, education—these are open markets. Red Hat and other distributions have become successful by providing such services.

A lesson Irving Wladawsky-Berger (IBM Linux Tsar, as Moody dubs him) has learnt: "in the areas where it's most important to choose technologies that facilitate interoperability, integration, application portability, let's work with the industry...let's work with our competitors, let's work with the open source movement, because that's the layer that we'll [sic] facilitate pulling it all together. That still leaves plenty of areas to go add value and compete in the marketplace: how well you support your customers, how well you put it together, what happens when they have a problem." [p290] He puts the TCP/IP open standard as an example of truly multi-vendor interconnection: "The whole notion of separating application development [from specific platforms] has been the Holy Grail of the industry [not that it is conceptually difficult, but that there are two many vendors each wanting a piece of the cake and unwilling to adopt other vendor's suggestions] I think with Linux we now have the best opportunity to do that. The fact that it's not owned by any one company, that it's open source, is a huge part of what enables us to do that. If the answer had been, well, IBM has invented a new operating system, let's get everybody in the world to adopt it, you can imagine how far that would go with our competitors." [p292]

Wladawsky-Berger is also answering the following question: Do end-user applications benefit from the open source methodology?

Distributions, pushing it further

Linus concentrated on the Kernel. GNU in system applications. Then came distributions, third-party vendors, companies... [p88]. The lack of support and services was the main concern for companies and individuals who wanted to use Linux. When introducing a new OS you have to have an user base built around applications. Operating systems by themselves are empty, they are just the platform for third-party applications. This is when GNU comes to the rescue in shell, emacs, compiler, but at the time GUIs, office applications, etc, were still missing [p96ff]. For instance, Red Hat won most magazine awards for best product but users still chose Microsoft NT 4.0, not because of brand name or support, but because of an "ecosystem" of applications that surrounded the OS. You can do almost anything under Windows platforms because there is some vendor who can help you [p226-227].

Nowadays there is a need for standardisation (eg. LSB) in order to "move between distributions" [p310].

Currently, most big computer business and firms have moved toward Linux, especially due to the separation of the concept of open source from that of free software. Hackers have always suspected that companies use their software internally, as says Brian Behlendorf, but big companies have decided to back and support Open Source publicly. First was IBM's adoption and free contributions to Apache (despite initial middle management opposition—they had no profit, no metrics, p211). Then came Computer Associates, Oracle, SAP, etc. Hardware companies such as Compaq, HP, Dell, etc, are willing to sort out the hardware compatibility ("lack of drivers") issue. The investment in Red Hat, Caldera, and the major distribution companies, the alliances and the IPOs... Business support has given credibility and wide spreading to GNU/Linux and the open source movement.

Going to business

Business reshapes the open source movement, even if that upsets hackers like RMS. However, open source provides some guarantees: forks can be healed, there can be more than one strand of development (GCC vs EGCS), no vendor will become dominant because competitors can take the open source and start a new strand (quick technical catch-up) [p310].

Free software has no cost, "a product is manufactured for you for nothing by other people... Will the business models based on it work in the long term?" Larry McVoy thinks it is a failing model [p313] Open Source detriments the vendor in favour of the user. Companies based on Open Source (such as the distribution companies) will never attain the power of a Microsoft, Sun, etc. He also (correctly imho) pinpoints a key drawback: the scalability of the model. Big, complex programs need teams of well-motivated professionals that work for long periods of time on something college students cannot hope to do in a month and grasp it all. Many contributions have come from those company's engineers in their spare time. The development cost is not gone. Coordination is a key factor [p314]. Moody replies on page 315 that Open Source is being held by big companies that put their engineers into it, and that open source lives "beyond the market" (students, dedicated hackers like RMS, other engineers) which can take over. The issue is not whether Open Source companies will make big money, but whether the movement will grow and progress. There might exist different sorts of communities: some centred around small-sized projects, some centred around medium-sized projects, and others perhaps around "incomprehensible programs".

The time span dedicated by part-time hackers is very narrow, from months to a few years (RMS and dedicated hackers notwithstanding). The SourceXchange and Colab.Net initiatives try to solve this problem by hiring and paying hackers. Moody pinpoints developing countries (India, Mexico) as a source of new talent. There's also the issue of leadership (who will succeed, Linus or RMS? Moody suggests a few names). Hackers get hefty amounts of money from distribution companies like Red Hat for their contributions. Nothing wrong unless this money comes from anything related to close source or non-free software. Will this spoil the hacker spirit? ESR says no: "commercial demand for programmers has been so intense for so long that anyone who can be seriously distracted by money is already gone. Our community has been self-selected for caring about other things—accomplishment, pride, artistic passion, and each other." [p236]

Microsoft's wrong practices

Andrew Tridgell (creator of Samba) comments about MS: "They consider themselves the only implementation of this protocol [SMB], even though they know they're not. But they control the server and they control the client, so they can just stick the code at the client and the server and it will work, and they don't really think much beforehand about designing it appropriately." Moody adds that this is a note of warning. A dominant position also masks bad coding. [p273]

Read ESR's Halloween papers for more on MS's practices. Another example is given by Moody on page 311ff: Vinod Valloppillil's comments on "de-commoditising" protocols and the Kerberos example, that is, the "embracing, extending, and extinguishing" of a protocol. A summary of these ideas can be found in Karl Fogel's celebrated essay Microsoft And The Rest Of Us.