Happy new year to you, faithful readers! 2008 brings the continuation of my series, Creationist programming vs Evolutionary programming, of which this is the fifth installment. Previous installments: Part I (Creationist programming, The Devil), Part II (Intelligent Design, Polytheism), Part III (Unintelligent Design, Lamarckism), Part IV (Supernatural Selection, Teleological Evolution).
Natural Selection
As far as paradigms for understanding software development go, the notion of evolution under godly guidance was an improvement over that of direct design by purposeful gods, which was itself an improvement over the notion of immediate creation. But in each case, this was only pushing back one level the assumption of a driving intent external to the world. Real evolutionary theory does away with this assumption. Survival of the fittest does not suppose an external criterion of fitness to which living creatures are submitted; rather, survival itself is the only criterion, tautological and merciless. Survival is its own purpose: those programs that survive, survive; those that don't, don't. Changes that improve the odds that their host software should survive and propagate, thus statistically tend to propagate themselves and colonize their respective niches. Changes that decrease the odds that their host software should survive and propagate, thus statistically fail to propagate themselves and eventually disappear. The cumulative result of this natural selection is an evolutionary process that favors bundles of traits that tend towards their own reproduction. This freewheeling evolution necessitates no godly intervention, neither by an intelligent conscience, nor by madmen. More remarkably, programmers are no gods above it, and their actions are no such interventions. They are but machines like others, bundles of self-reproducing traits competing to exploit the resources of the universe. As compared to other machines in this programming universe, certainly programmers are unique and different -- we're all unique and different; that doesn't exempt them from the laws of natural selection. Programmers are machines trying to survive in a wild machine-eats-machine world; their actions are their attempts to survive and reproduce by gaining an edge in the race for ever more efficient acquisition and use of reproductive resources. If God exists, then ever since He created the world, He has just been relaxing, sitting back and enjoying the show. Evolution is not guided by God, it is God's Spectator Sport. Such is the paradigm of Natural Selection.
With this new understanding of the world of software development emerge new tools to improve our development processes. We think in terms of self-sustaining systems, evolving and competing based on their ability to survive and spread. We understand that the hosts and actors of this memetic competition are humans as well as machines, or even more so. We may then notice that systems are never born big, and that the only big systems that work are those that were born small and evolved and grew in a way that they were kept working at every step. We explain the spread of ideas in terms of generations of humans and machines passing on their forking and mingling traditions. We understand that pieces of hardware, software and wetware survive as part of ecosystems, with cycles of development and use by various humans, where economic and legal aspects have their importance as well as technical and managerial aspects. We realize that these systems compete on a market ultimately driven by economic costs, of which technical aspects are but a small part, sometimes not decisive, though they are what the technicians obsess about.
Because the forces opposing creation are no devil but malicious humans indeed, we use of computer cryptography and cultivate networks of human trust to achieve security. A Third Wave of Cybernetics attempts to re-create artificial life and life-like phenomena through the emergence of behaviour from many software agents.
Natural Selection provides a big picture that puts haughty programmers down from their godly pedestal and back into the muddy real world. It doesn't offer direct solutions to design problems so much as it dispels our illusions about fake solutions and unearned authorities. No one is a god, above the others, to predict what will work and dictate what to do; our experts' dreams are often but vain obsessions, whereas some rare amateurs' successful experiment may start a revolution. Life is the ultimate judge -- accept no substitute, and respect its sanction.
Inside Evolution
Natural Selection may appear to look down on the world as a soulless marketplace. It will only appear soulless if you imagine yourself in the seat of that laissez-faire God above the world. But face it, you're no god, you're not outside the world and above it. There may be a god, who may or may not be intervening in this World, but you have to come to the realization that He's definitely not you. You're one of us earthworms, trying to make the best out of what you have (or not trying, and thus probably failing and promptly disappearing into irrelevance). Evolution is not something for you to enjoy watching, it is something you are part of, willy nilly. You can't just let nature decide, you're part of the nature that will decide. Whichever genes and memes you carry may or may not survive -- it is largely up through your actions that they will succeed or fail. You're in the experimental set of changes that may or may not work out well, or you're in the control set of the obsolete that will surely be replaced. Such is the view from Inside Evolution.
The tools that matter are those that are available to you. Your resources are limited, and you should invest them wisely. Which tools will make you most productive personally? Opportunities are there to be seized; if not by you now, by someone else later. On the other hand, it may be too soon to invest in some ideas, and too late to invest in others; timing is key. Specialization will help, and can be a long-term investment that provides compound interests. As for cooperation with other non-gods, you can only go so far with your own efforts, and success lies in being able to leverage the efforts of other people. Which tools allow you to reuse as much as possible of these people's efforts? Tools can be technical, or can be social. Not just software libraries, but software communities, software market niches, software business contracts. Of course, you always need some kind of exclusive resource to ensure a revenue stream; free software or not, your combined proficiency, trustworthiness and time are ultimately the only such resource you have, and ample enough to live well if you can market it, though it will probably not make you super rich. On the devil side, intellectual frauds will try to have you adopt their bad ideas, and other scammers will try to divert your resources in their favor; you must learn to avoid them.
As you fully grasp the fact that all actors are individuals, not just yourself, and you take into account incentive structures. Incentive structures will put you and your associates in a position to productively cooperate at your full potential, or to work at a fraction of it; so carefully watch both your legal and business arrangements. You may see that proprietary software destroys incentive from anyone who doesn't fully trust the software owner, and that trust can last but until the eventual catastrophe inevitable in any centralized management; any proprietary software has a suspended death sentence. On the contrary, you may see that free software creates an insurance against disagreement with associates, and ensures pereniality of software investment. With a systematic view of incentives, you stress the importance of contracts and accountability as a way to structure human interaction, re-uniting liberty of means and responsibility for results in complex software arrangements. For instance service-level agreements will allow to robustly build larger, more complex structures than direct command chains. You may recognize the value of free markets as a way to organize people and to evaluate ideas, rewarding those able to invest their resources in the good ones rather than the bad ones. You may celebrate startup companies as light innovation structures with highly motivated personel.
The Inside view to Evolution restores the soul in the market place for software. This soul is yours. You're the entrepreneur of your own life.
TO BE CONTINUED...