Here is the fourth installment in this series for my essay Creationist programming vs Evolutionary programming. Previous installments: Part I (Creationist programming, The Devil), Part II (Intelligent Design, Polytheism), Part III (Unintelligent Design, Lamarckism).
Supernatural Selection
Though Unintelligent Design helps further the field of software engineering, one may realize that while small parts of software are understood, software at large is not understood, much less designed. Lamarckism, by shifting the spotlight to the change process, leads to asking why and how programmers lacking complete understanding choose to keep or change some or some other parts of the software. The immediate answer is that as god programmers write, they stumble upon good or bad features that they winnow by propagating the good and by eliminating the bad. The software writing process is thus some kind of artificial selection, under the careful, intelligent guidance of the programmer God. The programmer God impresses upon the process a definite direction, Progress, and otherwise lets software evolve organically in this divine order. This software paradigm is Supernatural Selection.
Under this paradigm, new tools are selected into prominence. Prototyping tools help the programmer God flesh out as many ideas as possible as quickly as possible, so he may select the correct ones. Formal specifications help define what software should be doing, without worry about how it will be doing it. Heuristic search algorithms use intelligently designed strategies to systematically explore spaces of potential solutions too large to be explored by the programmer themselves. The combination of these two approaches leads to declarative programming, where the programmer God focuses on the intent, and delegates the implementation to the machine. From one phase to the next, programs are transformed through systematic metaprograms. To prevent the devil from corrupting software, formal proofs are developed that perfectly exclude undesired behaviour. To coordinate multiple programming gods, software modules separate interface from implementation, allowing for experimentation and adaptation separately in each part; rational developer communities are created, conferences are given, journals are published.
This whole approach has also been called the First Wave of Cybernetics, combining an understanding of the natural dynamics of software with a faith in the ultimate power of an intelligent and purposeful programmer god, culminating with expert systems using explicit knowledge representation in an attempt to solve complex real-world problems.
Teleological Evolution
The paradigm of Supernatural Selection obviously suffers from the same shortcoming as did the theory of Intelligent Design before it, in that it supposes that the programmer God (or at least some of them) are supremely intelligent. The only reason this shortcoming was not immediately grasped is because these successive paradigms were adopted without ever being articulated as clear theories. Now, an immediate improvement over the previous paradigm is to stop believing that the programmer Gods are intelligent. Gods may guide the evolution of software, but their contribution to the process is hardly an overall intelligent coherent purpose; rather it is through a number of interventions based on partial knowledge, intuition, randomness, towards a progress that can be felt but not defined. Such is the theory of Teleological Evolution.
With the transition from intelligent guidance to unintelligent guidance, we are lead to the appearance of new tools, that roughly correspond to the Second Wave of Cybernetics. Genetic Algorithms, connectionist neural networks, probabilistically approximately correct learning methods allow to mine information from large databases without any explicitly designed representation of knowledge. Weakly structured computations allow to manipulate data despite limited understanding. At a smaller scale, programmers are satisfied with randomized algorithms that have good enough performance in practice despite having dreadful worst case guarantees. To protect from the devil, checksums and probabilistic proofs can be more useful than unattainable formal proofs. To synchronize multiple gods, user communities come to prominence, as users, though the least proficient, are those who possess the best distributed knowledge of what makes the software useful or not.
The paradigm of Teleological Evolution loosens the strictures of Design or Supernatural Selection, and opens the space for practical software solutions to problems beyond the full grasp of the programmers. While it reckons the importance of reasonable endeavor, this importance is de-emphasized; indeed, even reason can be seen as but a fast-track internal process of random production and selection inside the programmer's mind, as guided by his godly intuition. In the end, Teleological Evolution embraces an unfathomable mystical intuition as the ultimate divine source of creation.
TO BE CONTINUED...