Software has held more promise and yet met with more failure than any other technology. Meanwhile the hardware guys have been merrily creating more and more CPU cycles for us to swamp.
A lot has been written about why software sucks. Fred Brooks famously predicted there would be no silver bullet improvement in software productive over the next ten years. That was twenty years ago, but there are signs that the dark age of software might be slowly and quietly coming to an end.
Brooks was right about the lack of a silver bullet, of course. But since then we've learned that we've been trying to kill the wrong monster. Building a program was compared to the reliable timeline and quality of building a bridge or skyscraper, and we were mystified on how to get there. This is the wrong monster; instead we need to view code as design and adopt our practices to match. Coding is a creative process with much more in common with design than with any industrial process -- Jack Reeves got it right.
Notice successful development methodologies jibe with viewing code as design. We unit test because a design isn't complete until validated with a simulation. We work in iterations because a change in one part of the design affects others. We document in the source code itself because that is the definitive source of design.
The code-is-design perspective also predicts many supposed mysteries: huge productivity variance, a high number of defects, and failures of rote process are all expected in unvalidated, untested design.
Mysteries and Problems
Fortunately we've taken an important step: software development was once a mystery in search of a silver bullet; now it is a problem with real progress. Noam Chomsky pointed out the first step to understanding the natural world is to turn mysteries into problems. We're seeing the same with software engineering.
Given the premise of code being design, here are some steps to take us in the right direction:
1. Boil out the remaining accidents. We can still have race conditions or leak resources in many programming languages. Newer languages should make these mistakes impossible. We can also learn from existing languages. For instance, the messaging model in Erlang prevents many types of timing problems. Functional languages may eliminate the many types of bugs which arise from side effects. The essence of a design should not be concerned with timing or unexpected side effects, so whenever possible this should be solved in the language itself.
2. Make validation of designs easy. How about the following requirement for a new programming language:
It must be straightforward to write a unit test to reproduce any possible bug.
Since designs must be validated with computer simulation, we must make those simulations easy. Huge strides have been made with various unit testing frameworks, but there are still shortcomings. Proofs of correctness are impractical for most systems, but the ability to easily assert a system doesn't have a particular bug is the next best thing. Furthermore, any bug discovered at any time should be added to the test suite to make sure it never occurs again. This ensures the defect rate of a system trends towards zero.
3. Design and code should merge into a single artifact. We still need design at a higher level than current programming languages, but this design needs to be an asset rather than a burden that gets out of synch with the code.
We are already seeing this: documentation has been merged with source code and tools convert code to UML diagrams and vice versa. But there is a lot of room for improvement. Code in object-oriented languages often includes many unessential objects that are just noise from a high-level view. These should be eliminated from the language, or at least filtered from the high-level design.
There are surely many other examples that I'm overlooking, and functional languages do hold promise. What's important is we now have an idea of why software sucks, and an inkling of what the solution might be.