The Guru Myth

Anyone who has worked in software long enough has heard questions like this:

I'm getting exception XYZ. Do you know what the problem is?

The questioner didn't bother to include a stack trace, an error log, or any context leading to the problem. He or she seems to think you operate on a different plane, that solutions appear to you without analysis based on evidence. This person thinks you are a guru.

We expect such questions from those unfamiliar with software; to them systems can seem almost magical. What worries me is seeing this in the software community. Similar questions arise in program design, such as "I'm building inventory management. Should I use optimistic locking?" Ironically, the person asking the question is often better equipped to answer it than the question's recipient. The questioner presumably knows the context, knows the requirements, and can read about the advantages and disadvantages of different strategies. Yet this person expects an intelligent answer without supplying context. He or she expects magic.

It's time for the software industry to dispel this guru myth. "Gurus" are human; they apply logic and systematically analyze problems like the rest of us. Consider the best programmer you've ever met: At one point he or she knew less about software than you do now. If that person seems like a guru, it's because of years dedicated to learning and refining thought processes. A "guru" is simply a smart person with relentless curiosity.

Of course, there remains a huge variance in natural aptitude. Many hackers out there are smarter, more knowledgeable, and more productive than I may ever be. Even so, debunking the guru myth has a positive impact. For instance, when working with someone smarter than me I am sure to do the legwork, to provide enough context so that person can efficiently apply his or her skills. Removing the guru myth also means removing a perceived barrier to improvement. Instead of a barrier I see a continuum on which I can advance.

Finally, one of software's biggest obstacles is smart people who purposefully propagate the guru myth. This might be done out of ego, or as a strategy to increase one's value as perceived by a client or employer. Ironically this attitude makes a smart person less valuable, since they don't contribute to the growth of their peers. We don't need gurus. We need experts willing to develop other experts in their field. There is room for all of us.

Tearing Down the Software Factory

Tom DeMarco said it well:

The idea of a software factory is a joke -- that we can build software by rote -- that's ridiculous. If the work is deterministic, we will do with it what we do with any other big piece of deterministic work. We'll let the computer do the deterministic portion, leaving the person who interacts with the computer -- the other half of the system -- to do the work whose roteness has decreased, not increased. Every time you automate something, what's left of the person's work is less deterministic, until eventually, when you automate enough, there's no deterministic element left for the person's work--no rote. ... Our work is not deterministic. It's far too inventive. We're knowledge workers, not factory workers.

Of course, similar thoughts have been articulated many times before, and was a theme of the previous post on this blog. The idea of a software factory contradicts our best understanding of the essence of software, yet industrial style command-and-control management of software continues. Why is this? One problem is we developers haven't effectively presented a convincing alternative. Remove command and control, and to some extent developers must manage themselves. From Watts Humphrey:

Since your manager’s performance depends on your performance, and since the performance of software groups has historically been so poor, managers do not trust software professionals to manage themselves. To overcome this problem, all we have to do is to convince management that we can manage ourselves and then perform that self management so well that management will continue to trust us.

The theme of trust and credibility runs throughout Humphrey's extensive writing on this topic. This is not new, but progress has been slow. The major obstacle is that managers rightly want concrete, objective data on which to base their decisions. This conflicts with the black box that software development so often becomes. We need better transparency. It is time to open up the black box of software engineering.

The black box of software
Opening the black box means programmers and managers must meet each other halfway. Managers must create and adapt to a new post-industrial management science, and programmers must produce data useful to that management science. This does not mean attempting to make programmers into assembly line workers. To the contrary, it means embracing the creative nature of software, and managing the output as a side effect of development.

How do we do this? Empirically manage everything that can be empirically managed, and complement it with the judgment of your best engineers. Many pieces of the puzzle already exist. Unit testing, code coverage reports, bug tracking, static code analysis, dependency management and others provide transparency into the state of a project. Such data is purely informational, but technically inclined managers can and should use it to ensure a project is on track. With context, problems like bloated dependencies, poor test coverage, or fixing related bugs many times are all signs of a project going astray. Modern software organizations must be able to detect and correct problems before they grow.

Unfortunately many managers today are not equipped to work with such data. This must change. Managers must build their skill sets for the post-industrial world.

Trust through transparency
Of course tools like code coverage and defect tracking only tell part of the story. Code is design, and no set of tools can define the quality or progress of design. Therefore we must complement these tools with the best judgment of our best engineers. But if managers don't trust the best engineers, this judgment is wasted.

So how do we solve this? Use transparency into software as a tool for building trust. Concrete data on the progress and quality of software gives managers greater confidence in engineers, even if the picture is incomplete. Trust begins to grow. Engineers should qualify empirical data and use it appropriately as a basis for design decisions. If we can provide managers a glimpse into software and prove we are making progress, they will be more willing to accept our opinions.

Some tension between managers and developers may be inevitable, but we can meet each other halfway. Our development practices should yield hard data for everything appropriate. In exchange managers must accept that code is design and trust the judgment of developers.

What we can learn from databases

While not perfect, relational databases are among the most successful software ever used. Their behavior is easily understood even under high concurrency. Sets of changes can be composed with almost no effort, with strong guarantees of consistency. The question is: would this be true if we built our databases the same way we build applications?

Transactions and read consistency would be the first to go. Users must then cooperatively synchronize on some external monitor; any failure to do so can result in invalid data. Next, system state and logic become intermixed. Gone are simple ways to inspect the state of the system, or create a well-defined state space with known transitions. In short, if databases were written like applications, they become vulnerable to all of the bugs we see in applications.

Suppose we turn the question on its head, and ask what characteristics of the database we can apply to the rest of software? Consider the following advantages:

• The system itself guarantees a consistent view of data under concurrency, eliminating many types of race conditions
• State changes are composable; updates are committed atomically, ensuring the database is never in an invalid state.
  
• The state space is understandable. Because updates are composed to an atomic state change, countless permutations of state are eliminated

The key is this: we can confidently inspect, reason about, and change the state space of a database. The complexity of large models is mitigated by allowing only for valid, composable transitions. Imagine if we had such guarantees when building software in general. We could understand the state of an application at any time, and ensure all changes are valid and consistent. Our system would be much more understandable and predictable.

In fact, much language research is focused on this area. Software Transactional Memory in languages like Haskell is the most visible. The question is how such progress will reach the mainstream. History suggests an evolutionary model. Languages that gain adoption tend to have a good deal in common with an established language, lowering the barrier to entry. Because of this, I have yet to see a language with the above characteristics that I think will achieve widespread adoption. Hopefully that will change.

Understandable systems today
A couple of recent posts point out the burden of large code bases. I agree, as suggested by the title of this blog, but it's easy to confuse a symptom with the problem itself. So I phrase it differently: Unmanageable complexity is the enemy. Code size is often what the enemy smells like.

Developers can better manage complexity even without guarantees similar to what a database offers. Code should have a clear, easily understood state space, preferably applying related changes atomically. Such systems are easier to reason about and change because developers need not concern themselves with side effects of unrelated code; they can focus on the problem at hand. For those who haven't explored this, I'm indirectly describing the functional style of programming. This is the great hope for pure functional programming: it may spread predictability and a simple model to all development.

The Design Crisis

Imagine waking up tomorrow and learning the construction industry has made the breakthrough of the century. Millions of cheap, incredibly fast robots can now fabricate materials out of thin air, have a near-zero power cost, and can repair themselves. And it gets better: given an unambiguous blueprint for a construction project, the robots can build it without human intervention, all at negligible cost.

One can imagine the impact to the construction industry, but what would happen upstream? How would the behavior of architects and designers change if construction costs are negligible? Today, physical and computer models are built and rigorously tested before investing in construction. Would we bother if the construction was essentially free? If a design collapses, no big deal...just find out what went wrong and have our magical robots build another one. More implications follow. With models obsolete, unfinished designs evolve by repeatedly building and improving upon an approximation of the end goal. A casual observer may have trouble distinguishing an unfinished design from a finished product.

Our ability to predict time lines will fade away as well. Construction costs are more easily calculated than design costs -- we know the approximate cost to install a girder, and how many girders we need. As predictable tasks shrink toward zero, the less predictable design time grows to dominate the project. Results are produced more quickly than before, but reliable time lines slip away.

Of course, the pressures of a competitive economy still apply. With construction costs eliminated, a company that can quickly complete a design gains a market edge. Pressure on getting design done fast becomes the central push of engineering firms. Inevitably, someone not deeply familiar with the design will see an unvalidated version, see the market advantage of releasing early, and say "This looks good enough."

Some life-or-death projects will be more diligent, but in many cases consumers learn to suffer through the incomplete design. Companies can always send out our magic robots to "patch" the broken buildings and vehicles they sell. All of this points to an amazing, counter-intuitive conclusion: our sole premise was a dramatic reduction in construction costs, and the result is quality got worse.

The Design Crisis
It shouldn't surprise us the above story has played out in software. If we accept that code is design -- a creative process rather than a mechanical one -- the software crisis is explained. Now we have a design crisis: the demand for quality, validated designs far exceeds our capacity to create them. The pressure to use incomplete design is strong.

Fortunately, this model also gives us some clues on how we can get better. Physical simulations equate to automated testing; software design isn't complete until it is validated with a brutal battery of such tests. To make such tests more effective we are finding ways to rein in the huge state space of large systems. Improved languages and design practices give us hope. Most importantly, there is one inescapable fact. Great designs are produced by great designers dedicating themselves to the mastery of their craft. Code is no different.

von Neumann's Long Farewell

A big step in solving the software crisis is to accept that programming is not about computers. Yes, it used to be, and still is for brave souls working on operating systems or compilers. But in the long list of the world's software struggles, compilers rank pretty low.

It's easy to think programming is about computers because that's where we see problems. Of course, these are only symptoms of human error. Developers are constantly stretching to express behavior in terms friendly to a computer, making life harder so we can be easier on our machines. Such problems have been pointed out decades ago by the likes of Dijkstra, Backus [pdf], and Abelson and Sussman. Surely there is a way to describe our systems better than the terms of the von Neumann architecture.

For years we were limited by the capabilities of the machines, but this limitation has vanished for most of us. We've entered a new era, where improving the way we build software is as much a social problem as a technical one. Fortunately, there are signs of a growing social bedrock that could finally help us.

The Thirst for Abstraction
For a long time I didn't understand Guy Steele's comments on how Java can pull C++ developers about halfway to Lisp. Surely that fraction must be off; writing code in Java certainly feels a lot closer to C++ than Lisp. But independent of the Java language, the platform did a major service for the industry: it proved to the masses that we no longer need to express our designs directly in terms of the von Neumann architecture.

The abstraction offered by Java-like languages is small but significant. The conventional wisdom in software has long held that code must be expressed in terms of the machine to achieve passable performance. This myth is gone. Nearly every major software company is adopting some form of garbage collected, virtual machine-based language. We rely on the platform to map our higher-level expressions to efficient machine code.

This is a big step because accepting this level of abstraction opens the door to others. Why should we explicitly state data type information, when it can be inferred at compile or run time? Why not abandon primitives and arrays for objects and lists, since our platforms can now optimize away unneeded allocation? Why struggle with keeping track of many small state changes when major operations could change state atomically? Such abstractions make life easier for us, and previous successes suggest it will work. The result is a self-perpetuating thirst for abstraction. Expressing designs so humans can easily reason about and manipulate them is addicting.

von Neumann's Long Farewell
For many of us the shift away from the von Neumann architecture is painfully slow. Unfortunately the best ideas are often slow to be adopted. People are used to the way they've worked before, or averse to risk. So our transition toward better software is incremental -- but it is happening. Mainstream languages are offering higher-level concepts in their latest revisions. Adding concepts like closures, type inference, and applicative libraries does the software industry a service. It moves us toward a better way to write code. Hopefully some day we'll realize the low-level complexity we've dealt with isn't really necessary and abandon it altogether.

Why are we so polarized about Java?

It seems impossible to constructively praise or criticize Java without eliciting strong reactions. It comes from both sides; some seem to love to hate Java, and parts of the Java community become defensive as a reaction.

For example, Paul Buchheit recently posted some insights into API design, referring to poor examples that happen to be written in Java. He immediately states he doesn't hate Java. The post is about APIs, not the language. Even so, he still had to update his post saying "It's not about Java, really", presumably because of some polarized reactions. This is unfortunate because it distracts us from Paul's excellent commentary.

I suspect the polarization starts from a trap I have fallen into myself. Like many involved in language discussions, I use Java professionally and other languages at night, so I run into examples analogous to the one described by Paul Buchheit. I sometimes get frustrated with poor APIs, and also with the Java language -- either directly or by association with poor APIs. I wonder, couldn't it use powerful constructs I use in other languages, or could the APIs just plain be simpler?

The problem is my frustration may linger when I discuss programming languages, and (knowingly or unknowingly) be reflected in my tone when discussing Java. The resulting post is more acidic, and less level handed than I would like. Those who disagree with my arguments are likely to pick up on that tone and may respond in kind. Before long we've spiraled into isolated camps, and meaningful discussion is hard to achieve. My last post had signs of this.

So, where did I go wrong? When discussing constructive criticism of Java and its APIs, we need to view Java with some perspective. We need to remember its history. So my first point is:

We are critical of Java in a context different from the one in which it was designed.

A key goal for Java was to replace C++, and this goal led to design decisions that are often targets for criticism today. In 1996 it wasn't clear we could create a sufficiently fast language without primitive types and arrays. It wasn't clear how much boilerplate code would be required by anonymous callback classes or checked exceptions. It wasn't clear that since the new syntax exactly specifies implementation, factory patterns would proliferate, and so on. So my second point:

Criticisms of Java should not imply any negatives about the designers of Java. Critics of Java are likely working in a problem space that differs from Java's original design goals, or incorporating new knowledge from the past decade.

There are certainly some that disagreed with the above decisions in 1996, but it was surely more debatable then, and different decisions may have slowed Java's proliferation -- which was opposed to its original design goals.

The key is we need to keep this background in mind whenever discussing Java. We can criticize Java and its APIs without insulting anyone. When being critical of Java, any frustrations should be tempered by knowing the landscape has changed. Similarly, apologists should recognize that criticism need not be viewed as an attack; it might be an attempt to make things better.

Why Java lost its mojo, and what Sun is doing about it

It's old news to some, and hotly debated by others, but Java has lost its mojo. The Java community still has growth from large companies, but it hasn't been the darling language of hackers for a while (if it ever was). But don't think Sun isn't doing anything about it.

Java's popularity is a function of market position, the perception of one product compared to its competitors. When it first came out, Java was positioned against C++, to which it compares favorably. Yes, I miss operator overloading and a handful of other items, but these are outweighed by the fact that Java prevents entire classes of errors made by C++ developers. All of this was done before, but Sun was able to make it mainstream and practicable.

Now the Java vs. C++ battle is effectively over. Today Java's market position is measured against a variety of languages, most frequently C#, Python, and Ruby. For many of us, Java compares much less favorably to these than to C++. Proponents of these languages love to show problems solved in a handful of lines, with a Java-based solution including dozens of lines of boilerplate code. The poor Java developer is then pushed around and has his lunch money stolen.

Naturally, much of the Java community wants to do something about this. So closures and type inference have been proposed for Java 7, and the debate in the Java community begins. Some say "Hooray, Python programmers will stop making fun of me!" Others object "Don't we already have enough constructs to worry about in this language?" Throw in the glacier-like JCP, and we have no idea what the next release of Java will look like.

Can we fix Java?
I am sympathetic to both arguments. The language is getting big. If only we could get rid of artifacts like primitive types and arrays, we could remove clutter and make conceptual room for more advanced concepts. Even so, I still land firmly in the pro-closure camp. This is partially for selfish reasons; I got hooked on them while hacking in functional languages at night, and want to use them in my day job. My company uses Java because it is actually the right choice for us. No other platform offers that combination of portability, reliability, tooling, and performance. Too bad it's not with a better language.

With this in mind, the rumblings of the last few years make sense, and the future is clear. It's not about Java at all, it's about the JVM. And Sun is positioning the JVM to become a light, efficient virtual machine available everywhere. Consider everything that has happened:

• JavaFX has been announced, creating a simple, declarative model to compete with the likes of Flex.
• Sun has committed to releasing an update to Java 6 that will shrink the minimal JRE download to two megabytes.
• Sun hired Charles Nutter to make JRuby a first-class language for the JVM. The 1.0 release nears.
  
• Glassifish -- the open-source application server sponsored by Sun -- will run JRuby applications directly in v3, and be pluggable for new hosts. Java SE and EE need never enter the picture.
• The JVM is now available under the GPL, which should increase availability on Linux.
• The 2007 JavaOne conference included a Scala session for the first time

What does this tell us? The JRE is becoming the new POSIX. And the Java language is becoming less and less relevant. (Can't find where I first saw this idea posted...please send link if you have it.)

But what about the mean time?
The reality is Java-the-language will be around for some time to come, at least until a clear successor is established. Sun and other members of the JCP have an opportunity to define the language's legacy. As Guy Steele famously said, Java "managed to drag a lot of them [C++ programmers] about halfway to Lisp." Now Java can drag programmers one step closer to Lisp, and set the stage for the next generation of languages.

Of course, I'm referring to closures. Users of functional languages will attest how working with such constructs changes how you think. By adding closures to Java, we do more than provide a powerful tool for those of us already familiar with the construct. We introduce a large community to a better way of solving many problems. We'll see if it sticks, but I can't think of a better way to spread a thought model many believe to be superior.

Update: I actually don't think Java sucks. It meets the early goal as a straightforward replacement for C++ very well. I do believe we can create a language that is better than Java based on what we've learned over the last decade.