Tearing Down the Software Factory

Tom DeMarco said it well:

The idea of a software factory is a joke -- that we can build software by rote -- that's ridiculous. If the work is deterministic, we will do with it what we do with any other big piece of deterministic work. We'll let the computer do the deterministic portion, leaving the person who interacts with the computer -- the other half of the system -- to do the work whose roteness has decreased, not increased. Every time you automate something, what's left of the person's work is less deterministic, until eventually, when you automate enough, there's no deterministic element left for the person's work--no rote. ... Our work is not deterministic. It's far too inventive. We're knowledge workers, not factory workers.

Of course, similar thoughts have been articulated many times before, and was a theme of the previous post on this blog. The idea of a software factory contradicts our best understanding of the essence of software, yet industrial style command-and-control management of software continues. Why is this? One problem is we developers haven't effectively presented a convincing alternative. Remove command and control, and to some extent developers must manage themselves. From Watts Humphrey:

Since your manager’s performance depends on your performance, and since the performance of software groups has historically been so poor, managers do not trust software professionals to manage themselves. To overcome this problem, all we have to do is to convince management that we can manage ourselves and then perform that self management so well that management will continue to trust us.

The theme of trust and credibility runs throughout Humphrey's extensive writing on this topic. This is not new, but progress has been slow. The major obstacle is that managers rightly want concrete, objective data on which to base their decisions. This conflicts with the black box that software development so often becomes. We need better transparency. It is time to open up the black box of software engineering.

The black box of software
Opening the black box means programmers and managers must meet each other halfway. Managers must create and adapt to a new post-industrial management science, and programmers must produce data useful to that management science. This does not mean attempting to make programmers into assembly line workers. To the contrary, it means embracing the creative nature of software, and managing the output as a side effect of development.

How do we do this? Empirically manage everything that can be empirically managed, and complement it with the judgment of your best engineers. Many pieces of the puzzle already exist. Unit testing, code coverage reports, bug tracking, static code analysis, dependency management and others provide transparency into the state of a project. Such data is purely informational, but technically inclined managers can and should use it to ensure a project is on track. With context, problems like bloated dependencies, poor test coverage, or fixing related bugs many times are all signs of a project going astray. Modern software organizations must be able to detect and correct problems before they grow.

Unfortunately many managers today are not equipped to work with such data. This must change. Managers must build their skill sets for the post-industrial world.

Trust through transparency
Of course tools like code coverage and defect tracking only tell part of the story. Code is design, and no set of tools can define the quality or progress of design. Therefore we must complement these tools with the best judgment of our best engineers. But if managers don't trust the best engineers, this judgment is wasted.

So how do we solve this? Use transparency into software as a tool for building trust. Concrete data on the progress and quality of software gives managers greater confidence in engineers, even if the picture is incomplete. Trust begins to grow. Engineers should qualify empirical data and use it appropriately as a basis for design decisions. If we can provide managers a glimpse into software and prove we are making progress, they will be more willing to accept our opinions.

Some tension between managers and developers may be inevitable, but we can meet each other halfway. Our development practices should yield hard data for everything appropriate. In exchange managers must accept that code is design and trust the judgment of developers.