Start by embracing your limits

Nothing in human history has offered more promise but has seen as many failures as software. We’ve all seen moments of greatness, where a program seems like magic -- but such gems are surrounded by minefields of bugs and indecipherable interfaces.

The result of all this is we programmers are often a frustrated bunch. But should we be? After all, what makes us think that as a species we should have the aptitude to create great software? Our skill sets evolved in an environment that favored those who could hunt boar and find berries -- any capacity to succeed in the abstract world of software is pure, accidental side effect. Perhaps we should be awed by software’s successes rather than frustrated by its failures.

The good news is we can improve despite our limitations, and it starts with this: accept that we generally have no innate ability to create great systems, and design our practices around that. It seems like every major step forward in software has followed this pattern of embracing our limitations. For instance, we move to iterative development since we can’t anticipate all variables of a project. We aggressively unit test because we realize we’re prone to error. Libraries derived from practical experience frequently replace those built by expert groups. The list goes on.

This type of admission is humbling, but it can also be liberating. Here’s an example: In years past I would spend hours agonizing over an internal design decision for a system I was building. I figured if I got it right we could easily bolt on some new feature. Sometimes I was right, but often times I was not. My code often was littered with unnecessary structure that only made things more complicated.

Contrast that to today: I know I can’t anticipate future needs in most cases, so I just apply this simple heuristic:

1. When in doubt, do the simplest thing possible to solve the problem at hand
2. Plan on refactoring later.

The first step frees us from trying to anticipate all future needs -- but this is not quick and dirty cowboy coding. An essential element of code is to create an understandable and maintainable system. Don't try to code for future needs. Instead, structure code for present needs so it can be leveraged in the future.

So how do we do this? A couple things to keep in mind:

• When in doubt, leave it out. (also known as “You Ain’t Gonna Need It”)
• Unit-Testable designs tend to be reusable designs. Unit tests not only catch bugs that can result from refactoring, but they encourage modularity to enable that refactoring.
• Don’t try to design an API independently of an application. You won’t understand your users’ needs well enough to create a good experience. Build the API as part of the application to make sure its needs are met, then factor out and generalize.
• Group code together that tends to change for similar reasons. If your Widget class is responsible for rendering its UI and writing to the database and business logic, you can’t use it anywhere else without significant changes. High cohesion and loose coupling.

There are no hard-and-fast rules to building software, but we all need a compass to help guide us through the thousands of micro-decisions we make every time we write code. Hopefully this post can help build that compass.

Can "Agile in the Large" Succeed?

Agile Development isn’t perfect, but it got something right. Right enough, at least, to gain enough momentum to become a buzzword -- and for consultants to latch on by selling “Agile” to big enterprises. The result is “Agile in the Large”.

Agile in the Large ranges from the Half-Arsed Agile Manifesto to Cargo Cult practices. Too often we simply bolt on a few agile practices without understanding how they provide value. Adding scrums and iteration reviews to your current process and expecting improvement is like building a runway in your back yard and expecting planes to land.

Whether it’s labelled agile or not, a cohesive development team works for a couple reasons. It creates a focused team fully aligned to a common goal. It avoids “Us and Them” mentalities, enabling everyone to adapt to meet the goal. It offers a self-correcting strategy for the thousands of micro-decisions when building software: It must be testable, “You Ain’t Gonna Need It”, and it creates quick feedback to make sure you’re solving the actual problem.

Agile in the Large is flawed because it often fails to achieve these basic ingredients. Take Scrum of Scrums, for example. It tries to coordinate and resolve technical dependencies between teams by having representatives from each team get together and talk through them. It’s better than nothing, but an emerging system has too much ambiguity: the interaction between components is still being defined, bugs can lead to finger pointing, and the important sense of ownership is lost. Everyone feels like cogs in a machine rather than someone solving an important problem for a real user. Our ability to adapt the microdecisions that build great software is lost.

In fact, Agile in the Large seems doomed from the beginning. Once you’re beyond some number of people in a single project, it’s impossible to create that sense of shared ownership and adaptability. Great software is created by small, dedicated teams.

So, what then? It seems the only way out is to make Agile in the Large more like Agile in the Small. A team should be working on one and only one project, and include everything necessary for that project to be successful. How we do that depends on where we are in a project’s maturity curve.

New vs. mature systems
New development is the most easily handled. Look at the problems at hand, and make sure the team is equipped to handle them end-to-end. This new project may have several components, but now is not the time to split those components into their own teams; keep a single team aligned to the user’s goals. After all, if a component isn’t aligned to some user’s goal, what is its value? Organize around components too early, and team cohesion is lost.

So we are off to a good start, but we need to adjust our strategy as a successful project matures. You may find parts of your system has value for other uses. This seems timeless: C first was a successful tool for building Unix, Ruby on Rails was an application before it was a framework, Amazon was an online book store before it was a cloud platform, and so on. In all of these cases a successful system was built, and then groups arise around the reusable pieces. Reusable technology will naturally arise from a successful project. Embrace that, but don’t force it.

Open source as a model for reuse
Later in a system’s life we find ourselves consuming many assets from a variety of teams. Now it’s easy to let coordination and communication friction kill our focused project. Fortunately, the open source world gives us the answer.

Quite simply, needed enhancements to common assets should be written and contributed by consuming teams. This offers several advantages over logging an enhancement request. For instance, we reduce deadline haggling and misaligned schedules. We also reduce the need for frequent status updates and opportunities for miscommunication between teams.

Of course, not all changes to open source projects come in the form of patches. There should still be a team around the asset in question, responsible for its architecture and fundamental direction. That team also operates as a gatekeeper, ensuring all patches are properly unit tested, documented, and don’t threaten the integrity of the system.

Changes must be handled on a case-by-case basis, but the primary mode of operation should follow the agile ideal: a single team with responsibility for a project end-to-end. This includes contributing to assets it consumes.

What if even the initial project is too big?
I’ll touch on one final question: what if even the initial scope of a project is beyond what a single team can accomplish? Find a smaller initial scope, get it working, and grow from there. Do this even if the initial scope means building placeholders that are discarded in the final result. The time saved and friction eliminated by creating a single, focused team will outweigh this. Think of it as scaffolding that eventually gets removed.

In the end, Agile in the Large only works if we make it more like Agile in the Small. Hopefully this article is a step in that direction.

Beware of the Flying Car

Every so often we developers are asked to build a flying car. Our users obviously need one to avoid traffic and get to meetings on time. So we set down the path to meet those needs.

The trouble is most of us have no idea how to build a flying car. Even so, this is the project so we better get started! Our first release of the flying car will be small, and carried around on a string. We show good progress to our user's needs and gain approval from our superiors. We will simply remove the string in a later release.

Hopefully most of us will recognize a Flying Car Project and take time to understand and address the users’ goals rather than sprinting toward a brick wall. The Flying Car is a means, not an end in itself -- and there are other, executable means to the desired end.

Of course, there will always be someone eager to go build that flying car. If that happens, try to steer them in the right direction. If all else fails, just don’t be standing underneath it when they cut the string.

And We're Back

After an unbelievably long hiatus, I'm going to start blogging again. It's funny how becoming a parent makes everything else seem to go away for a while.

I don't expect I will ever post at a regular intervals here. I'll post when I feel like I can express something that gets closer to some truth about software -- at least to me. How often will that happen? Who knows?

The rebirth of this blog will come with a shift in material, at least for the near future. I've recently become more interested in the social aspect of building software. How should we organize ourselves to create great software? How should that change over time?

I am and always will be a programmer at heart. My shift in emphasis simply comes from the realization that our biggest challenges aren't technical. They're social.

The Guru Myth

Anyone who has worked in software long enough has heard questions like this:

I'm getting exception XYZ. Do you know what the problem is?

The questioner didn't bother to include a stack trace, an error log, or any context leading to the problem. He or she seems to think you operate on a different plane, that solutions appear to you without analysis based on evidence. This person thinks you are a guru.

We expect such questions from those unfamiliar with software; to them systems can seem almost magical. What worries me is seeing this in the software community. Similar questions arise in program design, such as "I'm building inventory management. Should I use optimistic locking?" Ironically, the person asking the question is often better equipped to answer it than the question's recipient. The questioner presumably knows the context, knows the requirements, and can read about the advantages and disadvantages of different strategies. Yet this person expects an intelligent answer without supplying context. He or she expects magic.

It's time for the software industry to dispel this guru myth. "Gurus" are human; they apply logic and systematically analyze problems like the rest of us. Consider the best programmer you've ever met: At one point he or she knew less about software than you do now. If that person seems like a guru, it's because of years dedicated to learning and refining thought processes. A "guru" is simply a smart person with relentless curiosity.

Of course, there remains a huge variance in natural aptitude. Many hackers out there are smarter, more knowledgeable, and more productive than I may ever be. Even so, debunking the guru myth has a positive impact. For instance, when working with someone smarter than me I am sure to do the legwork, to provide enough context so that person can efficiently apply his or her skills. Removing the guru myth also means removing a perceived barrier to improvement. Instead of a barrier I see a continuum on which I can advance.

Finally, one of software's biggest obstacles is smart people who purposefully propagate the guru myth. This might be done out of ego, or as a strategy to increase one's value as perceived by a client or employer. Ironically this attitude makes a smart person less valuable, since they don't contribute to the growth of their peers. We don't need gurus. We need experts willing to develop other experts in their field. There is room for all of us.

Tearing Down the Software Factory

Tom DeMarco said it well:

The idea of a software factory is a joke -- that we can build software by rote -- that's ridiculous. If the work is deterministic, we will do with it what we do with any other big piece of deterministic work. We'll let the computer do the deterministic portion, leaving the person who interacts with the computer -- the other half of the system -- to do the work whose roteness has decreased, not increased. Every time you automate something, what's left of the person's work is less deterministic, until eventually, when you automate enough, there's no deterministic element left for the person's work--no rote. ... Our work is not deterministic. It's far too inventive. We're knowledge workers, not factory workers.

Of course, similar thoughts have been articulated many times before, and was a theme of the previous post on this blog. The idea of a software factory contradicts our best understanding of the essence of software, yet industrial style command-and-control management of software continues. Why is this? One problem is we developers haven't effectively presented a convincing alternative. Remove command and control, and to some extent developers must manage themselves. From Watts Humphrey:

Since your manager’s performance depends on your performance, and since the performance of software groups has historically been so poor, managers do not trust software professionals to manage themselves. To overcome this problem, all we have to do is to convince management that we can manage ourselves and then perform that self management so well that management will continue to trust us.

The theme of trust and credibility runs throughout Humphrey's extensive writing on this topic. This is not new, but progress has been slow. The major obstacle is that managers rightly want concrete, objective data on which to base their decisions. This conflicts with the black box that software development so often becomes. We need better transparency. It is time to open up the black box of software engineering.

The black box of software
Opening the black box means programmers and managers must meet each other halfway. Managers must create and adapt to a new post-industrial management science, and programmers must produce data useful to that management science. This does not mean attempting to make programmers into assembly line workers. To the contrary, it means embracing the creative nature of software, and managing the output as a side effect of development.

How do we do this? Empirically manage everything that can be empirically managed, and complement it with the judgment of your best engineers. Many pieces of the puzzle already exist. Unit testing, code coverage reports, bug tracking, static code analysis, dependency management and others provide transparency into the state of a project. Such data is purely informational, but technically inclined managers can and should use it to ensure a project is on track. With context, problems like bloated dependencies, poor test coverage, or fixing related bugs many times are all signs of a project going astray. Modern software organizations must be able to detect and correct problems before they grow.

Unfortunately many managers today are not equipped to work with such data. This must change. Managers must build their skill sets for the post-industrial world.

Trust through transparency
Of course tools like code coverage and defect tracking only tell part of the story. Code is design, and no set of tools can define the quality or progress of design. Therefore we must complement these tools with the best judgment of our best engineers. But if managers don't trust the best engineers, this judgment is wasted.

So how do we solve this? Use transparency into software as a tool for building trust. Concrete data on the progress and quality of software gives managers greater confidence in engineers, even if the picture is incomplete. Trust begins to grow. Engineers should qualify empirical data and use it appropriately as a basis for design decisions. If we can provide managers a glimpse into software and prove we are making progress, they will be more willing to accept our opinions.

Some tension between managers and developers may be inevitable, but we can meet each other halfway. Our development practices should yield hard data for everything appropriate. In exchange managers must accept that code is design and trust the judgment of developers.

What we can learn from databases

While not perfect, relational databases are among the most successful software ever used. Their behavior is easily understood even under high concurrency. Sets of changes can be composed with almost no effort, with strong guarantees of consistency. The question is: would this be true if we built our databases the same way we build applications?

Transactions and read consistency would be the first to go. Users must then cooperatively synchronize on some external monitor; any failure to do so can result in invalid data. Next, system state and logic become intermixed. Gone are simple ways to inspect the state of the system, or create a well-defined state space with known transitions. In short, if databases were written like applications, they become vulnerable to all of the bugs we see in applications.

Suppose we turn the question on its head, and ask what characteristics of the database we can apply to the rest of software? Consider the following advantages:

• The system itself guarantees a consistent view of data under concurrency, eliminating many types of race conditions
• State changes are composable; updates are committed atomically, ensuring the database is never in an invalid state.
  
• The state space is understandable. Because updates are composed to an atomic state change, countless permutations of state are eliminated

The key is this: we can confidently inspect, reason about, and change the state space of a database. The complexity of large models is mitigated by allowing only for valid, composable transitions. Imagine if we had such guarantees when building software in general. We could understand the state of an application at any time, and ensure all changes are valid and consistent. Our system would be much more understandable and predictable.

In fact, much language research is focused on this area. Software Transactional Memory in languages like Haskell is the most visible. The question is how such progress will reach the mainstream. History suggests an evolutionary model. Languages that gain adoption tend to have a good deal in common with an established language, lowering the barrier to entry. Because of this, I have yet to see a language with the above characteristics that I think will achieve widespread adoption. Hopefully that will change.

Understandable systems today
A couple of recent posts point out the burden of large code bases. I agree, as suggested by the title of this blog, but it's easy to confuse a symptom with the problem itself. So I phrase it differently: Unmanageable complexity is the enemy. Code size is often what the enemy smells like.

Developers can better manage complexity even without guarantees similar to what a database offers. Code should have a clear, easily understood state space, preferably applying related changes atomically. Such systems are easier to reason about and change because developers need not concern themselves with side effects of unrelated code; they can focus on the problem at hand. For those who haven't explored this, I'm indirectly describing the functional style of programming. This is the great hope for pure functional programming: it may spread predictability and a simple model to all development.