Scrum has a direct and timely feedback loop built in for optimizing the software development process—the retrospective. Here are some of my thoughts about feedback and optimization of process in software development.

Development in large organizations

Many large organizations tend to have significant amounts of process around their development efforts—so much that sometimes, the development efforts slow to a crawl. By process I mean various rules and procedures where forms need to be filled out, steps need to be taken, checklists need to be followed, approvals need to be obtained, emails need to be sent, people need to be informed, tests need to be performed, etc. Speaking with the developers in such situations often reveals a similar story—they spend much of their time following complex procedures that don't add directly to development productivity or quality. It's not unusual to hear stories where it takes ten hours to complete one hour's development work due to nine hours of process overhead or delays. For example, one client told us a true story in which removing a single misplaced hyphen from a web page cost the organization nearly $100,000. The cost included time to schedule resources, develop project plans, consider and analyze the impact, do a risk assessment, do the actual development (delete the hyphen in a text editor), and then test, retest, and deploy the solution to a variety of staging systems, and finally to the production system. Despite everyone agreeing that the hyphen was a mistake, it took much more time to complete the process required than to delete the comma.

How do things get this way?

I suspect that this is partially related to hierarchical management—as managers become more separated from the developers, less communication occurs. And, when mistakes are made (which is nearly inevitable due to the complexity of modern software), management's response is often to create a new procedure or process to try to prevent future problems. This is completely well-intentioned, but by adding a bit of process each time something goes wrong, eventually process itself becomes part of the problem. And if the issue was a rare issue, then the process may consume thousands of hours in subsequent years to attempt to prevent something that would likely not happen again even without the process.

How to avoid this?

I think one way to avoid this type of problem is by using Scrum. In Scrum, you develop in short iterations (sprints), and at the end of each iteration you have a retrospective. In the retrospective, the dev team reviews what they liked and didn't like, and creates improvements. Often these improvements involve new process  or changes to process. So why is this any better than what was described above? In Scrum there is feedback from the dev team.

The importance of feedback

From my experiences with Scrum, the retrospectives will push the team in one of two directions—if they are lacking in process, then the team will create new process to assist it. But if the team has too much process, the team will recognize this and remove some of the less-effective processes. It's the feedback from the dev team every sprint (ie, every month or so) that keeps things from getting too far from an optimum state.

To make an analogy, the best way to drive a car down a freeway is to continually look out the windshield, see where you are, and based on that immediate, direct feedback, adjust course and speed. (This is analogous to the development team creating and removing process as they experience how everything is going.) On the other hand, imagine trying to drive a car blindfolded based on telephone instructions from someone who has driven down the same freeway a few years ago, and is watching your car from a nearby hill as you drive by. This is somewhat analogous to management creating process without being directly involved in the current design process. (I actually think this overstates the issues a bit, but it was the best analogy I could think of...) The feedback over the telephone may work, but the feedback is much less direct and timely than simply looking out the window and driving. To be effective, feedback needs to be timely and direct.

Consider the following hypothetical plot of the amount of productivity of a team as a function of amount of process. Just to be clear, it is not based on any measurements of productivity or process—I just made it up. It is just a general guess of what this curve might look like:

The sketch attempts to communicate the following points:

• If you have too little process (the far left side of the curve), your overall productivity declines—things become chaotic and error-prone, and less useful work gets done (and sometimes the work that does get done causes a lot of damage!) In the extreme, with no process (no planning, no guidance, no direction, etc), you don't get anything useful done because the work being done isn't the right work, so productivity is zero.
• If you have too much process (the right side of the curve), your overall productivity declines—the team spends too much of its time doing process, which often does not add directly to productivity, especially beyond a certain point. In the extreme of infinite process, no real work is ever done, so productivity is zero.
• For a given combination of project, team, situation, etc, there is an optimum amount of process where productivity is at a maximum—add more and productivity declines, but remove process and productivity also declines.

Given these assumptions, how do you optimize the amount of process? Feedback. The development team needs to be told what the priorities of the organization are, and then the team needs to be given the freedom to control the amount of process it uses. Usually management—especially if they are significantly removed from the development process—does not have as good an understanding as the team of what will really be useful process and what simply impedes development without adding much value.

To describe the feedback process in terms of the curve, when the team finds themselves in a chaotic and random environment, they will notice and, as part of the retrospective, create new process to assist in controlling the chaos. On the other hand, if the team notices that they are spending most of their time on process, they can remove the least-valuable of these processes and then see where they are. Eventually, after several sprints, the development team will hopefully be somewhere near the optimal point of the curve, and as the situation evolves from this point, they can continue to adjust to maintain the optimal position.

Some people might argue that the development team inherently doesn't like process and therefore will remove it regardless of whether it is useful or not. (It's only management that doesn't mind processes and therefore has the will to add them, because they will not be directly affected by them.) However, I've seen the opposite in many cases—for example, early in my current project, testing was very chaotic, because we had not defined much process around it. The team did not like this—it was chaotic and stressful for them. In the retrospectives, the team brought up that testing was chaotic and stressful, and the improvement they arrived at was more process. In addition, they defined the process—and it worked incredibly well. The process has since been fine-tuned and adjusted many times during subsequent retrospectives.

Will the team sometimes add too much process, or remove a valuable bit of process? Of course—but within a sprint or two, assuming there's a significant impact, they'll notice and correct. Constant correction via feedback is the key to optimizing the process. And Scrum includes the retrospective as a way of building feedback into the process at the most basic level.

One additional point that may not be obvious—in some cases, maximum productivity may not be the goal. For some organizations, reliability and stability or something else may be more important than pure productivity, but this is basically just a redefinition of what productivity is. (For most organizations reliability and stability are important, but so is the rate at which new features are added, so the definition of productivity needs to take balancing these various priorities into account.) The bottom line is that these priorities need to be communicated clearly to the development team—they are the ones who are steering the car down the freeway.