Ad Hoc to Agile, Scrum and Scaling

A colleague recently asked whether a team he encountered was agile.

• The team is managed by a smart computer scientist with deep architectural understanding.
• There are two specialist engineers on the team: one with database knowledge, and the other with Javascript/CSS/HTML.
• Periodically team meets with the manager and product manager to plan. Together they decide the overall functionality to be delivered by the team, and decide how long they need, typically two to four weeks, which they call “sprints”. They select functionality that they believe they can finish in that time. These are not user stories, but a collection of requirements that seem like they should go together. The work is not directly releasable.
• Since the manager is smart, the team usually defers to the manager’s architectural approach, which consciously considers team member skills. The team decomposes the requirements into tasks, carefully constructed to reduce the need for member interaction. They try to minimize source code clashes in advance, assigning tasks to people carefully.
• The team members typically work in isolation. Although they do not have daily stand ups, they are co-located, so they can always ask co-workers for advice and help. Each is responsible for the tasks he or she was assigned.

Some team members want to use Scrum, but the manager has said, “No. Scrum involves too much process, too many meetings, team members will feel micromanaged.”

I have seen ad hoc sprints implemented in other companies, often with Scrum names applied to dysfunctional activities: Standups that last an hour and occur weekly, Sprints that have fixed length, but where the manager assigns tasks, Sprint Plans where product owners add new functionality mid-Sprint. Etc.

Let’s tackle the simple question: If we are producing new functionality in short cycles with an ad hoc process, are we agile?

Sprints Alone Don’t Create Agility

A process with short software development cycles, to be sure, provide many advantages over the long-cycle time of waterfall. A good product manager can uses a short-cycle process to experiment with the target market, determining which features drive usage, finding good price points, and discovering new markets [Ries, Lean Startup, 2011].

This process also adhere to the Agile Manifesto, see http://agilemanifesto.org/, so it has an “agile spirit.” But it doesn’t adhere to the “Principles Behind the Agile Manifesto,” see http://agilemanifesto.org/principles.html. Specifically, these:

• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• Business people and developers must work together daily throughout the project.
• Simplicity—the art of maximizing the amount of work not done—is essential.

What dysfunctions arise from violating these principles?

Specialists who don’t teach their skills to others greatly inhibit sustainable development. Agilists believe that sustainable development includes planning for the possible temporary loss (vacation, health issue, new baby, etc.) or permanent loss (moving, leaving the job, etc.) of a team member. By assigning work to individuals, in this case, the most skilled for the tasks, we virtually guarantee that individuals will become specialists in the code base, thus incurring huge risks if one of them is temporarily or permanently gone. Even if team members are always there, we rarely have a situation where the most valuable features require skills that perfectly match those available. Because motivated generalists enjoy learning new languages and approaches, in sustained development a motivated generalist team will typically beat a skill-matched specialist team.

The team is not self-organizing and cannot easily adapt to changing circumstances and new knowledge. The manager is largely assigning work. A team can only self-organize if we give them a regular form of shared communication, and the authority and responsibility to determine and accept work assignments.. The team members as individuals have to feel personally responsible for the delivery of a working whole piece of software. Without this communication, they cannot reasonable feel personally responsible for the whole. (Other conditions, normally considered outside the strict bounds of agile, can improve team members feeling personally responsible for the whole, see [Christopher Avery, Teamwork is an Individual Skill, 2003].

The team members do not interact routinely and lose an opportunity to discover and resolve shared impediments. Team members may meet sufficiently that a Daily Scrum isn’t necessary, but the fact that there isn’t one could be hiding dysfunction. Time-boxed communication rituals have a purpose: they bring people at a known time, so they don’t have to worry when they will get to talk about something. Also, people feel more like a collaborating team if they have one time during the day that they can clarify issues potentially with managers and Product Owners. (I know that technically Product Owners don’t have to go to the Daily Standup meeting, but in my experience, the best attend as normally silent observers.)

This ritualized conversation with Product Owner allow the team to satisfy the last Agile Principle, which is the art of maximizing the amount of work not done. It is here, at the Daily Standup, when I’ve seen many negotiations between Team and Product Owner that helped the team complete something usable for the customer.

In my experience, we can become truly agile by addressing these four items: making our future development efforts faster than they would be. Agile practices promote sustained development that operates well even with changing staff and circumstances, self-organization that tackles new problems as they arise, communication structures that rapidly identify and address impediments, and dialog between product and development that maximize the work not done.

Scrum Nails Forecasting

OK, that was the Agile discussion. Now let’s have the Scrum discussion. What does Scrum bring to the table that more generic agile doesn’t have? A bunch of things:

• It brings a discipline of velocity measurement. Velocity is a measure of feature production. Each Sprint is a science experiment that measures how that Sprint’s specific process rules affect the rate at which the team can develop. If we estimate story sizes with Planning Poker, this gives us a nice measure of feature production rate. If we incorporate Reference Stories into our estimation process, this metric has long-term comparative value.
• With historic velocity measurements, our science experiments (aka Sprints) can determine whether team and process changes are improving or degrading production rate. The addition of a new team member typically causes the velocity to go down temporary, due to a learning curve, but does it ever go back up? The imposition of Ready Criteria (or Definition of Ready) to gate whether the team should accept a PBI, typically causes the velocity to go up. Does it? This new transparency sparks conversations that previously had to rely on perception and anecdote.
• Now that you have velocity measurement, your Product Owner can forecast future deliveries of products. Product Owners can create longer backlogs, and by using efficient, unbiased techniques for estimation (see http://agilecanon.wpengine.com/bulk-estimation/), they can get statistically defensible delivery forecasts for features well into the future.

What many agilists and even some Scrum trainers fail to appreciate is that Scrum was created, in part, to address the awful forecasting history of waterfall. Scrum did not throw forecasting out the window. Instead, it established a much more robust, defensible and “agile” form of forecasting.

The problem with waterfall was that it was not empirically-based (“empirical” = based on experimentation). In waterfall, engineering leads and architects would hypothesize that a huge set of features could be done by a particular date far into the future. But their hypotheses were never tested, never corrected. No experienced person today believes a direct estimate provided by anyone in engineering: they are false almost by definition. And so, in waterfall, we see sandbagging start from individual developers and then accumulating sandbagging while up the management chain. From bottom to top, it is all based on hypothesis. Even the most-sandbagged hypothesis can turn out to be wrong, sometimes dooming companies to bankruptcy and committing millions of dollars and years of earnest engineer lives to the toilet.

Some agilists argue that long-term forecasts are not valuable, but few such agilists have run successful companies. (Some have run engineering departments, but this is not the same thing.) Successful companies have external constraints, including available time and money. If a feature can’t be produced by a particular time, sometimes we ought to use the money for something else. Shouldn’t we know early on that delivering a feature by a desired date isn’t possible, so we don’t squander engineers on features that won’t be used? The delivery of features on-time and in-budget is something considered extremely valuable to external partners (because they too are constrained by time and budget). They pay suppliers a high premium if they have a history of providing features on-time and in-budget, so much so that those suppliers have a huge competitive advantage against other suppliers with no such track record.

Scrum practices drives production statistics to greater and greater accuracy. By reducing the cycle times to under one-month, and by requiring the each Sprint product be releasable, we are now measuring not just “feature development”, not just “tested feature development”, but “packaged and tested feature development” in small enough time-periods that we can forecast from them.

In Scrum, a team’s Definition of Done (composed of Done Criteria) typically evolves from current-practice toward a sophisticated, shipped product.  Here’s an example of some common Done Criteria, ranked by increasing sophistication:

1. A developer runs a build of the system incorporating the feature from a single build command.
2. A developer runs a manual smoke test that confirms the system still works and the new feature is operating.
3. A developer checks in the code.
4. An uninvolved developer reviews the code, or the code was developed by pair-programming.
5. A developer wrote an automated functional test and checked it into continuous build system
6. The continuous build system shows no faults at all.
7. The developer wrote feature usage monitoring into the code and wrote functional tests to confirm it works.
8. The team released the feature live to a percentage of its actual users, measuring the usage and bugs that occurred.

Scrum Nails Continuous Process Improvement

If we aren’t having regular Retrospectives, we lose a mechanism to reflect on the process we are using and improve it. Of course, the velocity accuracy discussed above helps drive discussion to actually improve production rate, rather than focusing solely on whether team members feel good.

This idea that we reflect on little impediments daily, in the Daily Scrum, improves production rates at a low cost (timeboxed maximum 15m meeting).

Then we reflect on big impediments every Sprint in the Sprint Retrospective. This improves our own production rates, but also typically looks at external impediments and considers how the team might change things there. Here is where advanced teams start thinking about A3 problem solving and other mechanisms for negotiating with outside stakeholders and managers.

But even without this larger thinking that comes in Retrospectives, we can just use Retrospectives as our “science review”. What production experiment are we going to run next Sprint? Different Sprint size? Different way of structuring Standups? Different Ready Criteria? Different Done Criteria? There are so many variables to tweak, and when we tweak them, Scrum gives us an objective way to measure the result, consider changes, and produce much more value.

Scrum can be Scaled

The biggest scaling issue in a large organization is establishing a broad, shared understanding of the basic principles of a process and its implications to the business. Because Scrum is by-far the most recognized and understood of the agile techniques, it has a huge advantage. If we use Scrum, we are going to be able to easily hire people that understand and can apply our process at the team level.

We don’t need all of our managers to be geniuses to implement Scrum. A genius leader can overcome many process dysfunctions in any arrangement, even waterfall. But what happens if that person goes on vacation or leaves the company? What happens when you scale one genius’s anecdotal process to a larger entity? Can it work? There are very few techniques that have been as well-studied as Scrum, with as wide a variety of teams and products. We know it works.

Too many people try to apply agile principles to large companies without scaling those principles thoughtfully. If you conclude that everything we know about team-scale organization can be directly applied to larger entities, such as that individual self-organization can operate effectively with teams of 100 people, it could be true, but evidence is slim. For a rare example, see Gary Hamel, “First Fire All the Managers,” Harvard Business Review (Dec 2011) https://archive.harvardbusiness.org/cla/web/pl/product.seam?c=573&i=15715&cs=7c855bfce2fd1c3860846954978b1181. With few solid guideposts to implement mass self-organization, you must be a brilliant executive with a clear vision, leadership skills and authority. If instead you are just annoyed by process and trying to find any way to avoid it, an ad hoc approach will likely not scale.

Too many companies have unconsciously adopted a “federation model,” where each team or product chooses its process: different groups have Scrum,  Kanban, Extreme Programming (XP), waterfall, or ad hoc. Such arrangements cheat the company of a broadly shared understanding of engineering, and virtually guarantee a lack of transparency at the executive level. High-level decision processes appropriate for Scrum can’t be effectively applied to a waterfall project, and vice versa. Such arrangements quickly devolve into product silos with turf-wars, politics, inflexible budgets and capital waste—these problems may not be apparent to individual teams and developers, but they can be disastrous for a company’s bottom-line.

Too many companies have ordered a “team-level-only agile model,” using Scrum at the team level and operating with waterfall arrangements to compose teams. This approach usually comes with “component teams,” where individual teams are agile, but the dependencies between them create a waterfall ecosystem. If executives are demanding a formal dependency management process, you probably are in this situation. You can give them a formal dependency management process that parallels standard waterfall project management (for the layer above teams), or you can help the company to gradually convert to a feature-team model. There is little middle-ground.

Software engineering is an optimization problem over a complex system: In what order should you develop features to maximize the return? If you try to do this optimization solely at the lowest scale, you get something called “the Greedy Algorithm,” which causes your system to produce a very bad result. To get good optimization in a complex system, you have to operate at multiple scales simultaneously, but your greatest improvement is likely to happen at the largest scale. Top-down works well for these problems; bottom-up does not. See [Dan Greening, “Simulated Annealing with Errors,” 1995].

Understanding Scrum can lead to thoughtful management of the company itself. I developed a department management strategy called Enterprise Scrum, see http://agilecanon.wpengine.com/enterprise-scrum-2/, which handles large portfolios of products, using the same feedback principles you see in Scrum. Because Enterprise Scrum has velocity metrics, estimation, feedback loops, Ready and Done Criteria, it provides the ability to forecast and to improve.

Summary

In answering a seemingly simple question, “Are short cycle times (“sprinting”) alone sufficient for agility?” we have made a journey from ad hoc sprints, to generic agile, to Scrum and finally to principles of organizational scaling. In the five years I’ve been heavily working with agile methods, I have learned that such simple questions often have a complex answer.

Agile methods, especially in the case of Scrum, use fixed-length experiments to measure production, to fuel double-loop learning for continuous improvement. In my experience, most creative-team managers (development managers, particularly) can apply Scrum’s spirit to adapt rapidly to changing environments, and accelerate beyond waterfall or ad hoc development, even beyond ad hoc sprint development.