Enterprise Scrum Thinking: Part 2 of 3

Dave Prior interviewed me on the topic of Enterprise Scrum Thinking. In the second part of three, I discuss the experimental mindset of an Agile-run project, how a Sprint is like a science experiment on production (we make a hypothesis, test the the hypothesis and learn from the results), and why fear of failure can doom Scrum. I also touch on the concept of Shu-Ha-Ri, stages of mastery that can apply to learning Scrum.

Listen in at http://www.projectsatwork.com/content/PodCasts/275457.cfm.

Did you miss part 1? Check out Enterprise Scrum Thinking: Part 1 of 3.

Transcript

Dave Prior: In a lot of the places I teach, I end up with people that work for big old financial companies. I just did one last week and they’re really excited about [Scrum], they really want to get it going.  They’re the first people in the company to start looking at agile.  And once they understand the basics of how Scrum is suppose to work, then there’s this giant overwhelming thing that they’re facing which is, “Oh my god, how are going to make that work in a company of this size?”

Unless we get instant company-wide adoption and get accounting and everybody else on board, and everyone instantly changes how they work, it’s very daunting, you know, to see them go from changing how they individually work, which is hard enough, to feeling responsible for their whole company.

When you talked about the pilot thing, within some of the older financial institutions or companies that are just been around for so long that the culture is just burned down to the core of their bones, and difficult to change, are there specific tips or techniques that you have for people facing that, to help them kind of get started beyond just, Take a pilot and make it work and then spread it around?

Dan Greening: I have to admit, I have not worked with a financial organization.  But my strategy in engineering organizations is not just to start a pilot and have that pilot run.

The way I like to look at organizations is as fractals.  So, a good example of a fractal is to go out into some yard and look at a tree.  And if you look at the forking pattern of the tree, from its trunk to its major branches, and then you look at the forking pattern of the tree at the level near a leaf, you’ll see that there’s a lot of similarity in the pattern of forking in the tree.  Basically, it just depends on scale, right?

Self-similar properties occur in organizations.  If you think about how a scrum team operates, it needs a product owner to prioritize things, and there needs to be only one. There’s a bunch of people that are producing value together as a team. You have a ScrumMaster who’s highly concerned with their productivity, shielding them from outside distractions and addressing impediments.  They’re the sacrificial lamb for impediments, right?

If you think about that property and then you scale it up, then you start dealing with some of the things that I started talking about.  If you’re deploying agile in an organization, it pays to look at the organization on multiple scales at once, and then address a little bit on each of the scales simultaneously.

So, of course, you do one or two prototype teams at the ground level, but now you actually start having a dialogue with engineering managers, directors, a vice president or two and say, here are the major principles of agile; here’s how they apply to the team, here’s the likely impact to you in terms of how you think of your business.

I think the biggest fundamentals of agile principles, and what I call lean product management (the stuff that Eric Ries is talking about in his book Lean Startup), is that all agile methods are iterative.

The idea is that you want to produce something that you can test within a small period of time, and you want to, in that small period of time, do all of the functions you would do in the larger scale of a huge product deployment, but you want to do it in the scale of the small iteration.

So, each of these iterations is really an experiment.  If you think back about, college or high school science class, science was defined as, some scientist observes something in nature and makes a hypothesis about a principle that might apply. There’s a hypothesis in Scrum, and the hypothesis is we can finish this set of backlog items at the top of the backlog  [or achieve the Sprint Goal] within the next sprint.  That’s your hypothesis.

Then you run an experiment.  And in that experiment, you try to keep the subjects [isolated], you try to keep the environment controlled; you don’t want interruptions, you don’t want distractions.  You want some clarity around, Are these items ready?  Have we eliminated dependencies? As long as you can control the experiment really well, the experiment can run effectively and you can draw some smart conclusions from your experiment.

So we apply Ready Criteria and other stuff like that at the beginning. And then when we run the experiment with the team actually trying to build the Product Backlog Items, the ScrumMaster is there as the scientist who is controlling their environment and helping make sure that everything goes smoothly.

Then we have a Sprint Review, where we look at what we did.  And here we are evaluating the results of our experiment.  So we thought we could do such and such, but we actually got done, let’s say, two-thirds of the things we thought we would do.

So now we’ve evaluated the results according to the metrics we setup, and we’ve got a velocity, which is one of our primary metrics in Scrum, and now we have a Retrospective.  Well, the Retrospective is really about, Hey, what did we learn from this experiment? What kind of interesting things arose during the experiment? and What could we try next time that might improve our productivity?

And so, it just iterates repeatedly over and over as you go forward, the idea being that it’s an experiment in productivity, but it’s an iterative experiment.  So the same thing happens with lean startup principles.  The lean startup principles are you want to run an experiment with a market, you say, for example, “Hey, you know what? I think people will pay for 3D images with my audio stream,” I guess.  Something silly like that.

And you say, okay, well, let’s devise some experiment that we can run a short period of time, like a month, where we determine whether people will pay for that feature.  So, you know, there’s a lot of different techniques in lean startup principles.  But basically, you setup this experiment and then you actually engage the market and see whether the market will actually pay for that stuff, and then, at the end you say, Hey, did we decide that the market would pay for it?  And if so, what experiment should we run next to better refine what they’re really looking for?

If not, did we learn anything that might help us find a better market?  And then that iterative experiment runs again and again.  So, for both of those things, Scrum and Lean Product Management, iteration is where it’s at.  And the reason that iteration is so important is because otherwise, we’re betting the farm on our internal productivity and our ability to manage dependencies.  That’s kind of the effort side of the world.  Or we’re betting the farm because we’re making a big bet on a market that we think is going to be there when our project is done.  And when the project will be done 18 months from now, who knows what might happen to the market?

That iterative mindset, that is an experimental mindset. Our typical managers, directors and executives, they’re not used to experimentation.  What you find culturally in organizations, I would guess especially in financial organizations, is that there is a fear of failure that will cause people to get enormous anxiety around agile methods, especially if they have never experienced them themselves.

Jobs seem to be at stake and other worries come up for middle level and upper level managers when we start talking about agile.  This really dooms agile unless you address it well.  So that’s why I talk about introducing these concepts at upper layers of management, that, Hey, you know, we’re doing experiments here and we have to be ready for a significant level of failure, because the way we’ve done business before, in slower moving markets that we dealt with before, we could afford to be plodding and slow and plan well in advance and, operate these long projects.

That’s not possible anymore.  The market is moving way too fast.  And therefore, we need to do some experimentation and assess whether to do a bigger project. We’re going to make small experiments that don’t risk enormous parts of the company.  Instead, we’re going to take little risks, we’re going to fail in-the-small, but that small failure is going to help direct the bigger picture in the future.  It takes a while to get these concepts across.  And so you should address them early.

If you don’t, what happens is now you’ve got a bunch of little teams down on the ground, using Scrum. I’ve been in situations where there’s 30, 40 teams operating with Scrum, where high level executives didn’t understand this sort of experimental nature, and then, something changes in the corporate configuration, company executives get scared, and that fear can cause them to go back to old ways, to go back to waterfall.

Dave Prior:  Yeah.  And that’s that reverting you were talking about before where they start to make progress and then they go in the opposite direction.

Dan Greening:  Yeah.  It’s interesting.  I had a conversation with Rob Mee, who’s head of Pivotal Labs.  I told him I’m very interested in this process of reversion, because I’ve seen it happen with some of the best names in the industry. They have coached companies to adopt agile, the companies got lots of great metrics, they were producing well, their market share was increasing, all sorts of great value was coming from agile.  Then some change happened in the executive staff, and then agile dissipated or disappeared.

It’s disturbing, of course.  You know, like you think you are bringing great value to the organization, and it does get the value.  And yet, for some reason, the organization doesn’t even know, or doesn’t give itself a pat on the back for doing a great job in increasing its market share and all this other stuff.  Instead, you know, it just kind of dissipates.  And that’s why, a) You have to educate your executives, and b) When you hire executives, they should have a mindset that is compatible with agile.

You know, it’s very common that we hire executives who look like they have a great resume, and of course, who knows how they got their job?  But the number one thing is do they understand this sort of experimentation mindset?  Are they failure tolerant?  Do they believe in continuous improvement?  And what does that really mean?  Like what are some good examples of how they’ve supported continuous improvement in the past?

Those kinds of questions are really key, if you want to hire executives that will serve you well in the future.  Anyway, I’m sorry.  We’re getting a little off topic.

Dave Prior:  No, no.  It’s okay.  Because you’re going in a direction I wanted to go anyway, that’s a little bit away from enterprise.  So, if we can keep going that way, and then we’ll come back.

Dan Greening:  Sure.

Dave Prior:  I want to talk about Aikido for a second, if you can.

Dan Greening:  Sure.

Dave Prior:  So shuhari.pro is the name of a domain that you own. That term is used a lot when we talk about agile.  But for those who aren’t familiar with it, can you explain what shu-ha-ri is, and then I’ll walk you through why I’m heading this way.

Dan Greening:  Sure.  So shu-ha-ri, it basically refers to a sequence of learning, a style of learning.

Shu, in my mind, means “replicate the master.”  So, if you’re in Aikido or some other martial art, the first thing that you try to do is you just try to replicate the master’s moves in terms of how they do different defenses and offenses and that sort of thing.

Of course, you’re learning.  But what you’re learning is how to mimic.  You’re not doing anything creative, really.  You’re just trying to replicate.  And that Shu level of learning is really important because once you have embraced that, you have really infused that notion in your being, in your actions. You have muscle memory that teaches you to do a particular thing when something happens.

Ha is when you start kind of improving that within the framework of your martial art, I would say.  So now you have some stylistic moves that are variations on the mimicry of the master.  They’re still allowed within the framework of the approach that you’re using.

Ri is when you combine techniques from other fields or when you take certain creative risks around your expression of the martial art and start really making it personal, I would say, but also really breaking rules in a sense.

You know, you have a set of rules that you had to operate under when you were mimicking. In the Shu level, you are mimicking. At the Ha level, you’re retaining those rules, but you are introducing some new concepts that were compatible with those rules. And Ri is where you’re saying, “Hey, you know what? Maybe those old rules applied when I was younger and less experienced.  And now that I know more and I’m comfortable with those techniques, I know when I can break these rules effectively to gain more value.”

So, shu ha ri is used in Scrum teaching, frequently, because the first thing we want you to do is apply the basic rules of Scrum, and then we want you to introduce your own variations within that and feel comfortable with it.  And then you may break those rules, once you get very sophisticated. You have to be careful not to break the rules when you’re in the Shu phase, because probably the basis for your rule-breaking is not very sensible.

So, also shu ha ri actually refers to how I learned Scrum.  When I learned Scrum, I just read tons of books on Scrum.  And a team of mine said, “Hey, you know, we really want to use Scrum in our team.”  And I said, “You know, it sounds great to me.  Let’s do it exactly right.  So let’s do pure Scrum.  Let’s not, you know, make compromises about whether we have a daily standup or a weekly standup, let’s do what they say to do, daily standup, let’s do four weeks sprints because we thought it would be better if we had time to actually build the application and deploy it.

We basically tried to follow every rule we could.  And we did that for, I think, three or four months. We started to get some great results, and we started taking variations of that.  So we reduced our Sprint cycle down to three weeks.  We started doing pair programming.  Those kinds of things that are totally compatible with Scrum but they’re not dictated by Scrum.

And then, as time went on, things got more fluid.  We brought in an operations person from outside, and we started doing almost continuous deployment, which is not necessarily an aspect of Scrum. One of the things that I’ve seen Scrum teams do, as they get better and better, is sometimes they don’t even do task breakdowns, or they do task breakdowns but they don’t assign people to those tasks, initially, until people are available, and the might have really bizarre rules like, when one person comes available, they work on the top priority item, even if they’re not skilled to do that item, and then they bring in someone to pair-program with them to teach them the skill that’s necessary to solve that item.

So there’s a lot of interesting ri forms of Scrum.  And I guess I would say Enterprise Scrum is kind of a ri idea.  You know, it’s like, we’re going to really break the rules, right?  We’re going to deal with multiple teams, and we’re going to put them in a big framework.  It sort-of breaks a tenet of Scrum, which is self-organization.

Normally, we think of groups of people and we want them to self-organize.  And if you look at some of the scaling techniques that were developed early on, they thought of self-organization as something that would scale to hundreds of people. They would put hundreds of people in a room [laughs]. We have these, you know, thousands of backlog items…Hey you hundreds of people! We need you to self-organize in the way that can actually address these thousands of items!

Okay.  Well, that’s—I don’t know.

Dave Prior:  It’s not going to work.

Dan Greening:  [laughs] It’s probably not.  You know, there are people who have done it, and who will argue to the death that it works well.  But I’ve certainly seen it fail dramatically, and I have tried it myself.  So, I do think that taking a step back and saying, “Hey, you know what, yeah, with self-organization eventually we could come up with a solution that was really effective for dealing with this scaling problem.  But the time that’s required for the self-organization to occur with these hundreds of people is likely to be on a scale that would put us in the dinosaur category, if we’re not careful.

So, let’s actually be thinking people, and take a step back and say, “Hey, self-organization at the team level, we get that.  So let’s talk about self-organization at higher scales, you know, the 300 people scale, 400 people scale.  Well, maybe it’s not self-organization among people.  Maybe it’s self-organization among teams interacting with each other or managers interacting with each other.

Other forms like that makes sense.  If you have too many people trying to self-organize, the communication cost of that self-organization is extremely high.

We talk about limiting the number of developers on a Scrum team to seven.  And when I say developers, I mean QA and developers and database and all that stuff. So we try to limit the number of people in those teams to seven.  And the reason is because when you go to eight, the communication cost between those eight people is so high that it reduces the productivity of the team, overall.  And so, it just doesn’t make sense to add another person, right?

Any of us who have seen a manager that was trying to effectively manage more than seven sub-managers, their job became very hard [or they didn’t organize those managers to do anything together], and likely their productivity declined. When I say productivity, I mean the amount of value their [group of managers] produced over a period of time. Because they had so much communication cost, they were unlikely to really have a highly productive group.

Anyway, so those are the ideas. Don’t assume that everything that works at the ground level among these individual actors, we can just bring up a level and keep the same scale of actors and using the same sort of communication for hundreds of people.  Well, if it worked, then we would have Scrum teams that had hundreds of people in them, right?

Dave Prior:  Yeah.

Dan Greening:  It doesn’t really work.  Anyway, this is an argument that I get into periodically with colleagues.

Dave Prior:  Thanks for listening to part two of my interview with Dan Greening on Enterprise Level Scrum.  Make sure to check back on the Projects at Work website for additional segment of this podcast interview which will be posted shortly. Dan can be reached sending him an email at dan@senexrex.com.