Measure the Things You Can Control

As I’ve mentioned in many previous articles, sports provide an outstanding backdrop to teach Lean lessons. This aspect of athletics was reinforced in a recent article in ESPN magazine. It takes a detailed look at the statistics behind one pitcher’s performance.

This player had an outstanding year in 2012. By one measure, he was responsible for adding 5.3 wins to his team. This year though, using the same metric, he is responsible for one additional loss. On the surface, this looks like a fairly significant drop in performance. The problem though, is that these metrics focus on results (NOTE: The link will take you to 55 minute video on metrics.) And results have both an internal and an external component.

When looking at the factors within the pitcher’s control, the situation appears much different. There are a handful of metrics that focus more on what the pitcher is doing and less on the performance of the batter. These numbers are much more consistent year-to-year. What that means is that his performance in 2012 was probably not as good as results indicated, but this year’s performance is much better than he is getting credit for.

The issue is that with enough players, there are going to be a few that fall onto the end of the statistical curves. Look at it this way. If 20 pitchers throw the exact same pitch, they may each get a different result. Some batters will watch the pitch come in. Some will swing and miss. Some will foul it off into the stands. And a few will put the pitch into play. Of those that do, some will result in outs and others will result in hits. The point is that the outcome is outside of the control of the pitcher. He controls the pitch placement and velocity, but once the ball leaves his hand, someone else takes over deciding what is going to happen.

In Major League Baseball, 30 teams play 162 games apiece. Each game normally requires pitchers to face around 35 or 40 batters. There are going to be some results that fall at the edge of the spectrum, but still within what we would expect to see, statistically speaking. This pitcher seems to be one of those whose results fell on the long tail of the normal distribution curve.

The lesson to take away from this is to make sure that you are using process metrics in addition to results metrics to check how well your team is performing on a day to day basis. And make sure that the measurements you choose are related to things within your team’s control.

For example, imagine you are a ditch digger. The weather probably plays a role in your work. On days below freezing, your productivity probably drops because of the hard ground. On rainy days, it is probably easier to dig than when the ground is dried out and rock hard. On hot days, water breaks and fatigue would slow you down. A process metrics might be scoops per hour or average scoop weight. Both of these would be affected by weather conditions. Similarly, productivity metrics may be influenced by parts shortages. If your process metric does not take these outside influences into account, you may find yourself trying to take countermeasures on the wrong things.

Done right, metrics can be used to drive outstanding performance. Done wrong, though, they can demoralize a team, send misleading messages, and have you chasing your tail.

Tips for Improving Metrics

• Minimize the time between activity and metric. Evaluate a metric as close as possible in time to when the work was done. That lets you get a more real-time look at what’s going on. Problems with metrics become more apparent when there is a shorter delay.
• Have a mentor take a look at how you measure. Outside eyes, especially experienced ones, can frequently identify pitfalls that you overlook.
• Ask your team. They are close to the process and will have good insight into whether the metric represents what they are doing, and if it is a good measure of how things are going.
• Look closely at outliers. Make sure you understand the activities that make your data jump. Pay special attention if the causes are outside of your team’s control.
• Stress your metrics. Try making changes and see if the metrics respond the way you expect them to.
• Benchmark. Take a look at similar processes and see how they are measured. They may have good ideas you haven’t thought of.
• Look for correlations. Try to identify if your metrics track against an outside factor. If they do, see if you can compensate for it in your measurement.
• Don’t neglect the things you can’t control. I don’t want the takeaway to be that you should not measure outside factors. I just mean that you shouldn’t let your process measures be influenced by them. Measure them separately.