Reasoning before measuring

(Originally published by Lv Yi on May 7, 2020)

I often get this question – what do you measure while adopting Agile?

The conversation usually goes like this:

[me] What is it about adopting Agile?
[him] … doing Scrum.
[me] What effect do you expect after adopting Agile?
[him] higher efficiency.
[me] How would you define higher efficiency?
[him] … finishing more tasks.
[me] Why would adopting Agile lead to higher efficiency?
[him] Agile means higher efficiency, doesn’t it?!

The thinking is fuzzy, to say the least. I suggest to focus on reasoning before measuring.

Let’s take Specification by Example (SbE) as an example of a concrete agile practice. What do you measure while adopting SbE?

Practice and Purpose

Firstly, we learn about SbE by asking 1) what is SbE, and 2) why do SbE ?

A good source of learning is of course the book “Specification by Example”.

1) What is SbE?

“Specification by Example is a set of process patterns that facilitate change in software products to ensure the right product is delivered effectively.”

Specification By Example

Suppose that among those process patterns, we choose to focus on the below two at the beginning of our initial adoption. We dive deeper into them.

  • Specifying collaboratively
    “Instead of relying on a single person to get the specifications right in isolation, successful delivery teams collaborate with the business users to specify the solution.”

  • Illustrating using examples
    “The team works with the business users to identify key examples that describe the expected functionality. During this process, developers and testers often suggest additional examples that illustrate edge cases or address areas of the system that are particularly problematic.”

2) Why do SbE?

There are various benefits from the whole set of SbE process patterns. Let’s focus on the benefits from “Specifying collaboratively” and “Illustrating using examples”. In other words, let’s understand the purposes of these two specific practices.

“Specification by Example helps to improve the quality of software products, significantly reduces rework, and enables teams to better align analysis, development, and testing activities.”

There are three areas of benefits.

  • Higher product quality
    “Specification by Example improves collaboration between delivery team members, facilitates better engagement with business users, and provides clear objective targets for delivery - leading to big improvement in product quality.”

  • Less rework
    “Specification by Example helps teams establish a collaborative specification process that lowers problems in the middle of an iteration. Most teams have significantly reduced or completely eliminated rework that occurred as a result of misunderstood requirements or neglected customer expectations.”

  • Better work alignment
    “Specification by Example enables teams to clearly define a target that’s universally understood and objectively measured. As a result, many teams find that their analysis, development, and testing activities became better aligned.”

The above reasoning could be summarized as:

Benefits of doing SbE

Only after getting this clarity we can go to the measuring part.

“Practice has purpose, and we measure both the practice and the purpose.” I learned this from David Hussman in a private workshop for Odd-e in 2013 – one of the most memorable experiences with “the dude”.

We may come up with the following measures.

  1. For the practice
    • %stories done with SbE (measuring the extent of doing SbE)
    • %stories with cross-functional conversations (measuring the extent of specifying collaboratively)
    • %stories specified with examples (measuring the extent of illustrating using examples)
  2. For the purpose
    • #defects due to misunderstood/neglected requirements after done (measuring product quality)
    • #defects due to misunderstood/neglected requirements during the sprint (measuring rework)
    • %stories reaching completion at the end of the sprint (measuring work alignment)

Causal Diagram

The Book of Why” introduces causal diagrams as a tool to move from correlation to causation. Causal diagrams are similar to causal loop diagrams in systems thinking – they are both based on causation.

After some understanding of the practice and the purpose, we can reason further with finer-grained causation chain by identifying intermediate steps. Why does SbE lead to higher product quality, less rework and better work alignment?

The below causal diagram illustrates our reasoning.

Finer grain benefits of doing SbE

Then, we can measure the intermediate steps, so as to learn the whole chain of causation via validation or invalidation. Note that some variables are harder to measure than others. For example, “amount of rework” is harder to measure than “internal defects”. We don’t have to measure all the variables. With a few intermediate ones, we build our confidence in this causation.

We may end up with measuring:

  • %stories of doing SbE (i.e. specifying collaboratively and illustrating using examples)
  • #internal defects during the sprint
  • #external defects after done
  • %story completion at the end of the sprint


We need to reason about causes and effects, before getting into measures. On the other hand, measuring can also improve our reasoning in two senses: 1) thinking about how to measure increases the rigor in the reasoning, and 2) measurement results validate or invalidate the reasoning.

Notice that measuring here is for learning and improving. See my previous blog “Two Guidelines for Metrics” for further reference.

comments powered by Disqus