About

We see a lot of people claiming proudly about their test coverage saying that they have achieved a 90+% test coverage.

That’s cool!

Having extensive coverage is surely something. But it’s not everything!

The intention of the blog is to give a deeper insight and not have false hope by proxy-metrics.

Some definitions

Proxy-metrics

Metrics are usually a way to measure something that you are interested in, i.e., length, time, etc.

When things get complex, you might not have a direct way to measure things that you’re interested in. Hence, you come-up with a measure that helps you measure the “complex goal” that you’re interested in.

For example, in contexts like application rewrite decisions, sometimes “Lines of Code” is used as metric for code complexity . Obviously, the rewrite might be achieved effectively with less LoC by choosing appropriate programming language, library , design pattern, etc. Hence "LoC" is a proxy-metric that is being used here for the lack of better metric or lack of resource to come up with a informed call.

Goodhart’s law

Goodhart’s law states that,

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

In other words, we define few metrics to measure a goal. And if we're chasing only the metrics to achieve the goal, we might miss the essence for which we have set the number.

In such cases, in addition to keeping track of the metric, an insight and perspective of the problem is important as well.

This is especially good for proxy-metrics. You can read more about an anecdotal story of cobra effect.

Now let's get back to how these are related to test metrics...

Test Coverage - another proxy metric

Being software engineers, “Test coverage” is another metric which we strive to increase at our projects. This roughly translates to “% of LoC covered by your test case”.

Though the metric is a useful one, it’s important to understand that “Test Coverage” is also another proxy metric.

Why is it a proxy-metric?

The fundamental goal is to ensure that we have maximum possible tests by ensuring “no untested code in our codebase”.

But does it mean that we are perfect if we have achieved 100% test coverage?

No, let’s demonstrate that with an example.

100% test-coverage doesn’t mean perfect quality

Let’s take an example snippet to find the maximum number in an array:

public static int findMax(int[] numbers) {
        // Max is set to 0 (obvious bug)
        int max = 0;
        for (int num : numbers) {
            if (num > max) {
                max = num;
            }
        }
        return max;
    }

Now we gain 100% test coverage for the above method with a 1-line test like this,

@Test
public void testFindMax() {
    assertEquals(5, findMax(new int[]{1, 2, 3, 4, 5}));
}

But apparently, there are many cases in which the function findMax will fail to give the expected results, like,

• Empty or null array.
• Array with all negative values, etc.

Thus, it's clear that test metrics alone cannot ensure the best quality of the code.

Conclusion

Though the test-coverage is an important and valuable metric on the extend of the tests that we have. It's also important to know that you might have a false sense of security based on the test-coverage.

The message here is not "Don't use test coverage", rather "Don't be satisfied just with the test-coverage and beware it's a proxy metric".

In the upcoming blogs, we will share some of the techniques to write better tests.