Posted by IN / 3 responses

100% Code Coverage

7 August 2016

100% code coverage? Why not?

In a recent post entitled “What I Believe About Testing” I wrote “It is rare that 100% code coverage is necessary”. This may have caught people off guard — that a test professional doesn’t believe in 100% code coverage. Understand why…

Coverage

Cost vs Benefit

I’m a strong proponent of Test Driven Development. Not every one is. Lots of code has no tests. The effort in these cases to get to 100% code coverage is extremely high for even the simplest application. With greenfield applications, teams tie testing into the process giving a higher percentage of code coverage. Even when that is the case, there will be code that is low risk. There will be code that is extremely difficult to write tests for. There will be code that is both low risk and difficult to test. In those situations we must weigh the cost of coverage vs the benefit. If not, we’re doing the business a disservice.

My work with clients has shown me over the years that this line differs from one team to the next and necessarily so.

Boilerplate Code

Much of today’s code is generated or boilerplate. Do we really need tests for simple getters and setters that do nothing more than get or assign values of properties?In ORMs that generate code like Hibernate and Entity Framework, do we need to test the code generated by the framework? My personal rule is, unless I have a reason to distrust boilerplate or generated code, I don’t write tests to cover them.

Opportunity Cost

There comes a time in many projects where we have to look at opportunity cost — the cost of missed opportunities due to our attention on current actions. This is a real thing, not just biz dude lingo. It’s easy to think as a tester or developer that quality is the most important aspect of a project or that refactoring a major subsystem is priority. The truth is for most organizations their goals are tied tightly to the market and what the competition is doing. That tie is about survival. Remember, if the company doesn’t survive, you’re looking for a job. What is the opportunity cost for seeking 100% coverage?

Maintenance

Much like the maintenance of software applications, it’s rare that teams plan for the labor required to maintain tests over time. The closer we get to 100% coverage the closer we get to unaffordable maintenance costs.

Finally

Don’t get me wrong. I love coverage. I love high coverage. It works, it makes sense as a mechanism to insure against regressions. But like most endeavors, thoughtful approaches tend to generate better results.

About the Author

Paul Merrill

Paul Merrill is Principle Software Engineer in Test and Founder of Beaufort Fairmont Automated Testing Services. Paul works with clients every day to accelerate testing, reduce risk, and to increase the efficacy of testing processes. You’re Agile, but is your Testing Agile? An entrepreneur, tester, and software engineer, Paul has a unique perspective on launching and maintaining quality products. He also hosts Reflection as a Service, a podcast about software development and entrepreneurship. Follow Paul on Twitter @dpaulmerrill.

3 responses to 100% Code Coverage

  • Laura Vachon says on August 8, 2016 at 2:55 pm

    I agree with you that mindful testing should be the goal over 100% code coverage. Often people confuse code coverage with path coverage. I see the value of code coverage, however path coverage is more important IMHO. The last 10% of code coverage might include hitting obscure scenarios. In those cases, it might make sense to focus effort elsewhere.

    Reply
    • Paul Merrill says on August 8, 2016 at 3:30 pm

      I agree Laura! Path (or branch) coverage can be even more helpful than code coverage. Sometimes new clients haven’t used code coverage as a mechanism for evaluating test suites. Code coverage can be a great resource for those teams. Quite often, we get to the point it sound like you’re at, where we want to know what branch coverage looks like. In the absence of any metrics code coverage is great, but once in place other tools can be very helpful. I’ll be talking about this at my workshop at STPCon in Sept, 2016 in Dallas. Looking forward to hearing other people’s thoughts and experiences there as well!

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *