Should You Write Automated Tests for Your Automated Tests?

Posted on November 28, 2016

Should You Write Automated Tests for your Automated Tests?

Recently, I posted 7 Ways to Better Test Automation on a few linkedin groups. The response was overwhelming. I appreciate all the comments, thoughts and other ideas. Several of the comments deserved more than a reply on linkedin, however. Over the next few weeks I’ll be posting some of these comments and my thoughts about them.

I would like your opinion on something. You talk about testing the test code, it is an interesting subject.

I have always used TDD/BDD (test first approaches). In the TDD cycle Red-Green-Refactor, the test itself is tested by the fact that it initially fails and after the code is written passes.

Would you consider this sufficient testing for the test code?

To sum up, “Should you write automated tests for automated tests?”

I consider there to be 3 types of test code:

1. Implementation of test cases
2. 3rd party code (libraries, frameworks, services)
3. Custom utilities – code item #1 depends on

There are also 2 types of testing to be aware of:

1. Manual testing (eg running the test code to see if it works)
2. Automated testing (unit and integration tests, some call these “checks”)

My personal rule is, don’t write automated tests for the automated tests. Write automated tests for any other significant piece of code I own.
So in the first list above, manually test #1. While manually testing #1, make note of the behavior of #2. If #2 is broken or acts differently than expected, write automated tests to learn if something is broken. If so, pass along automated tests to those supporting it while working around the issue internally. Use TDD for #3.

The tests we’re writing test the framework itself. We see if they work when we run them.

Write tests for anything non-trivial used by the our automated tests.

For instance, if we need to create a utility class to record all the data we input into the system so that later we can delete it out of the system, use TDD to create this utility. This leaves a trail of unit tests behind to help us know when the code has been broken.

On the other hand, if I’m writing test implementation code – like a page object that defines the locators for components on a page in a web app, I won’t use TDD and unit tests for it. Instead, I’ll code by intention (a technique employed by TDD). I’ll begin creating a test I need and as I do, I’ll implement the page objects I need – but only when I need them. This way, the test case I’m writing will pass until I get to the next unwritten locator. By doing this, I get many of the benefits of TDD. I also get the benefit of creating only the code I need and nothing else.

So to answer the question “should you write automated tests for your automated tests”, my answer is no. But you should write tests for the utilities they use and sometimes for 3rd party code.

 

Image by Won Young Park, used with permission from unsplash.com.