Since originally posting this in 2012, I’ve re-thought some of these ideas.
Yes, it is important that test cases are readable, but for whom should they be readable?
If a tester without coding skills will be reading test scripts, tools like cucumber and robotframework are ideal. They allow for human-readable test cases. If, on the other hand, developers or automation engineers are going to be the ones reading test cases, sometimes it makes sense to write the tests in the programming language and toolset they already use.
Enjoy the original post…
This is from my series, The 5 Secrets to Automated Testing Success.
If we’re writing automated tests, what do we need to do in order to succeed?
· Write test cases are readable by human beings
· Write test case names that define (succinctly) what they do
· Write test cases that fail for one and only one reason
· Organize test cases from the perspective of reporting
· Extract repeated test case code
· Extract repeated literals
Readability
One cost that is rarely accounted for in automated testing is the learning curve of an individual who wants to read a test case.
Perhaps a Business Analyst wants to see if the dev team has created a test for a certain situation. Perhaps a manager wants to investigate why a build failed. Perhaps you have a new tester writing automated tests for the first time, or a developer wants to see what a test case is doing. How long does it take this individual to read and understand a test case?
Reading a test case should be like reading one’s native language. Make test cases readable, and make them look like a story. Test cases don’t need lots of comments in the code, but they do need keywords and methods that signify their function. This has the added benefit of helping you understand what you did as time passes.
Naming Test Cases
A test case should fail for one and only one reason. Why is this important?
We create automated tests for a number of reasons. One reason is to replace human labor with computer labor. Too often, we think of that as the only reason. Another reason is to provide a clear assessment of what works in the system and what doesn’t work in the system at any given time.
The faster a project team understands what’s working and what isn’t, the faster we’re able to isolate a defect and fix it.
The clearer and more succinct we make the name of a test case, the easier it is to see which part of the system broke in the last test. It’s much quicker to look at a report and know by the name of the test case what failed than it is to look at a test case number (for instance) and then have to look through the test to understand what it does.
This can save hours per day in understanding what worked and what didn’t in a test run.
Fail for One and Only One Reason
This is one of the biggest reasons for failure I’ve seen in automated testing. For some reason, we think that we need to verify each step in a test case. Well, that’s simply not true.
Most humans don’t like to do the same thing over and over. We know that when we go through a set of actions in the system under test, we should go ahead and check to make sure that each action works properly, because it saves time – as a human.
Computers, however, don’t care about doing the same thing over and over again. Instead of adding 20 verification points to one automated test, break that test up into 20 test cases. Make the repeated parts of the test cases reusable, and have one verification point in each test case.
In many cases, what you’ll find is that you weren’t testing all cases with the original 20 verification points to begin with. Sometimes, you’ll need to go back and create multiple cases for each of the 20 new test cases.
When your test cases have one verification point, or one reason to fail, you’ll see that it’s much easier to name your tests. Additionally, it’s much easier to understand them, organize them, report on them, and give and get feedback on them.
Finally, if you’re not yet convinced, think of what happens when you’ve looked at a test case with 20 verification points. You’re hard-pressed to know why the test even exists or what purpose it serves!
Organize Tests from a Reporting Perspective
If your team isn’t using continuous integration, you’re missing the boat on automation. You should be able to sleep at night knowing your test cases are running and your system under test will have a reliable status in the morning.
Part of most continuous integration systems is reporting on tests that fail. Your project team needs the ability to scan reports and know in minutes where in the system a defect resides (or doesn’t reside). If your tests are organized in such a way that reporting is easy to look at and understand, your team will save a large amount of time looking into defects that were produced in the last 24 hours.
Reuse Repeated Test Code
As we discussed in Secret #4, we want to select a tool that allows you to reuse your test code. Maybe it allows you to create keywords, maybe its methods or functions or modules…regardless, the ability to reuse test code is a fundamental characteristic of a good testing tool.
Why?
Because one of the biggest costs or benefits to your automation effort will be maintenance of test cases. Maintenance will be reduced if you can look at test code and make a change or a fix in one place as opposed to multiple spots in the code.
This is also one of the reasons traditional “record and playback” tools don’t work well. In order to reuse repeated code and to reduce maintenance of test cases, you have to stop using record and playback.
So whenever you find code that looks similar, make it the same by introducing variables, and creating a function, method, or keyword with those variables as parameters.
But bear in mind: as with test case names keywords, functions, or methods are more easily used and read when they are succinct and clear. The same goes with introducing variables into test code.
Want to learn more about Automated Testing? Contact us about Training or Consulting.
Paul,
I enjoyed meeting you at TISQA. I’d be interested in taking you up on your offer to do a brown bag book talk at our Chapel Hill offices.
One thing I’d be interested in digging into on your visit is the potential conflicting tensions between
“Fail for One and Only One Reason”
And
Data-driven testing / creating one reusable test case and use a table of external variables.
Justin,
It was nice meeting you too! I’ll reach out to you over email for setting up a brown-bag for your team.
For those that don’t know, I recently (March 2016) spoke at TISQA 2016 in Chapel Hill, NC. The talk was called “Data Strategies for Automated Testing”. I made the offer to come in and do a “Lunch and Learn” for any of the local teams, presenting the same talk to their folks who may not have been able to attend. The offer still stands, just reach out to me! I’d love to meet your teams and talk through your special circumstances.
Justin, you’re correct, it’s easy to find points of friction when working to implement a tactic like this. Data-driven tests provide a challenge here. There are a number of ways to get around it, but initially, I’d look at determining the intention of the template. What is the function of the template you’re using and passing data into? If you can’t describe it or name it easily, that’s generally a sign that the test is a bit vague.
Is the data that’s driving the test the force that’s pushing that template into vagueness? If so, are there ways to break down the data sets in such a way that meaning is more clear? Are there different aspects to the same template that can be broken down to more concrete concepts?
Our team typically finds that yes, there is a balance each client and situation has to find between abstraction (like using a data-driven approach) and approachability (both in reporting and understanding the meaning of the test).
Knowing what Hexawise does, I would imagine this is of high-importance to you and your customers.
Let’s talk more, but hopefully that gives you an idea of where I’d go with that particular set of forces!