I originally posted this in 2012. Since then, we’ve had tremendous success in implementing automated testing solutions – recently cresting with a 16X ROI in less than 6 months (trumping “Project B” below in both lower costs and higher output).
Even in the case of the lowest performing projects since then, we’ve had positive ROIs (beating “Project A” heartily) and clients have been pleased.
The leadership aspect of this article is perhaps the most intriguing. Since I wrote it, I’ve realized organizations don’t need to have a General Patton leading the effort, we don’t need a Theodore Roosevelt corralling organizations. Instead, we need effective leadership.
The difference is as long as you have technical expertise (insert clever pitch for our expertise here) from a journeyed guide that you’re willing to listen to AND management supports the effort — pragmatically, not in ideation — you have the leadership necessary to collect on automated testing.
Here is the original post…
This is from my series, The 5 Secrets to Automated Testing Success.
A good leader must understand her objective, communicate it clearly, and work with her team to achieve the objective. This preparation is particularly crucial for successful automated testing.
If you’re leading an automated testing effort, what do you want to achieve? Have you or your manager defined your goal or goals? If not, you will not and cannot succeed. Goals must predate means.
Vaguely defined goals such as “automating as much as possible” are not attainable. Instead, an effective leader might create a goal of “95% code coverage by December 31st.” Define success by setting a clear target for your automation effort.
Obstacles
You will encounter obstacles along the way to your objective – that’s expected. A good leader prepares her team to meet and conquer them.
Your automation tool may fail to accomplish a critical goal. There may be some misunderstanding between what the salesperson told you and what you find the tool can actually do. How are you going to handle that? Are you prepared? Do you have the expertise and skill-set to overcome the challenges that will arise in the course of your project?
A good leader staffs her team to overcome obstacles.
Expect a High Return
What is a good return?
I had this question too. In order to find an answer, I analyzed a number of projects we’ve completed over the last decade. I wanted to gain insight into what the key metrics in an automation effort were and how they related to project success or failure.
The primary justification most managers use for an automation effort is saving labor. I determined the labor costs (Cost of Investment) for these projects by talking with recruiting agencies that specialize in recruiting these skill-sets locally. Additionally, I needed to understand the Gain from the Investment. There are several ways a person could determine this, but in this case, I was looking for the amount of testing time the company gained by pursuing automation.
Each test case does work that a human would otherwise do. I took the amount of time a human would take to do the work and multiplied it by the number of tests cases. Each type of test case had a different average amount of work it would take a human to do, so that was considered.
So the total amount of labor hours generated each time the automated test suite ran multiplied by the number of times the suites ran over the course of the project was the gain.
Knowing these two factors, I could calculate the Return on Investment (ROI).
Return on Investment gives us a good understanding of how each invested dollar pays off.
Of the projects I analyzed, I decided to take the most successful and the least successful projects to use as reference points in this discussion. Once again, these are real projects that intended to use automation to improve the efficiency of their testing efforts.
Project A
ROI is calculated as a ratio. On Project A ROI was -0.97. Yes, that’s negative. That means that for every dollar invested in the project, 97 cents were thrown away.
Project B
Project B was quite different. For every dollar invested in labor, the client got back $8.32.
Notice the amount of labor was not all that different in these two projects. After 10 months, Project A was estimated to be at $231k and Project B was at $228k. The gains on the two projects, however, are significantly different. I’ll talk more about these projects and what made Project B so successful as I reveal the other four secrets.
[…] do you make the gains we mentioned in Secret #1? In Project A, automation failed to pay for itself. Instead, it cost hundreds of thousands of […]
Great blog here!
Thanks! I’m so glad you like it! Make sure to keep checking back, there is more to come!
[…] Is it helpful to know that the system broke last night? Absolutely. But the cost of adding hundreds or thousands of defects to a defect system is too high. Unless, of course, you want to make development look bad, and then you have other problems. See Secret #1. […]