Posted on October 31, 2019
In my last post, I released a simple ROI calculator. If you didn’t download it yet, or watch the webinar about it, this post won’t make any sense. So do one of those and then come back and read this. I’ll wait…
OK, now that you’re back, I know what you’re thinking. “Paul, this calculator doesn’t even begin to cover all the factors associated with good test automation”. You’re right. I’m ok with that. Let me share with you my thinking.
Iterating
After working in Agile, Lean, and DevOps environments for many years and bootstrapping two companies, I’ve been humbled. I know I can’t create the “perfect” anything. But I also know that with help, I can improve nearly anything over time.
So consider this spreadsheet an iteration of an ROI calculator. It’s actually about a 3rd or 4th iteration already. Several employees helped create version based off of data we collect from our clients and use internally.
It is simple because it is the first version.
It is also simple because I wanted it to be simple. The people who come to this site have all different levels of understanding test automation and testing. Many only have a few simple metrics, like the ones I request in the calculator – number of releases per year, calendar days per test cycle, number of testers in test cycle, etc. I want to take the most basic data and help people start a conversation around test automation and the value of it.
Execution Time
Some times clients assume that the biggest benefit of test automation is reduction in test execution time. The assumption being, a person takes more time to run a test than a computer (many times this is true).
Additionally many folks believe that execution time is the major factor in how long test cycles take (spoiler alert: it’s not).
In my experience thinking, communicating, experimenting, and planning take the most time in testing – not executing test cases. Further the assumptions above do not consider the time it takes to create a test case for a person executing it immediately versus for a person writing an automated script. That difference can go either way (faster to write automation or faster to do it manually) depending on skillset, testability, scenario, and many other factors.
Because so many assume execution time is the major factor in creating a gain, I wanted to present a calculator that made the same assumptions. Try out the calculator and see if it works properly based on these assumptions.
Costs
I’ve only included labor costs during the first year of a test automation effort. That’s silly.
Good test automation teams continue investing in automation after scripts are written. A test script is a living document of behaviors of the system at some point in time. When the system changes the script must change. Maintenance is a required line item in your budget. Better automation reduces the cost, but the cost will always be there.
I do not itemize the cost of licensed a tool in this calculator. Play with the spreadsheet. Add it in. Send me a copy so I can see how you look at the calculations.
Decision Making
Don’t make decisions in your business about test automation based on what this calculator tells you. Instead, have conversations. It’s meant as a conversation starter. When you’re ready to make decisions connect with us and let us help you plan.
Finally
Thanks for joining our webinar on ROI yesterday (10/30/2019). We had some amazing questions and really wonderful insight shared by you all in that webinar. I appreciate it and look forward to more discussion in the comments below. What did I miss? How do you see us making a new version of this calculator be more helpful? What discussions did your team have around ROI and the value automation brings as a result of this webinar and calculator?