Measuring the ROI of Test Automation: A Starting Point Model
Blog  //    //  Written By:

Cost and ROI are two top considerations when it comes to implementing any kind of IT program at a hospital or health system. Based on our knowledge and experience with with Test Automation, we would like to provide our insight into the process of measuring the practices of the ROI of EHR Test Automation.

At Santa Rosa, we’ve helped many clients prepare a reasonable, realistic model to estimate test automation ROI, and one overarching lesson we’ve learned is: “Simplicity is the ultimate sophistication,” to quote Leonardo da Vinci.

This does not suggest that creating a complex and sophisticated ROI model shouldn’t or can’t be done. Instead, it means you have to be careful not to tie yourself up into a Gordian knot when doing so – remember, decimal point precision won’t build a stronger business case to justify test automation than a rounded estimate will.

How Do Healthcare Organizations Justify Test Automation?

There’s one more bit of essential context: the vast majority of test automation programs aren’t justified through reductions in existing testing staff.

In our experience, manual testers spend most of their time performing tedious, repetitive, and thankless work for the good of their organizations and the patients they serve. Test Automation is not a means for replacement, but a way to add enrichment to their jobs and use their skillset for higher-value activities.

Enter test automation. Follow the steps and process below or use our handy calculator to determine your Test Automation ROI.

The most common justifications for a test automation program are: 1) Risk reduction, 2) Cost avoidance.

Risk reduction is obvious – traditionally inadequate and “cut corners” manual testing exposes the organization, physicians, and patients to escalating dangers in terms of revenue cycle integrity, patient safety, and clinical care. Don’t just take our word for it – check out the many industry headlines over the past couple of years about EHR conversion and upgrade disasters. We’ve said it before: testing isn’t about technology or bugs. Testing is all about risk reduction.

And shedding high-cost processes are a foundation for what all healthcare providers should be focused on. Test Automation lends itself to this process via software automation. Let the software do the driving, providing valuable feedback, insights, and exportable video snippets that can be utilized for change management, learning engagement, and adoption but all staff members.

Step 1: Identify Current Costs of Testing

Estimating the ROI of a comprehensive test automation program such as Santa Rosa’s Test Automation as a Service (TAaaS) begins with finding an anything-but-simple answer to a simple question: what does it cost today to test the existing scope?

Most healthcare organizations have little idea what the actual cost of traditional testing is throughout their organization. They clearly see the direct cost of the testing team. However, they often fail to consider all of the other direct costs, such as the costs of SMEs and analysts who get pulled out of their “day jobs” to test (often much higher than the testers themselves), and the costs incurred on the backend when users have to spend time discovering, reporting, and working around bugs and other issues. Not to mention the cost of problems introduced into the revenue cycle, issues around patient safety, or patient care due to inadequate and unreliable testing processes. The following table summarizes what is included and what is not included within Santa Rosa’s starting point model for estimating test automation ROI:

Healthcare testing chart

It is common to find that the cost of SMEs and analysts equals to or exceeding the cost of testers. Certainly, this ratio varies by the type of system/platform and tends to be higher for EHRs.

It is essential (and realistic) to note that this model is per system and should be prepared separately for all major platforms – EHRs, ERPs, ancillaries, etc. In doing so, be careful to accurately reflect today’s actual testing cycles and labor content (while avoiding double-counting, where applicable). Don’t try to estimate these costs for all systems; focus on “bigger ticket” ones in your organization’s IT portfolio.

Step 2: Determine Costs Required for Additional Testing

Now that you have a sense of what testing actually costs today for your existing scope, you need to consider how to estimate all additional testing required in manual mode to achieve the desired value and impact for your organization.

There are many ways to do so, but we’ve found the most straightforward way mechanically is to increase the number of annual testing cycles and the duration of each cycle. It is an imperfect approach, but it is one that most people can get their heads around and ultimately support.

The tricky part, of course, is determining by how much to increase testing cycles and durations to achieve the comprehensive, rigorous testing that makes everyone sleep better at night. Remember Leonardo. Give it your best shot based on your rule-of-thumb, such as “we should complete X times additional testing” based on an approximation of today’s testing inadequacy and the risks to be avoided.

Step #3: Compare Manual Testing and Test Automation

All that’s left is comparing the cost of traditional, manual testing to that of an end-to-end test automaton program like Santa Rosa’s Test Automation as as Service. Feel free to calculate this based on the financial metrics commonly used within your organization. Our starting point model includes Net Present Value and estimated ROI.


Take the First Step toward Test Automation with Santa Rosa

The most successful test automation programs recognize that testing is about risk reduction, not staff reduction. As vendor upgrades/releases become more frequent in an increasingly complex IT ecosystem, risks to revenue cycle, patient safety, and patient care will skyrocket. Traditional, manual testing cannot possibly keep up unless your organization has the wherewithal to hire a vast army of new testers – and even then, that investment is likely to fail to deliver the necessary value and impact.

Even still, it’s tough to get started on a test automation program without creating an ROI estimate. That’s where our starting point model (and advice from Leonardo da Vinci) can help. Consider this food for thought when justifying a program to your CFO. And, if you need more information on Santa Rosa’s test automation solutions, contact us today.


Related Articles