For the past 20+ years, I have been part of teams that have built test automation frameworks and suites that have resulted in bringing in huge efficiencies to the project. I decided to write on this topic based on the recent experience from one of the clients, who has a multi-vendor test automation team in place but has not realized any benefits in the last 2 years.

Even before we venture into this topic, I think a very basic understanding of test automation is required – “Test automation is a method in software testing that makes use of special software tools to control the execution of tests and then compares actual test results with predicted or expected results”. This is something every software testing professional knows. The part that is unstated is that “the tests are done automatically with little or no intervention from the test engineer”. In essence, a test automation exercise is going to bring in efficiency gains only if the tests are executed in an unmanned fashion. With this background, I shall pen down the top 5 things that make or break test automation – these are absolute basics, does not matter whether you have 2 or 20 years’ experience in software testing.

Requirements Stability

No matter how skilled your resources are, no matter how well requirements are managed in the agile project, unless the application is stable in terms of requirements, no automation exercise can be successful. Today, everyone looks at automating all the tests from the previous sprint – however, when there are changes to the requirements that were rolled out in earlier sprints, there is a huge effort that needs to be spent to maintain the test automation scripts. So, any application with unsettled / frequent changes to requirements is not a good candidate for test automation. This is very basic, but it’s hurting to see people trying to automate everything under the sun (especially for projects in agile mode) for the sake of automation.

Automation coverage

We need to be clear on what needs to be automated? 100% test automation is not desired and not recommended too. Common sense tells you to automate only those tests that are executed for every sprint or release. So, sanity tests, high priority tests and to an extent, portion of regression tests qualify to be automated. Believe me, one still needs to spend some manual effort to validate things before a release. So far I have not come across any team that has successfully carried out 100% test automation.

Apt automation frameworks

Even when the requirements are “stable”, they do undergo some amount of “change”! Leverage automation frameworks that help you leverage benefits like code reusability, greater repeatability, low effort for code maintenance and standardized reporting. There are plenty of automation frameworks available to be leveraged for applications under test like data-driven, keyword driven, modular, linear and hybrid – based on the type of application and the functionality that’s being validated, the right kind of framework needs to be chosen.

Automation of UI as well as Non-UI tests

Today, 90% of the automation is UI driven. The common trend observed is to use Selenium web driver for web application and Appium for mobile applications. However, there are lots of defects that can be captured in the API layer – none of the new generation applications work without an API layer and there is a lot of non-UI testing that can be accomplished in that layer to deliver maximum defect unearthing. Everyone would have their own preferences, but I have found that a 60% UI automation combined with a 40% API automation provides the optimal mix when it comes to spending effort and extracting a return on investment through automation.

Unmanned automated test execution

Any automated test execution needs to be unmanned test execution. If we have an automation team sitting in front of their laptops to monitor all the automated test runs, we may not really derive any benefits out of automation. One of the best practices that has emerged out of the agile based product development is CI/CD pipeline usage. Today, there are plenty of solutions like Jenkins that allows automated test execution to be kicked off as soon as a build is complete. Leveraging this is absolutely important to experience the truly unmanned test execution.

Do you think there is more to the success of test automation than what is stated? I am sure there are more. Let us converge on achieving maximum benefits of test automation. Please do share your views and suggestions.