Do you suffer from low test automation coverage for code or devices? Even skilled app development teams hit a wall when trying to improve their delivery speed and app quality. Swamped with an existing backlog of defects from the last release, development teams have no time to deal with broken automated tests, so they often disable the tests so they can move on.
Good news though: resilient automated tests can help improve speed and quality.
I’m picking a number here, 30%, as an arbitrary proxy for your own number. It may be 50% or even 70%, but most teams run into similar problems when making the leap from low to higher test coverage. A major roadblock is “flaky” tests that inconsistently fail when run in the continuous integration or build verification testing phases.
What makes for flaky tests? We commonly see:
- Devices aren’t reset to “clean” state before executing tests
- Tests don’t accommodate for differences in permissions between OS versions
- Pre-conditions that fail on tests that depend on other tests to run beforehand
- Direct dependency on hardware that differs between devices and platforms
- Tests rely on deprecated aspects of device APIs
- Test data sets are out of sync with new functionality or workflow conditions
Rather than fixing what makes certain tests flaky, teams may simply fall back to manual testing that will ultimately allow more defects to reach the public. It all leads to the feeling that you can never “reach the hoop”, either some lofty automation goal set by upper management or a percentage arbitrarily called out by some tech writer on a company blog.
What’s the big deal? It’s just a test….
Testing provides feedback about the impact of code changes. Without testing, we have no visibility on the risks we’re incurring as we improve our apps. Though manual testing is cheaper in one-off scenarios, well-written automated tests pay for themselves in the long run. We automate tests so we can run them at will when we commit changes that kick off a build in continuous integration or as part of larger regression testing.
Read our new e-book: Best Practices for Expanding Quality into the Build Cycle
Disabling a test is the first sign that a pain needs to be dealt with. It’s a payday loan that’s bound to bite you in the leg at the worst possible time in a future release. Instead of disabling flaky tests, we need to see them for what they are: stumbling blocks to automation that we need to prioritize and overcome to achieve faster releases.
If we ignore the feedback we receive from testing, we risk slowing releases by passing down defects to later phases of delivery or worse, breaking the experience for real users. Poor quality digital experiences disrupt critical revenue streams and even permanently damage corporate brands.
Why are teams stuck at less than 30% test automation coverage?
When asked by upper management about the percent of automated testing, development managers often report very low numbers like this one. Low automated test coverage doesn’t just look bad on a spreadsheet, it severely limits the speed at which development teams can deliver products. It’s a sign that something’s very wrong.
The story usually goes: “We tried to keep our tests in sync with code changes, but we moved too fast and tests broke, so we fell back to manual testing.” However, this makes things far worse because regression defects are often not caught by manual — let’s just call this what it is, “minimal” — testing. More defects take time to resolve, further reducing development speed and the ability to improve the customer experience. I call this the Downward Spiral of Defects.
Going back to teams that disable tests: “Flaky” tests are often the result of poor communication and/or a technology mismatch between development and testing. Scripted tests grow increasingly out-of-sync with code as specifications and visual elements are changed. Without a formal testing strategy, app teams lack the experience and foresight to address this pain effectively.
What does going beyond 30% test automation look like?
When there’s early feedback about how new features and code changes impact the real user experience, both quality and speed improve. Defects are often resolved quickly and effectively, impacting as few people as necessary.
As software teams seek to improve speed and quality, it’s become necessary to measure the efficiency of their continuous delivery pipeline. Code coverage metrics, build frequency, burndown charts and even adoption analytics from production all help to quantify the effect that a new feature or code change incurs on business and consumers.
What we see in “high performers” — organizations that outpace the release frequency of their competitors up to 200 times — is that app teams adopt a culture of quality as fundamental to their app release velocity. Practices such as exploratory testing encourage early feedback on new features and code changes. Rather than ignoring problems, competent teams foster a healthy competitive spirit and chose to “bring pain forward” to get problems out of the way before they multiply.
What do you need to move beyond 30% automation?
Testing technologies need to be reliable — The right tools also play an important role, especially when your build and release process depends on them. The more your velocity improves from fast feedback, the more you’ll want to automate. This will increase the demand and expectations you place on your execution frameworks and resources.
Autonomy to choose the best testing framework for you — Teams also need to choose what works best based on their app and infrastructure, and having to switch contexts or languages between code and test activities slows things down. Different test frameworks (such as Espresso, XCTest, Appium, and Selenium) have different advantages over each other. Therefore, it’s important to choose components for your software delivery pipeline that accommodate multiple skill sets, languages and testing frameworks.
Choose levels of testing that meet the needs of developers — However, tools are only part of the solution; a mature testing strategy maximizes code and target device/platform coverage by carefully applying tests that meet the development team’s velocity needs. Splitting up monolithic regression test suites to smaller, tagged test sets enables more comprehensive testing on every commit and minimize lengthy test-fail-fix cycles. Methodologies and practices must adapt to the needs of the teams and business, finding new ways to achieve the new goals while maintaining insight on how to improve the delivery process and the quality of the software.
An example of a view of coverage incorporating levels of testing and schedules that maximize coverage could be visualized as follows:
Have clear automation goals — A mature, adaptable testing strategy also includes automation goals that are easy to re-evaluate when changes occur. As shown above, each pie chart indicates a relative percentage of how much coverage might occur at each level of testing at each stage, and the resources required to accomplish that goal.
How does Perfecto improve your delivery speed and app quality?
We consistently hear from development teams that real devices are the best way to get real feedback, but the teams don’t want to manage a lab.
Perfecto knows the importance of providing highly reliable infrastructure, but equally important is rapid support and same-day device availability for development teams that just want to get back to the code.
If you’re trying to manage a homegrown lab, scaling to meet the demand of continuous testing can overwhelm your resources. Both the demand placed on your lab by continuous testing, debugging and troubleshooting as well as the variety of devices and platforms in the lab can outpace your ability to support and maintain your own lab:
The following testing demands also make a homegrown lab hard to sustain:
- Regression testing needs to include mainstream devices in the field
- Compatibility testing needs the latest and greatest devices as soon as they’re available
- Continuous integration increases the volume of testing and need for parallel execution
As the industry leader in test automation, Perfecto’s Continuous Quality Lab enables you to achieve more automated testing with support for the world’s widest range of devices and platforms. Perfecto also provides expert support as you grow to meet new automation and continuous delivery goals.