How to Improve Your Continuous Testing While Balancing Velocity, Coverage, and UX Risk

How to Improve Your Continuous Testing While Balancing Velocity, Coverage, and UX Risk

/, Mobile Application Testing/How to Improve Your Continuous Testing While Balancing Velocity, Coverage, and UX Risk

A stage-based methodology

Continuous testing is one of the keys to the DevOps kingdom.  Your pipeline needs to move fast to keep up with ever-shrinking release schedules but you can’t afford to sacrifice quality or UX in the name of speed. The solution? During each stage of development, Development teams need to balance testing every scenario against the amount of time needed to generate meaningful test results.  However, it’s understood that a “test everything” approach isn’t practical;  therefore, you’re left with a balancing act for teams to negotiate. This blog focuses on a continuous testing methodology to determine which devices to test at each stage of development.  The highest-performing teams are the ones whose game plans match target platforms with each development stage; this stage-specific testing strategy is fundamental to meeting your fast feedback needs while ensuring a great UX.

Breaking Down the DevOps Team Processes by Stage:

  • Unit Testing
    • Developers execute unit tests to get fast feedback – “does the code I just wrote behave as expected? Is it ready for integration and more rigorous testing?”  Maximizing platform coverage in this stage is inefficient and unnecessary. In this early stage of development, unit tests executed before or after a commit often use emulators and simulators to provide a quick thumbs up or down on whether the code works. In later test phases, most top teams agree that moving to real devices is required to assure user experience.
  • Acceptance Testing
    • Teams typically focus on verifying that new functionality- as well as old-  works according to the user story, and tests are executed over a large set of platforms that mirrors realistic customer patterns.
  • Test in Production
    • Many teams adopt DevOps; testing in production becomes part of the continuous testing scope. Once code ships, the objective changes from “does it work” to “is it still working as expected?” Teams recognize the value of leveraging hourly testing of key flows to create an early warning mechanism. Early awareness of production issues jump starts resolution efforts while (hopefully) few users are negatively impacted.

Factors: Your Coverage Crib Sheet for Continuous Testing

We’ve established that it’s important to know which platforms to test against, in which environments, and when to execute, in order to streamline the continuous testing process.  Everyone involved in the product release should understand both the testing trigger points that must be defined in each stage and how their tests fit into the overall pipeline in order to meet project schedules and reduce UX risk.  Perfecto’s Factors reference guide gives you a head start with guidelines for determining which platforms you need to cover and how to fit them into your DevOps process.  The table below summarizes the Perfecto’s research.

Our methodology:

  • Unit testing should be executed by devs on a small subset of platforms that may include emulators and simulators, and should be triggered pre- and post-commit locally against the developer workstation.
  • Build acceptance tests should be executed on a larger number of platforms (real devices and web platforms) daily and as part of the continuous integration (CI) process.
  • Acceptance tests should be executed on the full set of platforms in the lab to get maximum coverage and quality visibility. These cycles should run on a nightly basis, orchestrated by the CI process.
  • Production testing should run hourly and continuously to detect regression defects, outages, or performance degradations in the service. Such tests should not focus on maximum coverage of platforms;  select the top 2-3 platforms from the web and mobile and execute against these.

Up Your Application Testing Game Plan

In this blog, we’ve taken a look at DevOps team quality objectives and highlighted the differences in coverage levels required at each stage. In addition, we provide a methodology for tailoring platform coverage for mobile and web for each development stage in order to enhance continuous testing and minimize UX risk.  Perfecto’s Factors reference guide provides invaluable insight into developing your testing methodology as well as the current data you need to make critical coverage decisions.  Whether you’re a dev tester, a developer, or an R&D manager, it’s a tool that you need in your toolbox.  Grab your copy today!

About the Author:

Eran Kinsbruner is the Chief Evangelist at Perfecto and Author of the Digital Quality Handbook and Continuous Testing for DevOps Professionals books. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at ADT, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional continuous testing blog at

Leave A Comment