The Collapse of Test Automation: Why Most Companies are Failing At It

The Collapse of Test Automation: Why Most Companies are Failing At It

/, Digital Experience/The Collapse of Test Automation: Why Most Companies are Failing At It

There’s no doubt that most of the organizations across multiple verticals are aiming toward advanced agile practices, DevOps and continuous deployment. They don’t really have a choice, due to the digital transformation that forces more frequent releases and appealing functionality to continuously be released.

Software Automation Reality

The reality, however, shows that the industry stands today on average ~20% test automation (The Broken Promise of Test Automation).

Source: WQR 2017-2018

There are common reasons that are driving this trend – some are related to limitations in test automation tools, skillset maturity, organization maturity curve and lack of best practices.

You can read more about these challenges in the below posts I’ve recently published on InfoWorld:



The Market Needs A Different Approach to Automation

The market needs a better working test automation solution, that can drive productivity in a smarter way and can increase the market average to a higher rate.

There are a few pillars in the test automation domain that if addressed properly, can support the automation percentage growth:

  • Better object and visual analysis to drive automation and compare the app across platforms automatically (Test authoring support).
    • Flakiness is in many occasions attributed to poor object locators used during the test code
    • Visually identifying UX glitches per each app version can enhance test automation stability as well as increase quality

          An Example for iPhone X UI Glitch on the WWDC Native App

  • Efficient test automation suite maintenance that guides based on rules which tests to include in a test cycle, which to remove, which to “fix” etc.
    • As product matures, tests become either obsolete, or in some cases duplicated and therefore useless. Being able to automatically scan through suites, compare to user stories, and drive decisions can dramatically improve the coverage and productivity of both developers and testers.
    • Learning from production the end user experience, and guiding the right tests against the right platforms is another great step forward
    • Code coverage analysis (SonarQube) should also be part of the solution and continuously report on blind spots, inefficient tests, uncovered UX flows and more.

  • Addressing the test execution throughout the entire pipeline is another key element. In the above we addressed the “what to run” on “what platforms”, and in this pillar, the solution should advise the frequency of “when to run” these tests – Per commit, each build acceptance tests, each regression etc.
  • Test automation backed by a “smarter” Lab is an additional key to enhance automation productivity and coverage.
    • Auto configuration of relevant environment – setup the “right” environment based on the test context is an error prone activity, automating it per the relevant environment can solve this challenge (auto config IOT environment, Banking database, connected cars, unique engagement environment etc.)
  • Continuously measure and provide dashboard with metrics on cross organizational automation efficiency and health is another way of controlling automation maturity and continuously improving on it.

Bottom Line

As the market moves closer and closer toward digital engagement and adds more layers of complexity as part of the use cases, the more platforms, automation tools and methodologies also need to adapt. Having a low level of automation means an even lower automation level in the future unless something changes in the tools, discipline and methods.

As a DevOps organization, we  recommended to identify what’s blocking your teams from moving faster, and see what required changes (perhaps pick from the above , list) that can help mature your organization velocity.

To learn more how to avoid falling into the pitfalls of test automation, checkout this blog post.

About the Author:

Eran Kinsbruner is the Chief Evangelist at Perfecto and Author of the Digital Quality Handbook and Continuous Testing for DevOps Professionals books. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at ADT, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional continuous testing blog at

Leave A Comment