With today’s rich set of tools and frameworks, it can be hard to decide which mobile testing framework to use. Fortunately, this question comes up a lot with Perfecto customers and other software teams, and we have a few considerations to help you make your own decision.
Consideration 1: Scope of mobile testing
Each testing framework satisfies a unique set of challenges and capabilities. For instance, xcTest and Espresso are unit testing frameworks for entirely separate platforms (iOS and Android respectively). While they provide deep support for their own platform, they quickly lose value when used at the unit validation level on cross-platform app projects.
Read our new e-book: Best Practices for Expanding Quality into the Build Cycle
Developer-friendly unit testing frameworks primarily focus on capabilities that solve the “depth” problem (i.e. deep support for a specific platform) and often not the “breadth” problem (i.e. support for multiple platforms). This is not an issue so long as you scope your expectations over a unit testing framework. In other words, don’t ask a microscope (white box unit testing) to do the job of a voltmeter (black box UI testing).
Consideration 2: Fit the tool to the project(s)
Appium is a good example of a cross-platform test framework that doesn’t provide deep instrumentation detail but does provide rich syntax support across a variety of languages through its underlying use of Selenium WebDriver. Spanning across multiple platforms is impossible with xcTest or Espresso, and at some point these scripts cease to be “reusable” because you’d end up duplicating the same logical test for separate platforms with separate test frameworks.
Cross-platform test frameworks (like Appium and Selenium) pick up where uni-platform test frameworks begin to drop the ball. If you’re dealing with one or more projects that span multiple platforms, it’s best to acknowledge up front that developer-level unit testing should be done with a native framework and common UI functionality should be tested using a cross-platform test framework.
Consideration 3: Fit the tool to the skill
On agile teams with a mix of developers and test engineers, it’s normal for them to use more than one programming language or test framework. Teams with a mix of skills and knowledge have an advantage in that they have more options to be flexible to the needs of their business.
Speaking as a developer, I’m less inclined to spend time unit testing when I constantly have to switch from language to language or learn yet another framework. I’m not lazy — I’m human. Everyone is averse to pain. As a developer looking for a unit testing framework, I ask:
- Does it support the language I’m using to write the actual product (mobile or web app)?
- Does it get me the level of diagnostic detail I need?
- How good is the documentation and community support?
- Are there integrations/plugins to my IDE to further simplify use?
Conversely, test engineers performing UI functional testing are concerned less with diagnostic detail at the code level and more with simulating realistic use. Reuse of a platform-specific unit test becomes less feasible the more the breadth of a script (cross-platform support) is valued. To pick a UI testing framework, consider the following questions:
- Does it allow the same script to run on all the platforms I need to support?
- Can I elicit the specific behaviors I’m targeting (i.e. iOS Force Touch) easily?
- Will I be able to script tests in the language of my choice?
- Can the test run on simulated (VMs, emulators) and real devices without modification?
For a full comparison between the top five most popular frameworks, read this related Perfecto blog post: “Selecting the Best Open-Source Test Automation Tool for You.”
Consideration 4: Integration with other tools
Automation is a key driver behind testing frameworks, unit, UI or otherwise, to distill our validations of features and processes into a form we can set and forget. That is, until we do something that causes them to fail.
Therefore, you need to consider a mobile testing framework’s ability to plug in to your continuous integration (CI) environment. The reported flakiness of xcTest in Jenkins and Bamboo notwithstanding, you may need to develop a code-commit strategy that includes flagging tests that are consistently problematic for CI while compensating with black-box UI testing to maintain proper coverage.
Ultimately, development teams should have their own point of view. Before committing to one team standard, I recommend doing a proof-of-concept on each framework and see how efficient each approach is in light of the existing ecosystem of development tools.
Consideration 5: Reusability and longevity
Beyond the development and testing cycles, a great testing framework is reusable in multiple stages of the larger product lifecycle. Testing artifacts that can be used in production monitoring allow product teams to track the same measure of success from the developer’s machine all the way to real user environments. Developers can diagnose a production issue far faster when they have details on exactly which step failed in a test and the means to reproduce the error, rather than sifting through a bunch of logs and user anecdotes.
Using unit and UI functional tests in production monitoring reveals that testing artifacts can be used for more than just QA or development CI cycles; they are business artifacts that keep various teams in sync with what’s happening with their apps and systems. This embeds tests into the DNA of the business and maximizes the value of testing across the entire life of the app.
Final Thought – Testing accelerates time-to-market
The most efficient software teams take ownership and responsibility for the final result and optimize their workflows to reduce delays.
Testing helps us catch bugs; the earlier the testing, the less we delay downstream activities like UI and non-functional testing. We need different perspectives on quality, and various members of our software teams need to figure out how far they can go with their own mobile testing. Whether we commit to higher quality and/or faster delivery, testing helps us avoid tripping over our shoelaces and maintain a steady running pace.