There’s no doubt that the bulk of software delivery today relies on test automation through open-source test frameworks.
As a quick summary, the key points that were identified as critical for success in adopting an open-source is to have these 3 robust ingredients supporting that framework:
- 24×7 available and elastic Lab
- Advanced automation features
- Deep quality analysis post execution
In this post, let’s double click the 2nd item above – Advanced Test Automation
Extending Test Automation Coverage for Open Source Frameworks
While it is important to have support for the basic automation features in the OS level, it is also important to extend framework capabilities to support:
- Visual analysis
- User condition testing
- Full system level control
- Same day support for latest platforms
To give examples of why some of these extensions are important from a test automation coverage perspective, let’s look at testing an app using Espresso or XCTest UI – while these frameworks are great, fast, and support the latest features from an OS level perspective, they do not support the full system level. That limits testing scenarios which cover things like incoming calls, events, popups, and things like the recent iOS 10.3 2FA (2 factor authentication) that requires an acknowledgment of an incoming text message with code, etc.
In some cases, the test automation developers would like to run an additional validation of the policy and check how it appears when rotating the screen – are the buttons still visible? Are there any truncations of text?
Also, when testing such an app using Espresso and/or XCTest UI, the process requires 2 apps – one is the app under test (AUT). The 2nd is the testing app, which also requires validation that the initial install of the app triggered the policy screen and, upon a new test, won’t show up again (mostly from a test flakiness perspective).
For the above scenarios, it is important to state that most of the leading open-source test frameworks do not support visual test automation and analysis.
Consider Community Support for Open Source Frameworks
Finally, open-source is all about community. If the community is not strong enough, or is biased toward vendors, its value to end users will be weakened from the perspectives of features, adaptation to free tools and more.
A good source to consider when measuring open-source community strength is usually the github of these tools – how many branches? How many contributors, and where are they coming from? How many releases? With that in mind, let’s compare a few frameworks just to see…
As can be seen in the above examples, all of these communities are active, people are contributing and branching out of the master repos (not at the same volume of course) and each has different star gazer ratings and watchings
- Stars would measure the interest in that specific repository – by people using bookmarks.
- Watchers, would measure the amount of people that subscribed to receive notifications on new discussions around that repository.
If we consider the entire measures of contributors, releases, branches, stars and watchers we can learn a lot about the level of support, interest, and traffic that a community gets. This makes it easier to pick one framework over another – especially when there is a debate after teams examined things like the above functional comparison matrix across all tools.
A winning recipe is an open source test automation framework that meets the automation capabilities your app requires like; visual validations, environment conditions etc along with maintaining a robust and active community around that framework.