With a plethora of testing solutions, whether commercial or open-source, some are selenium WebDriver based, some are proprietary, organizations are finding it hard to match the right tools for their cross-browser testing needs.
Bryan Osterkamp, Lead Technical Architect, Dev and Ops Infrastructure Architecture Division at USAA, made a very nice analogy for this selection process referring this as an infrastructure as a freeway.
“There are a few things about a freeway. For instance, once a freeway is built, it allows its motorists to go as fast as they want up to the speed limit allowed by the authorities. Now, people aren’t forced to go on the freeway. If they decide to, they can take side streets. It may take longer, but there is nothing stopping them from taking that path. Let’s also say this freeway has something a lot of us in Texas (and especially San Antonio) don’t like, and that is a toll”
As mentioned above, organizations have many options, but bringing these options back to the software terminologies, it always comes down to a mix of technical and organizational fit requirements.
As described in my recent eBook, these 2 segments of requirement fit consists of the following (each of them is described in details in the paper itself):
- Project/s complexity and organizational future roadmap
- Existing resources, skillset and budget allocated for the project
- Existing tool stack within the DevOps teams
- Test type and test practices adopted by the organization (BDD, TDD, COE, Agile etc.)
- What kid of organization executive dashboard are required as part of the project/s?
- Tool community size, support, documentation, contribution sources, latest commit and more
- SDLC process fit of the test framework – can the tool support the organizational practices?
- Level of feedback loops and reporting provided by the framework?
- Cloud, automation at scale capabilities
- Level of automation coverage capabilities (can the tool support HAR file, environment mimicking, accessibility testing, visual navigation testing, and more)
- Automation robustness built-in test framework capabilities (sync based testing tool vs. a-sync, page object model (POM) easy setup, environment setup wizard and more)
The recommended practice as identified in the eBook, is to not only choose the right tools based on a mix of technical and organizational fit, but also acknowledge that in some cases a mix of test frameworks would make the most sense from a test coverage and test requirements perspective (e.g. leverage Protractor with JSDOM or PhantomJS can address both unit headless testing as well as E2E acceptance or functional testing coverage requirements).
Choose your path to success or build your “infrastructure” as a freeway and allow your testing practitioners the freedom to combine both toll based freeways as well as side roads that can overcome freeway pitfalls. The end goal is to release as fast as possible a working high quality product that can be validated throughout the release pipeline by various teams with varying tool stack selections and skillsets.