After gathering feedback from customers, we have found the main problem companies usually face when building and sustaining a mobile device lab is the size of their lab- matching the device list to their organization’s requirements and sizing the lab accordingly. If the lab size is not accurate, improving test cycle velocity and overall quality will be difficult.
In this post, we’ll highlight one approach to sizing a mobile device lab based on specific requirements. To help with your lab sizing, we’ll use the following calculations:
- Number of automated test cases
- Average duration of a single test automation
- Test cycle time constraints (e.g. complete automation regression took XX hours)
- Number of unique devices for the lab
- Test automation suite classification/prioritization (How many tests are critical? How many are low priority?)
Let’s add some numbers to these metrics.
Assuming a customer has defined 16 unique devices (iOS/Android) for their mobile project, and the time constraints for the test automation cycle is 18 hours. Each test is estimated to take 10 minutes end-to-end (including test/device setup, execution, cleanup and moving to the next case), and the test suite consists of 400 different tests.
Some simple math for the above would look like this:
As seen above, the initial number of 16 devices cannot meet the required QA cycle duration of 18 hours. To meet this required cadence, a team needs to multiply the set of devices by four and size the lab with up to 64 devices.
Organizations that face this lab sizing challenge can optimize their mobile testing efforts by categorizing both the devices/OS list and the tests themselves into different buckets.
As recommended in the image above, teams can create two buckets within the list of 16 unique device/OSes and call the buckets Primary devices and Secondary devices. Figuring out which devices go in which bucket relies on either analytics and web traffic trends or industry reports about test and device coverage.
As a side note, the mobile market is growing in a bigger phase than teams are “retiring” their support for legacy devices. This leads to an annual lab size growth YoY as we can anticipate in the below graphic:
To learn more, download Factors Magazine, a guide for testers that covers every aspect of planning, building & releasing apps.