Businesses today are rapidly moving towards a digital engagement channel with their customers, their employees, their partners, and their vendors. This shift to digital is not only driving productivity and efficiency gains, but its disrupting existing business models. Just think of what iTunes has done to Tower Records, Netflix to Blockbuster, and more recently Uber to taxis. Each were caught off-guard by a rapid shift to digital customer engagement that disrupted existing engagement channels. With the advancements on the horizon related to connected car, IoT, block chains, and more, will your business be ready?
This shift to digital is driving a revolution and IT departments are leading the re-architecting of their business with digital channels. With the ability to engage end users instantly, anywhere in the world, user expectations for a personalized experience will determine the winners and losers. No longer does one engagement channel work. It is a requirement to engage end users across a digital Omni channel, where mobile and web are the primary touch points. Thus, a digital lab capable of bringing the real user experience into the hands of developer and testers is required to ensure quality engagements.
In this blog, we explore the complexity and costs associated with building a centralized digital lab, consisting of real devices and browser platforms. You can follow along, or design your own lab using Perfecto’s online lab calculator.
We will begin by exploring what is required to support a local or globally distributed app team and then dive into the components required to build a functioning digital lab for those of you that want to dive in to how the numbers were calculated.
Right-Sizing the Digital Lab for Your Organization
At a high level, a digital lab provides access to real devices and browsers though an automation interface with a guaranteed level of uptime, or service availability, to support agile development of web and mobile apps. We will reference Factors Magazine as a guide to identify the digital platforms to include in our lab as a representative set. You are encouraged to define your own devices and browsers based upon your customer data. From the index, we will select the top 25 devices and the recommended 33 browser/operating permutations. We assume the lab has the capability to test across all devices and browsers concurrently. The cost calculations are estimated using Perfecto’s lab calculator and summarized in the screen capture below.
To illustrate how the lab would scale based enterprise needs, we ran two scenarios through the tool, and summarized in the table below. The inputs are highlighted in yellow, and the summary calculations are displayed in black. We looked at these two scenarios:
- Local app team (1 centralized lab operating at 8 hours x 5 days)
- Global app team (3 geographically distributed labs operating 24×7)
From the results, we can see that it would cost over a half million dollars to build and staff a digital lab as a shared resource for a localized app team. Nearly half the cost is from manpower needed maintain the lab for continuous testing throughout regular working hours. The automation service is assumed to use open source Appium and Selenium grids. The browsers mix of VMs and the devices are connected to commodity servers (Windows/Linux) and Mac hardware. The details can be found in the following sections for the interested reader.
To support a globally distributed application team, the scenario would be expanded to at a minimum of three digital labs, to cover Americas, Europe, and Asia for example. Note that in this scenario, each lab operation has been expanded to 24×7 to support the global nature of the distributed team needing access to the labs. This allows the teams to run nightly automated test outside of the normal 8-hour workday in each region to support a global enterprise.
In this scenario, the cost scales to over $4 million, almost 8 times the cost of a local lab, driven primarily in the labor to support 24×7 operation of the global labs. Note that each region would also have a different mix of devices and carriers that represent that regions demographics as well. Looking at a cost of over $4M, you may be asking what about the cloud? Moving to the cloud would save you over 50% or more of this cost and would vary depending upon your specific needs. If your convinced the cloud is for you, check out our subscription plans to get started, or contact us to price a custom cloud hosted lab for your enterprise.
If you’re still considering building your own digital lab, continue reading through the next sections. To understand what the numbers are composed of and the assumptions behind them, let’s dive into the details. We’ll use the local app team as the reference to explore.
Breaking Down the Lab Components
At a minimum, a digital lab requires seven components.
- Devices – representative distribution of devices covering user’s hardware, OS, OS version
- Browsers – representative distribution of browser types, versions, OS, and OS version
- Computers – a mix of PC and Mac hardware
- Networks – Access to Wi-Fi, wired Ethernet, and cellular
- Automation – servers and software to coordinate and execute tests
- SAN – storage area network to save test logs, reports, videos, and screen captures
- People – maintain and monitor operation of the lab to desired level of availability
Devices – Why Does a $620 Device Cost $6,200 in a Lab?
Devices under test are more expansive than you think. It costs around $6,200 per device to host a device in a lab. That is nearly 10X the cost of the device. Why is it so expensive? It’s not as simple as connecting devices to a laptop. You need infrastructure and people to care and feed the devices, such as what’s shown in the image below.
A device lab consists of devices, servers, network, rack infrastructure (power, connection, Ethernet), health and safety certifications, and people to maintain the lab. The table below summarizes the expenses. Note that the capital expenses are typically amortized over 3 years and reported in the Annualized Cost column as the actual yearly cost of the device lab. Year 1 total reports the cost if all capital was purchased outright in year 1.
Let’s drill down into each area.
Mobile devices are designed for consumer use, not for continuous automated testing under load. When placing a real device into a testing scenario, the average lifespan is around 9 months. Therefore, a device lab of 25 devices, would need to be replaced on average every nine months, or approximately 37 devices in total[i] throughout the year.
Each device is assumed to have a carrier contract at an average price of $55/month.
Hardware to access and control the devices consist of a mix of Mac and PC servers in the ratio of 1 Mac mini to 4 iOS devices, and 8 non-iOS devices per PC. This gives a requirement of 2 Mac and 3 PC’s to support testing on 25 devices.
Infrastructure to support the devices consist of Wi-Fi access points, cellular repeaters (to support adequate cellular signal), and racks to host the devices and servers. One Wi-Fi access point can support up to 50 devices. An average cellular repeater can support 8 devices. The lab would require one Wi-Fi access point, and 4 cellular repeaters. One rack would be adequate to support 25 devices.
Care and feeding of the device lab and computers are required to maintain operation for automated testing. The devices require 3 hours/month/device on average of maintenance, diagnosing and fixing unresponsive devices, configuration issues, and so on. Similarly, Mac and PC servers need maintenance at a rate of around 2 hours/month/server.
And finally, if more than 20 devices are connected, a RF inspection to certify the lab has safe RF levels of operation needs to be completed. This is an annual cost of around $10,0000.
The net result is that it costs $6,200/device, or $156K annually for the full device service. This is just to host all the devices as a service. We have yet to add into the ability to automate testing. We’ll get into that, but first let’s build out the desktop browser-as-a-service component of the lab.
Browsers – Why Does a Free Browser Cost $2,700 in a Lab?
A browser lab is just a collection of virtual machines, right? Yes and no. OS X complicates the standard model of spinning up a VM. Apple requires Mac hardware to host OS X. IT organization generally do not support Mac hardware for VM hosting or have the capability internally. So, Mac hardware needs to be introduced into the environment. A lab would look like the representation below.
Supporting browsers on OS X is more complicated, requiring Apple hardware and limitations on the number of OS X virtual machines that can be run off Apple hardware. In this case, we’ll assume using Mac minis, and OSX browsers are offered in a 1:1 ratio, to allow concurrent execution of testing on each machine. 8 Mac minis are required to support the OSX browsers requirements for the local app team lab.
For Windows and Linux based browsers, commercial VM software and servers can be used. For the analysis here, we assume one server will support 8 concurrent browsers.
The result is an average browser VM cost of $2,700, or $90K annual costs for the full browser service.
Now Automate with Appium and Selenium
Setting up an automation environment is the final step to enabling the lab to for practical testing. The automation environment here will assume setting up open source Appium and Selenium grids with a storage area network (SAN) to save test logs, screenshots, or video recordings. For the SAN, it is assumed that 3 TB of storage per year will be required for the 25 devices and 33 browsers under test, at an average cost of $6/GB. Installing and configuring Appium and Selenium is assumed to take 1 day each to implement of manpower. The table below breaks down the cost for setting up and maintaining the selenium grid, with annual re-installation of new releases assumed. Setting up the grid and maintaining it adds about $20K to the total.
Keeping the Lights On
The lab to this point costs around $260K per year. But it isn’t guaranteed to be running 8 hours x 5 days a week. There isn’t anyone monitoring the operation or troubleshooting issues. At a minimum, the lab needs two people to monitor (Ops) and maintain (IT) the operation of the lab to provide the test team with assurance every device and browser will be available on-demand for testing during normal working hours. This is a substantial increase, nearly 50%, bringing the total to $500K.
The key assumption here is that the lab must be always on during the work day to support an app team working in an Agile/DevOps enabled environment (i.e. testing in and outside of continuous integration processes). The reality is that the current assumption does not provide support outside of normal working hours, so nightly testing is not accounted for. This would add to the cost but was ignored for simplicity.
What We Have Not Considered
We’ve built a test capable device + browser lab. However, this lab is still incomplete to help you achieve your test coverage goals. You need to consider real user conditions, such a device rotation, network conditions, locations, audio/video support, and more. The infographic below provides a bigger picture of what is required to achieve high digital test coverage. The lab we’ve priced out here will help you achieve, optimistically, around 60% test coverage. But it will leave you exposed to potential blind spots related to real user conditions. To achieve 80-90% you’d need to add the ability to control the environment to mimic real user conditions through scripting (to support automation). There is no open solution available here, so you’d have to invest in R&D to add this capability to your own lab. The final gap to close would be the ability to perform manual testing to test the remaining flows that you cannot automate. This adds the requirement to be able to remotely access and interact, from any location in the world, the devices and browsers available in the lab. Both add substantial cost to get you a modern digital lab.
Oh, I also forgot to mention, what about adding a reporting solution – to reduce the time it takes to analyze test results, perform root cause analysis, and view quality across digital platforms?
Are you ready to build your own lab?
Make sure to check out our lab calculator – and really find out the cost and effort that goes into building your own lab vs. a cloud based approach.
[i] Note that this assumes 10% of the devices are lost or broken in addition to average life span of 9 months.