Do you know why some web and mobile apps succeed better than others? Does anyone? We think we have some information that can help to answer these important questions.
As part of our ongoing search to find out what helps people make great apps, in April 2016 we asked technical experts and business leaders questions about how they find success in delivering digital experiences across mobile and web platforms. Some themes were clear, some assumptions were validated, but more importantly we found that many questions were still left unanswered.
The four main findings from the survey are:
- Delivering a reliable and consistent digital experience across multiple devices quickly enough to meet customer demand is challenging for technical teams of all sorts.
- Mobile app testing is not guaranteed to represent real-user conditions; minimal testing is done for user conditions such as traffic spikes, memory consumption and low battery.
- The rate of change in the mobile market, platforms, and technologies regularly outpaces teams’ ability to capitalize on these new opportunities.
- Building and maintaining an internal test lab is a challenge for most teams that want to stay up-to-date with their users’ expectations.
This article goes further than our initial findings, but if you want to read about the main challenges and our recommendations, download our full “Why Apps Succeed” report.
Why are mobile apps a mandatory digital channel?
In the past three years, mobile growth far surpassed other platforms as the channel through which people are realizing the value businesses provide. At a 90% increase in time spent between 2013 and 2015, it’s clear that the time consumers spend on-the-go is a huge opportunity for businesses to capture their attention in more effective ways.
Just think about all that time spent on the subway, waiting for meetings to start, standing in line for coffee — all moments where you can still be productive through your phone. Adding up those moments can be significant too; a 2015 study by Informate Mobile Intelligence shows that people in the U.S. are spending 4.7 hours per day on their phones. Unlike web activity, being on a cell phone is a very interactive activity, making it a better indicator of how people are consuming content and communicating with each other. Accepting meetings and sending quick responses can also have a positive impact on the time spent in the office. Mobile devices are the way to get things done wherever you are.
For businesses, mobile represents entirely new ways to engage consumers. Instead of trying to use old channels like email that ultimately distract people, mobile gives people a tailored experience that incorporates brand and value on the same device. In combination with a web presence, a multi-channel engagement strategy can yield higher rewards than only interacting via a single channel. However, delivering a mobile app presents more than just technological challenges; people and process often need to change.
How are digital teams meeting the demand to release faster?
From our survey of over 1,000 enterprise-level web and mobile professionals, we see a significant number of respondents answer that they have combined web and mobile teams. This makes sense in small shops where businesses literally don’t have any other option but to use the same people to develop and test their web and mobile assets. Yet while some still maintain these platforms through separate teams, over 50% of large organizations report that they take the same approach as 80% of small organizations, that of having combined web and mobile teams.
Why is the unified web/mobile team approach prevalent in organizations of all sizes?
- Small business only have a few people, so by definition, teams are combined.
- Combined teams facilitate collaboration earlier on and throughout the software lifecycle.
- Enterprises acquire small organizations that bring their combined team approach along with them.
- Approaches that don’t meet market demand (i.e. great apps) tend to get sidelined (ref: Resource Dependency).
There is no conclusive theme explaining why large organizations are split between combining teams and keeping them separate. In many cases where there is no mandate to adopt a methodology, teams self-organize based on the best approach.
How does an aligned technical strategy impact release frequency?
One thing’s for sure, a unified development and testing strategy is critical for larger organizations if they want to maintain their position in the market. Mainstream Agile implementations increase the speed of software releases and shrink delivery times to the point where waiting for another team for days is unsustainable. In many cases, project teams cannot wait for mobile and end-to-end manual testing by their own QA resources, and turn to automation to speed up the delivery process.
Of the 21% of respondents who release software 17 or more times in 2015, there were a few note-worthy attributes:
- We saw a 15% difference between those who provide a responsive web experience and release frequently (75%, n=145) and those who release less frequently (59%, n=759).
- One out of every four respondents said that they only automate around 10% or less of their mobile testing.
- Support for new devices is the number one driver for responsive web, less than consistency across devices in a digital experience.
Teams building responsive web apps and sites have greater flexibility over release frequency compared to native mobile app teams for two reasons: changes don’t have to wait to be approved through an app store and device owners don’t need to accept updates to receive the changes. Additionally, responsive web teams share a single codebase for both mobile devices and desktop browsers, encouraging a single-team approach that ultimately produces a more seamless user experience across devices.
Just over a quarter of respondents also said they automate 10% or less of their testing activities. In case you didn’t know, this is not enough automation. Automation enables timely regression testing and anything that is not timely (i.e. manual regression testing) usually gets cut out of the plan. With no regression testing and poor test coverage, defects tend to make it out into the real world, which take more development time to fix the longer the defects are left for users to find.
Test automation with proper coverage is a necessity if you want your development team to keep up with the pace of the business; it’s that simple.
Which mobile testing activities increase velocity?
The process of testing often leads to questions about the intentions and design of a mobile or web app. From the survey, we see that there was clear alignment across the board about what testing strategies help to increase the frequency of releases.
While everyone responded that the number one approach was to do more automation and unit testing, a high percentage (57%) of teams — no matter how often they release — cite that they are testing more devices in parallel. A common bottleneck to mobile testing is the unavailability of devices in an on-premise lab; there are only so many devices to go around and there’s an increasing number of triggers throughout the software pipeline to do testing. This has led to more software teams moving from on-premise device testing labs to cloud-based device providers.
Additionally, there was a 10% lift in respondents who release more often to rely on visual and navigation testing scenarios in responsive web testing. Though seemingly small, this lines up with our conversations with mobile development teams that have been failed by cloud-based device farms that don’t provide visual validation, screenshots, hardware information or detailed device status.
Modern development and testing activities need unimpeded access to real devices and user conditions to keep up with consumer demand. Shouldering the responsibility of an on-premise device lab takes the focus away from building awesome apps, and teams often choose to resolve this challenge by selecting a cloud-based device provider with the right capabilities ready to go.
Where to go from here?
On June 30th, we hosted a webinar with a few experts to discuss the findings in this report. There was a live Q&A session where attendees were able to join the conversation with me and the experts on hand.
Click here to listen to a recording of the webinar. You can also view the webinar slides below.