Video streaming is becoming one of the key elements of modern digital applications. They enable innovation, new user experiences and new business models in many verticals, from retail, FinServ, social, media, etc. The entire ecosystem is going through a revolution: the application architecture is changing – the delivery formats, screens and devices used to consume content, etc. I am joined by my colleague Stephen Dolan Sr, Product Manager at Accenture, to discuss some of the changes in this space and describe a new, combined offering from Accenture and Perfecto.
Amir: What is the problem statement of “yesterday” in the space of video quality that you are already solving, and who was doing video quality testing?
Stephen: For years, TV operators have lived with the high quality expectations of pay TV customers. The expectations of analogue TV customers – that the TV would ‘just work’ and that they can channel surf very quickly – was carried over to the digital domain. In fact the sales pitch for digital was that the picture had a much improved quality. So, this quality (both video and performance) was baked into the expectations and, as a result, testing on set top boxes has always looked to optimize the end customer’s viewing experience with respect to speed and quality. The stability of the offering was always hugely important, which meant change was very slow to come.
AR: What changed in regards to video-related offerings? What business objectives are driving this change?
SD: The TV space always had a hugely expensive barrier to entry (building out a network is a massively expensive undertaking, not to mention acquiring regulatory licenses and the cost of content). The business model was based on bundling up a large amount of content and charging a large fee, in the knowledge that in many cases there was very little choice in terms of TV providers.
However, now the incumbent TV operators are being challenged by the new kids on the digital block – the digital natives like Netflix, Amazon and Facebook. These guys don’t need to build out a network and they move and innovate faster than the TV operators can even conceive.
The customers have changed also – they no longer accept that the TV shows they want to watch are tied to either a physical location or a specific time slot. Responding to these needs has required TV operators to go toe-to-toe with the digital natives to prevent them eating their lunch. That means innovating faster than ever before and offering the content on whatever device the customer requires.
AR: What we’re also seeing is evolution of digital applications across verticals to include rich media experiences. These could include retail-oriented videos, such as product recommendations, reviews and assembly demos or product troubleshooting guides. Rich media content has impact on customers’ buying decisions.
Figure 1: Video’s impact on buying decisions (Source: Adélie Studios)
In social, these could be the likes of Facebook live, etc. Also, the evolution of technology enabled both ad-hoc content broadcast from individual contributors as well as the consumption of movies and series on mobile devices. Perhaps one of the common cases is in the broadcast of live sports content. Organizations such as telcos are innovating around specific content streamed live to audiences in the stadium. It would only be natural to say the common denominator for all these experiences is that the user expectation for video content should be latency-free and flawless quality.
2: Smartphone video viewing demographic stats
The bottom line is – more services involving videos, more devices used to consume videos, more user flows, more 3rd parties in the application architecture and (overall) more complexity.
AR: What are the implications for video quality testing? Who is testing video quality, on what devices, how frequently do they test it, etc.? Do you see a change in that sense?
SD: So, with this increased innovation and development, there has been a problem. These organizations were built for long lasting waterfall development cycles. There were large QA teams that normally had little or no direct communications with a) the developers, b) the network operators, and c) the end customer (via CSRs).
For many, the answer has been to move to an Agile development structure with an automated CI/CD testing flow. This works really well in the STB space using StormTest, enabling all functional, performance, quality and stress tests to be automated. It is typical to test video quality and performance metrics across all STB models and software versions deployed in the field.
In the mobile video space, it’s taking longer to get to the same point. Initially, the mobile app was considered a ‘gimmick’. An add-on to the main TV service that didn’t need to hit the same quality markers as the STB. However, that is no longer the case. Customers are paying for the service and they demand the same quality regardless of the device.
Figure 3: Poor video quality can severely impact brand perception (Source: Brightcove)
The fact that they can switch service provider so easily makes this point even more critical. Testing of mobile video apps needs to be embedded into the automated CD flow, in the same way as it is done for STB testing.
AR: Indeed. What we’re seeing is a transition in the timing for testing, responsibilities and requirements from the lab. Essentially, video quality is becoming one of the requirements to be tested continuously during the design, development, launch and production stages of the SDLC. Even if the coder does not have deep video expertise, they would run video-related testing on a nightly basis to gain immediate feedback to changes that were committed that day. If something failed, it either has to do with functional aspects of the app or to changes to the backend infrastructure (e.g. change of streaming service vendor). When the test fails, the coder may share the detailed log with the subject matter expert, but at least there will certainly be an early feedback loop to these types of tests as well.
From a lab perspective, adding video testing means that the lab needs to:
- Provide the necessary coverage and video input to the StormTest system in high quality
- Be continuously available and reliable so execution failures reflect the true nature of the app
- Be able to scale across devices and executions
- Facilitate big data visibility to efficiently analyze many executions and get to the root cause quickly
AR: What are some of your recommendations to be successful with Agile video quality testing?
SD: Video service testing needs to be treated in the same way across all devices deployed to customers. The key points when considering video testing are:
- Assess your test cases – ensure that you initially automate the tests that soak up most manual effort. It gives you an instant ROI that will help your business case.
- Automate everything, not just the testing. You need an automated CI/CD pipeline so that your people don’t waste time doing things a computer can do.
- Be clear what you are looking for when performing video quality testing. Are you comparing quality across releases? Across devices? Are you assessing the effect of changes in other components in the video chain on the video quality? What pain points are causing you to look at the quality of the video?
- Don’t forget audio – audio quality is at least 50% of the customer’s experience of the content. If you have great video but terrible audio, you’ve lost already.
AR: Thank you for sharing with us, Stephen.
Video quality represents the next frontier of continuous quality requirements needed for any modern digital application. Accenture’s new offering, StormTest solution expansion to control Perfecto onprem devices is an excellent choice for those looking to ensure high quality customer experience. To learn more about this solution, please sign up for our upcoming webinar.