HITS

In a constantly shifting marketplace, new OTT streaming providers are entering the market, subscriber churn continues to be high, the number of devices that OTT platforms run on is increasing, and consumers continue to drive an environment of viewing anything, anytime, anywhere.

OTT providers face great challenges when it comes to catering to their current and prospective consumer audiences, and this means ensuring that their streaming platform is stable, performs at its best, and is free of glitches.

How can OTT platforms ensure that functionality remains the same irrespective of the device that it is accessed from? How can platform performance be measured across industry standard KPIs?

OTT testing and benchmarking can alleviate these challenges by supporting a quality customer experience by testing for security, connectivity, consistent and high-quality delivery, data transfer rates, compatibility with all playback devices, and more — essentially validating all functions of an app while it is performing.

BENCHMARKING FOR THE ENHANCED USER EXPERIENCE

A core element of testing is benchmarking, whereby essential metrics help an OTT provider know where its app stands, unlocking insights to help make it successful.

Benchmarks compare an app’s performance to the industry average, track key metrics and performance indicators, and can quickly identify red flags.

These key metrics give OTT providers an opportunity to see where they might be underperforming compared to other streaming companies.

However, manually benchmarking more than a few systems or devices at a time quickly becomes time-consuming, mundane, and challenging.

Automated benchmarking makes this much easier, quicker, and often more accurate.

AUTOMATED ANALYSIS

With automated performance analysis, service providers can leverage benchmarking to pin-point certain areas of an application that may need further testing, or areas of an app that may benefit from automated testing, providing a health analysis of how a platform is performing. An automation framework should support all known browsers, set-top boxes, mobile and smart tv devices.

Regardless of whether benchmarking and testing services are in-house, or offered in a fully managed, hybrid-managed, or remote-hosted environment by a service provider, a typical remote framework will allow users (OTT platform provider testing and automation team) to remotely access live devices and create/manage/schedule automation scripts.

Automation QA teams (in-house or 3rd party testing service providers) ideally should work closely with app development teams to challenge each other, build out automation frameworks for new and interesting test use cases, and develop their own best practices for the best possible end-user (consumer) experience.

AUTOMATION STRATEGIES

How do we identify a sensible approach to which types of testing can or should be automated?

Here are just a few use cases that are appropriate for automation; if tests result in human error, tests that are labor-intensive to run, tests that need to be performed on many hardware and software platforms, and tests that simply cannot be performed manually.

Navigating manual vs. automated processes and testing asks us to analyze the types of scripts and scripting techniques that should be used, what automation is capable within a desired tool or framework, and what types of performance measurements are usable in automated testing scenarios.

Plus, building an automation strategy around what tools to use, what frameworks to select, what automation mistakes to avoid, and how to leverage an existing in-house testing infrastructure all play a role.

The wide range of OTT devices across the whole ecosystem presents an ongoing challenge to development teams, and automation is now able to support all these platforms and devices under one umbrella offering key performance metrics across the board.

This can be where automated benchmarking can truly shine, helping to identify a platform’s defects before a customer does! Automation software frameworks can be suitably customized, offering capabilities far beyond standard automation.

Plus, automation software with its associated dashboards offers developers, as well as executives, a more visually pleasing way to analyze performance indicators, areas to improve, and provides a good competitive landscape.

As organizations continue to build out their IP video service offerings, OTT testing and benchmarking is a vital step before going to market.

However, building out an in-house testing infrastructure may be cost-prohibitive.

Automated benchmarking services provide an opportunity to build resilient automation scenarios, run parallel test scenarios, offer a visual toolset on a framework that removes the need for extensive coding, and provides meaningful results that ensure a quicker time to market, reduction in long-term QA costs, and the seamless delivery of content.

* By Kevin Lebre, Senior Information Technology Project Manager, Giant Worldwide *

=============================================

Click here to download the complete .PDF version of this article
Click here to download the entire Winter 2022 M&E Journal