For as long as consumers have been downloading apps, conventional wisdom has suggested that user satisfaction and download speed go hand in hand. The assumption made sense: home internet speeds are packaged and advertised by the megabit; the data speed increase has been the most notable user change in going from 2G to 4G; and, not insignificantly, mobile network providers have been running adverts based on speed for years.

But as LTE networks have evolved, the average download speed has crept ever higher, and 5G coverage slowly begins to spread, subscribers haven’t necessarily been getting happier. 

Why speed doesn't tell the whole story

In a study by the Wall Street Journal, 53 journalists assessed their internet usage over a period of months by streaming popular services such as Netflix, Amazon Prime, and YouTube simultaneously. What they found was that "quality didn't improve much with higher speeds. Picture clarity was about the same. Videos didn't launch quicker." Fundamentally, the subjects of the study just weren't using most of their available bandwidth. 

Similarly, a 2015 study from the Boston Consulting Group tested key network performance indicators against a survey of users’ happiness with their connection, and found that even when streaming video, any increase in speed over just 1.5 Mbps saw little increase in user satisfaction.

Looking at the minimum requirements from streaming video providers, you can see why. Popular services like YouTube, Netflix, and Hulu -- which account for a significant proportion of mobile data traffic -- have a minimum requirement of 1 Mbps to stream SD video, while even 720p or 1080p streams only require a maximum of 4 or 5 Mbps.

Introducing Consistent Quality 

So if download speeds aren’t a satisfactory proxy for a ‘good’ network experience, what is? Tutela has designed a set of metrics, named Consistent Quality, which use the widely-accepted network requirements of different use cases to categorize the quality of connections.

We measure five key performance indicators -- the average download speed, upload speed, latency, jitter, and packet loss -- and calculate what percentage of tests meet or exceed two thresholds. 

The Excellent Consistent Quality thresholds have been designed using the latest available network performance recommendations from services like Netflix, Skype (including Skype for Business) and others, as well as emerging use cases such as live video streaming on services like YouTube and streaming video games.

Excellent Quality

KPI Average download speed Average upload speed Latency Jitter Packet loss
Minimum acceptable value 5 Mbps 1.5 Mbps 50 ms 30 ms 1%

 

Core Consistent Quality is designed to represent common but less-demanding use-cases than Excellent Consistent Quality, specifically standard-definition video streaming, web browsing, email use, and photo-sharing apps like Snapchat and Instagram. 

Core Quality

KPI Average download speed Average upload speed Latency Jitter Packet loss
Minimum acceptable value 1.5 Mbps 500 Kbps 100 ms 50 ms 5%

 

Understanding the Key Performance Indicators 

Including download and upload speeds in any test remains important: even if it’s not the only thing that matters, a 128kbps connection still won’t work for any of our most important apps. A download speed of 5 Mbps is sufficient to stream HD video from any of the most popular streaming services, while the Core Consistent Quality threshold of 1.5 Mbps is in line with streaming standard-definition video -- and, in many cases, is the maximum speed that some carriers will allow standard-definition video to be streamed at. 

Users download significantly more data than they upload, making a faster download speed the more important of the two, but our Excellent Consistent Quality upload speed threshold of 1.5 Mbps means users can conduct a video chat, upload images or videos to Snapchat, or send emails with large attachments without noticeable delay. 

Once the throughput thresholds have been reached, latency is arguably the biggest difference-maker in user experience. Our threshold of 50ms is fast enough for online games like Fortnite, as well as voice and video calling` At faster download speeds, lower latency is also the most significant factor in web page load time. Using the 100ms latency threshold for our Core quality measurement ensures a connection is still usable, although subscribers will notice a difference in performance. 

Jitter is a measure of the variation in latency, and has a major impact on the quality of real-time services like voice and video calling. A low average latency is fine, but if the packet latency jumps between 25ms and 75ms regularly, users will see video artefacts and jumpiness, which degrades the experience. The  excellent quality jitter threshold of less than 30ms ensures that operators are achieving a consistently low latency, rather than just a low average.

Packet loss is the fifth KPI to be measured by our consistent quality score, and reflects the percentage of data packets that fail to arrive at the destination. It has the greatest impact on video calling, where missing packets result in jumps in audio and video, but at a high percentage, can also cause noticeable slowdowns in web browsing or data downloads. Research also suggests that sufficiently high packet loss can cause artefacts and diminished quality of experience for streaming video. More demanding use cases are less tolerant of a high packet loss percentage, which explains the low threshold for packet loss on our Excellent quality measurement. 

Consistent Quality Updates

As consumer habits change and applications continue to improve, the definition of a "good" network connection will also continue to change. Tutela continues to monitor the network requirements of popular applications, as well as engage with industry experts on the measurment of end-user Quality of Experience. As a result, the thresholds used for Consistent Quality are subject to revision in order to provide the most accurate representation of Quality of Experience. The Consistent Quality standards were most recently revised September 1st, 2019.