For as long as consumers have been downloading apps, conventional wisdom has suggested that user satisfaction and download speed go hand in hand. The assumption made sense: home internet speeds are packaged and advertised by the megabit; the data speed increase has been the most notable user change in going from 2G to 5G; and, not insignificantly, mobile network providers have been running adverts based on speed for years.

But as networks have evolved, the average download speed has crept ever higher, while subscribers haven’t necessarily been getting happier with the quality of their mobile experience.

Why speed doesn't tell the whole story

In a study by the Wall Street Journal, 53 journalists assessed their internet usage over a period of months by streaming popular services such as Netflix, Amazon Prime, and YouTube simultaneously. What they found was that "quality didn't improve much with higher speeds. Picture clarity was about the same. Videos didn't launch quicker." Fundamentally, the subjects of the study just weren't using most of their available bandwidth. 

Similarly, a 2015 study from the Boston Consulting Group tested key network performance indicators against a survey of users’ happiness with their connection and found that even when streaming video, any increase in speed over just 1.5 Mbps saw little increase in user satisfaction.

Looking at the minimum requirements from streaming video providers, you can see why. Popular services like YouTube, Netflix, and Hulu — which account for a significant proportion of mobile data traffic — have a minimum requirement of 1 Mbps to stream SD video, while even 720p or 1080p streams only require a maximum of 4 or 5 Mbps.

Introducing Consistent Quality 

So if download speeds aren’t a satisfactory proxy for a ‘good’ network experience, what is? Tutela has designed a set of metrics, named Consistent Quality, which use the widely-accepted network requirements of different use cases to categorize the quality of connections.

We measure six key performance indicators — the average download speed, upload speed, latency, jitter, packet loss, and time to first byte — and calculate what percentage of tests meet or exceed our two thresholds, Core and Excellent Consistent Quality. 

In addition, we have added “reliability” into Consistent Quality – that is, the percentage of tests attempted which did not succeed due to a connectivity issue on either the download or server response component. These will contribute to the total percentage of Consistent Quality tests that are considered to have failed both the Core and the Excellent Consistent Quality thresholds.

The Excellent Consistent Quality thresholds have been designed using the latest available network performance recommendations from services like NetflixSkype (including Skype for Business) and others, as well as emerging use cases such as live video streaming on services like YouTube and streaming video games.

Excellent Consistent Quality

KPI Average download speed Average upload speed Latency Jitter Packet discard Time to first byte
Acceptable value 5 Mbps 1.5 Mbps 50 ms 12 ms 1% 0.8 s

Core Consistent Quality is designed to represent common but less-demanding use-cases than Excellent Consistent Quality, specifically standard-definition video streaming, web browsing, email use, and photo-sharing apps like Snapchat and Instagram. 

Core Consistent Quality

KPI Average download speed Average upload speed Latency Jitter Packet discard Time to first byte
Acceptable value 1.5 Mbps 500 Kbps 100 ms 20 ms 4% 1.2 s

 

Understanding the Key Performance Indicators 

Including download and upload speeds in any test remains important: even if it’s not the only thing that matters, a 128 kbps connection still won’t work for any of our most important apps. A download speed of 5 Mbps is sufficient to stream HD video from any of the most popular streaming services, while the Core Consistent Quality threshold of 1.5 Mbps is in line with streaming standard-definition video — and, in many cases, is the maximum speed that some carriers will allow standard-definition video to be streamed at. 

Users download significantly more data than they upload, making a faster download speed the more important of the two, but our Excellent Consistent Quality upload speed threshold of 1.5 Mbps means users can conduct a video chat, upload images or videos to Snapchat, or send emails with large attachments without noticeable delay. 

Once the throughput thresholds have been reached, latency is arguably the biggest difference-maker in user experience. Our threshold of 50 ms is fast enough for online games like Fortnite, as well as voice and video calling. At faster download speeds, lower latency is also the most significant factor in web page load time. Using the 100 ms latency threshold for our Core quality measurement ensures a connection is still usable, although subscribers will notice a difference in performance. 

Jitter is a measure of the variation in latency, and has a major impact on the quality of real-time services like voice and video calling. A low average latency is fine, but if the packet latency jumps between 25 ms and 75 ms regularly, users will see video artefacts and jumpiness, which degrades the experience. The excellent quality jitter threshold of less than 12 ms ensures that operators are achieving a consistently low latency, rather than just a low average.

Packet discard is the fifth KPI to be measured by our Consistent Quality percentages, and reflects the percentage of data packets that fail to arrive at the destination. It has the greatest impact on video calling, where missing packets result in jumps in audio and video, but at a high percentage, can also cause noticeable slowdowns in web browsing or data downloads. Research also suggests that sufficiently high packet discard can cause artefacts and diminished quality of experience for streaming video. More demanding use cases are less tolerant of a high packet discard percentage, which explains the low threshold for packet discard on our Excellent quality measurement. 

Time to first byte represents the time taken for a connection to be established. Adding this KPI helps to represent the full experience of the user from the moment they attempt to connect to a service. This is because a slow initial connection time will make the download experience feel slower, even if the speed is fast once the connection is made. The time to first byte thresholds is set in line with the required download speed for each tier of Consistent Quality.

Consistent Quality Updates

As consumer habits change and applications continue to improve, the definition of a "good" network connection will also continue to change. Tutela continues to monitor the network requirements of popular applications, as well as engage with industry experts on the measurement of end-user Quality of Experience. As a result, the thresholds used for Consistent Quality are subject to revision in order to provide the most accurate representation of Quality of Experience. The Consistent Quality standards were most recently revised on September 1st, 2021.