For as long as consumers have been downloading apps, conventional wisdom has suggested that user satisfaction and download speed go hand in hand. The assumption made sense: home internet speeds are packaged and advertised by the megabit; the data speed increase has been the most notable user change in going from 2G to 5G; and, not insignificantly, mobile network providers have been running adverts based on speed for years.
But as networks have evolved, the average download speed has crept ever higher, while subscribers haven’t necessarily been getting happier with the quality of their mobile experience.
In a study by the Wall Street Journal, 53 journalists assessed their internet usage over a period of months by streaming popular services such as Netflix, Amazon Prime, and YouTube simultaneously. What they found was that "quality didn't improve much with higher speeds. Picture clarity was about the same. Videos didn't launch quicker." Fundamentally, the subjects of the study just weren't using most of their available bandwidth.
Similarly, a 2015 study from the Boston Consulting Group tested key network performance indicators against a survey of users’ happiness with their connection and found that even when streaming video, any increase in speed over just 1.5 Mbps saw little increase in user satisfaction.
Looking at the minimum requirements from streaming video providers, you can see why. Popular services like YouTube, Netflix, and Hulu — which account for a significant proportion of mobile data traffic — have a minimum requirement of 1 Mbps to stream SD video, while even 720p or 1080p streams only require a maximum of 4 or 5 Mbps.
So if download speeds aren’t a satisfactory proxy for a ‘good’ network experience, what is? Tutela has designed a set of metrics, named Consistent Quality, which use the widely-accepted network requirements of different use cases to categorize the quality of connections.
We measure six key performance indicators — the average download speed, upload speed, latency, jitter, packet loss, and time to first byte — and calculate what percentage of tests meet or exceed our two thresholds, Core and Excellent Consistent Quality.
In addition, we have added “reliability” into Consistent Quality – that is, the percentage of tests attempted which did not succeed due to a connectivity issue on either the download or server response component. These will contribute to the total percentage of Consistent Quality tests that are considered to have failed both the Core and the Excellent Consistent Quality thresholds.
The Excellent Consistent Quality thresholds have been designed using the latest available network performance recommendations from services like Netflix, Skype (including Skype for Business) and others, as well as emerging use cases such as live video streaming on services like YouTube and streaming video games.
|Average download speed
|Average upload speed
|Time to first byte
Core Consistent Quality is designed to represent common but less-demanding use-cases than Excellent Consistent Quality, specifically standard-definition video streaming, web browsing, email use, and photo-sharing apps like Snapchat and Instagram.
|Average download speed
|Average upload speed
|Time to first byte
Including download and upload speeds in any test remains important: even if it’s not the only thing that matters, a 128 kbps connection still won’t work for any of our most important apps. A download speed of 5 Mbps is sufficient to stream HD video from any of the most popular streaming services, while the Core Consistent Quality threshold of 1.5 Mbps is in line with streaming standard-definition video — and, in many cases, is the maximum speed that some carriers will allow standard-definition video to be streamed at.
Users download significantly more data than they upload, making a faster download speed the more important of the two, but our Excellent Consistent Quality upload speed threshold of 1.5 Mbps means users can conduct a video chat, upload images or videos to Snapchat, or send emails with large attachments without noticeable delay.
Once the throughput thresholds have been reached, latency is arguably the biggest difference-maker in user experience. Our threshold of 50 ms is fast enough for online games like Fortnite, as well as voice and video calling. At faster download speeds, lower latency is also the most significant factor in web page load time. Using the 100 ms latency threshold for our Core quality measurement ensures a connection is still usable, although subscribers will notice a difference in performance.
Jitter is a measure of the variation in latency, and has a major impact on the quality of real-time services like voice and video calling. A low average latency is fine, but if the packet latency jumps between 25 ms and 75 ms regularly, users will see video artefacts and jumpiness, which degrades the experience. The excellent quality jitter threshold of less than 12 ms ensures that operators are achieving a consistently low latency, rather than just a low average.
Packet discard is the fifth KPI to be measured by our Consistent Quality percentages, and reflects the percentage of data packets that fail to arrive at the destination. It has the greatest impact on video calling, where missing packets result in jumps in audio and video, but at a high percentage, can also cause noticeable slowdowns in web browsing or data downloads. Research also suggests that sufficiently high packet discard can cause artefacts and diminished quality of experience for streaming video. More demanding use cases are less tolerant of a high packet discard percentage, which explains the low threshold for packet discard on our Excellent quality measurement.
Time to first byte represents the time taken for a connection to be established. Adding this KPI helps to represent the full experience of the user from the moment they attempt to connect to a service. This is because a slow initial connection time will make the download experience feel slower, even if the speed is fast once the connection is made. The time to first byte thresholds is set in line with the required download speed for each tier of Consistent Quality.
As consumer habits change and applications continue to improve, the definition of a "good" network connection will also continue to change. Tutela continues to monitor the network requirements of popular applications, as well as engage with industry experts on the measurement of end-user Quality of Experience. As a result, the thresholds used for Consistent Quality are subject to revision in order to provide the most accurate representation of Quality of Experience. The Consistent Quality standards were most recently revised on September 1st, 2021.