Introducing Consistent Quality - measuring more than just speed


For as long as consumers have been downloading apps, conventional wisdom has suggested that user satisfaction and download speed go hand in hand. The assumption made sense: home internet speeds are packaged and advertised by the megabit; the data speed increase has been the most notable user change in going from 2G to 4G; and, not insignificantly, mobile network providers have been running adverts based on speed for years.

But as LTE networks have evolved and the average download speed has crept ever higher, subscribers haven’t necessarily been getting happier. In our 2018 US networks LTE report, Verizon had the fastest average download speed, with 22 Mbps; the slowest network, Sprint, saw an average download throughput of just over 8 Mbps.

Capture2

 

Why speed doesn't tell the whole story

The chasm between those two numbers may seem significant, but most users simply aren’t going to notice a difference. A 2015 study from the Boston Consulting Group tested key network performance indicators against a survey of users’ happiness with their connection, and found that even when streaming video, any increase in speed over just 1.5 Mbps saw little increase in user satisfaction.

Looking at the minimum requirements from streaming video providers, you can see why. Popular services like YouTube, Netflix, and Hulu -- which account for a significant proportion of mobile data traffic -- have a minimum requirement of 1 Mbps to stream SD video, while even 720p or 1080p streams only require a maximum 3 or 4 Mbps.

 

Introducing Consistent Quality

So if download speeds aren’t a satisfactory proxy for a ‘good’ network experience, what is? Tutela has designed a new metric, named consistent quality, which uses the widely-accepted network requirements of different use cases to categorize the quality of connections

We measure five key performance indicators -- the average download speed, upload speed, latency, jitter, and packet loss -- and calculate what percentage of tests meet or exceed two thresholds. 

Excellent Quality

KPI Average download speed Average upload speed Latency Jitter Packet loss
Minimum acceptable value 4 Mbps 2 Mbps 50 ms 30 ms ~0%

 

The first, called excellent quality, is designed with the most demanding current mobile use-cases in mind -- specifically, HD video calling and 1080p video streaming. For those connections, not only is a faster upload and download speed -- 4 Mbps and 2 Mbps, to be precise -- required, but also a much more exacting standard for latency and stability. With a connection that exceeds all five excellent quality KPI thresholds, users should expect to use any app on their phone with no perceptable quality issues. 

Basic Quality

KPI Average download speed Average upload speed Latency Jitter Packet loss
Minimum acceptable value 512 Kbps 128 Kbps 100 ms 50 ms 5%

For a basic quality connection, the target use-cases are less demanding: simple web browsing, applications like email, and VOIP calls. The requirement for VOIP calls means the latency still has to be relatively low -- less than 100 ms one-way -- but the tolerance for packet loss and jitter is much higher. Upload speeds are given less emphasis, needing to hit just a 128 kbps average, and download speeds are 512 kbps. 

 

Understanding our Key Performance Indicators

Including download speed and upload speed is the most obvious decision: even if it’s not the only thing that matters, a 128kbps connection still won’t work for any of our most important apps. A download speed of 4 Mbps is sufficient to stream HD video from any of the most popular streaming services, or download the 72MB Facebook app in under three minutes.

Users download significantly more data than they upload, making a faster download speed the more important of the two, but our excellent quality upload speed threshold of 2 Mbps means users can conduct a video chat or upload images to Snapchat or Instagram without a noticeable delay.

Once the throughput thresholds have been reached, latency is arguably the biggest difference-maker in user experience. In that same BCG study, a decrease in latency from 100ms to 50ms saw a 10 percent jump in user satisfaction. Our threshold of 50ms is fast enough for onlines games like Fortnite, as well as voice and video calling. At faster download speeds, lower latency is also the most significant factor in web page load time. Using the 100ms latency threshold for our basic quality measurement ensures a connection is still usable, although subscribers will notice a difference in performance. 

Jitter is a measure of the variation in latency, and has a major impact on the quality of real-time services like voice and video calling. A low average latency is fine, but if the packet latency jumps between 25ms and 75ms regularly, users will see video artefacts and jumpiness, which degrades the experience. The  excellent quality jitter threshold of less than 30ms ensures that operators are achieving a consistently low latency, rather than just a low average.

Packet loss is the fifth KPI to be measured by our consistent quality score, and reflects the percentage of data packets that fail to arrive at the destination. It has the greatest impact on video calling, where missing packets result in jumps in audio and video, but at a high percentage, can also cause noticeable slowdowns in web browsing or data downloads. More demanding use cases are less tolerant of a high packet loss percentage, which explains the low threshold for packet loss on our excellent quality measurement. 

 

Where Consistent Quality fits in

Consistent quality is not a perfect tool for measuring every facet of an operator’s network, but it does give a much better idea of how often users are happy with their connection than average download speed.

In its first outing in the US LTE network report, consistent quality provides a very different picture of the performance differences between Verizon, AT&T, T-Mobile, and Sprint than that painted by average download speed. While Verizon ran away with the average LTE download speed crown, the difference between Verizon and T-Mobile in excellent consistent quality was less than 2%. AT&T, which came in second on average download speed, was in third place on consistent quality, and much closer to Sprint’s last-place finish than T-Mobile’s second.

National Threshold Percentage Bar Chart

 

As download speeds keep increasing and operators eye expensive investment in the 5G infrastructure that will define networks for the next decade, the gap between what makes a network better and what makes a network faster will keep growing. New technology and new use cases demand new metrics, and our consistent quality score provides just that.    


Download our brand new LTE networks report to see a true picture of real-world mobile experience in the US and how each operator performs, no email address required.

Download report

 

 

 

 

 

US Report Cover 2