Introducing Consistent Quality - measuring more than just speed

For as long as consumers have been downloading apps, conventional wisdom has suggested that user satisfaction and download speed go hand in hand. The assumption made sense: home internet speeds are packaged and advertised by the megabit; the data speed increase has been the most notable user change in going from 2G to 4G; and, not insignificantly, mobile network providers have been running adverts based on speed for years.

But as LTE networks have evolved and the average download speed has crept ever higher, subscribers haven’t necessarily been getting happier. In our latest US networks LTE report, Verizon had the fastest average download speed, with 22 Mbps; the slowest network, Sprint, saw an average download throughput of just over 8 Mbps.

Capture2

 

Why speed doesn't tell the whole story

The chasm between those two numbers may seem significant, but most users simply aren’t going to notice a difference. A 2015 study from the Boston Consulting Group tested key network performance indicators against a survey of users’ happiness with their connection, and found that even when streaming video, any increase in speed over just 1.5 Mbps saw little increase in user satisfaction.

Looking at the minimum requirements from streaming video providers, you can see why. Popular services like YouTube, Netflix, and Hulu -- which account for a significant proportion of mobile data traffic -- have a minimum requirement of 1 Mbps to stream SD video, while even 720p or 1080p streams only require a maximum 3 or 4 Mbps.

 

Introducing Consistent Quality

So if download speeds aren’t a satisfactory proxy for a ‘good’ network experience, what is? Tutela’s new metric, named Consistent Quality and debuted in the US network report, provides a measure of how often an operator provides an adequate network experience to the subscriber.

We measure five key performance indicators -- the average download speed, upload speed, latency, jitter, and packet loss -- and calculate what percentage of tests meet or exceed a threshold on all of those metrics.

KPI Average download speed Average upload speed Latency Jitter Packet loss
Minimum acceptable value 4 Mbps 2 Mbps 50ms 30ms 5%

The specific threshold used to calculate our Consistent Quality score are a download speed above 4 Mbps, upload speed above 2 Mbps, latency less than 50ms, jitter less than 30ms, and packet loss less than 5 percent.

In choosing these KPIs and thresholds, we have built a measure of what is required to run any of the most popular apps that users in the US are using. Top 10 apps include YouTube -- which obviously means streaming video -- as well as the Google Play Store, Facebook Messenger, Snapchat, Instagram, and Amazon.

Using those apps means users need to be able to stream video, download apps, send messages, upload and download pictures and short videos, and browse the web, all of which will be possible with a connection that meets our consistent quality threshold.

 

Understanding our Key Performance Indicators

Including download speed and upload speed is the most obvious decision: even if it’s not the only thing that matters, a 128kbps connection still won’t work for any of our most important apps. A download speed of 4 Mbps is sufficient to stream HD video from any of the most popular streaming services, or download the 72MB Facebook app in under three minutes.

Users download significantly more data than they upload, making a faster download speed the more important of the two, but our upload speed threshold of 2 Mbps means users can conduct a video chat or upload images to Snapchat or Instagram without a noticeable delay.

Once the throughput thresholds have been reached, latency is arguably the biggest difference-maker in user experience. In that same BCG study, a decrease in latency from 100ms to 50ms saw a 10 percent jump in user satisfaction. Our threshold of 50ms is fast enough for onlines games like Fortnite, as well as voice and video calling. At faster download speeds, lower latency is also the most significant factor in web page load time.

Jitter is a measure of the variation in latency, and has a major impact on the quality of real-time services like voice and video calling. A low average latency is fine, but if the packet latency jumps between 25ms and 75ms regularly, users will see video artefacts and jumpiness, which degrades the experience. The jitter threshold of less than 30ms ensures that operators are achieving a consistently low latency, rather than just a low average.

Packet loss is the fifth KPI to be measured by our Consistent Quality score, and reflects the percentage of data packets that fail to arrive at the destination. It has the greatest impact on voice and video calling, where missing packets result in jumps in audio and video, but at a high percentage, can also cause noticeable slowdowns in web browsing or data downloads.

 

Where Consistent Quality fits in

Consistent Quality is not a perfect tool for measuring every facet of an operator’s network, but it does give a much better idea of how often users are happy with their connection than average download speed.

In its first outing in the US LTE network report, Consistent Quality provides a very different picture of the performance differences between Verizon, AT&T, T-Mobile, and Sprint than that painted by average download speed. While Verizon ran away with the average LTE download speed crown, the difference between Verizon and T-Mobile in Consistent Quality was less than 2%. AT&T, which came in second on average download speed, was in third place on Consistent Quality, and much closer to Sprint’s last-place finish than T-Mobile’s second.

National Threshold Percentage Bar Chart

 

As download speeds keep increasing and operators eye expensive investment in the 5G infrastructure that will define networks for the next decade, the gap between what makes a network better and what makes a network faster will keep growing. New technology and new use cases demand new metrics, and our Consistent Quality score provides just that.    


Download our brand new LTE networks report to see a true picture of real-world mobile experience in the US and how each operator performs, no email address required.

Download report

 

 

 

 

 

US Report Cover 2