GWS Approach to Benchmarking

a primer on the methodology we use to compare, contrast, and evaluate mobile networks

What makes for a well-founded methodology?

GWS has been conducting mobile network benchmarking for nearly thirty years. We have tested over 18,000 markets in 25 countries. This includes driving or walking over 16 million miles while conducting our benchmarking programs. We perform nationwide and regional benchmarking – collecting data and evaluating network performance while mobile, in buildings, outdoors, and at major events. We have invested considerable resources and expertise across three decades to craft and enhance our methodology. From the selection and configuration of specific test equipment to the type and placement of collection devices, we know that the collection process is critical to ascertaining a network’s true performance. We have spent years calibrating our processing algorithms and reporting methodologies to properly classify voice call and packet task results so that we can provide a clear understanding of the test results and corresponding evaluation; which, in turn, provides constructive direction for “next steps”, whether that involves network optimization or some other effort. Further, GWS’s experience in processing terabytes of daily collected data also contributes to validating and ensuring reliable data. In all stages of the process, we believe it’s critical to employ quality control methods and accurate scientific analysis of the collected data because that’s what will generate the best results.

This isn’t meant to be marketing speak, it’s meant to demonstrate that we are experienced and have strict processes and controls in place to realistically compare and contrast the performance of mobile networks. We provide very specialized expert analysis that is rock solid and defendable. As our trademark states, GWS is “The Standard for Benchmarking.” With this in mind, the following provides a deeper dive into our benchmarking methodology approach.

It starts with quality and control

28+

Years Measuring Networks

_________

25

Countries Benchmarked

_________

18,000

Markets Tested

_________

16 MILLION

Miles Walked and Driven

It starts with quality and control

GWS believes that the more control there is during the collection, data processing, and evaluation phases, the better. While no collection method can collect data in all locations, GWS’s approach is to cover as much of the population as possible, with more performance data collected in areas of higher population density (i.e., cities and busy roads). Our objective is to find the balance between making the collection reflect the user experience and making sure that the results are reflective of the performance of the mobile operator’s network. For example, we want results that highlight a failure or poor performance that is due to the operator’s network and not exclusively due to a user’s device or a cheaper data plan. To this end, the quality of the data collected is key. We make sure that the categorized results of each task and metric are representative and accurately reflect the performance of a network. GWS employs a team of highly trained engineers to understand each task and make sure it is representative before it is used for reporting.

Mobile phone store

Mobile devices – we buy from the same place consumers do

GWS tests with recent and popular devices that are publicly available. This also includes testing with publicly launched firmware. The objective is to conduct testing that more closely represents the user experience. We also make sure that new features launched on a network show themselves in the everyday tasks that are performed by or available to most consumers. Put another way, we concentrate our testing efforts on measuring network performance as it relates to the daily activity of a typical consumer.

Mobile devices – we buy from the same place consumers do

GWS tests with recent and popular devices that are publicly available. This also includes testing with publicly launched firmware. The objective is to conduct testing that more closely represents the user experience. We also make sure that new features launched on a network show themselves in the everyday tasks that are performed by or available to most consumers. Put another way, we concentrate our testing efforts on measuring network performance as it relates to the daily activity of a typical consumer.

Mobile phone store

Definable metrics and transparent evaluations

GWS focuses on standard industry metrics and straightforward analysis to enable comparisons between mobile network operators. Further, the metrics we use are calculable through the tests we run using the consumer devices. In addition, our overall methodology (including our patented OneScore ranking methodology which we introduced in 2016 and formalized over the next three years) closely aligns with the methodologies provided in the ETSI Technical Report regarding Benchmarking Best Practices (TR 103 559 – first published in 2019).

benchmarking

Definable metrics and transparent evaluations

GWS focuses on standard industry metrics and straightforward analysis to enable comparisons between mobile network operators. Further, the metrics we use are calculable through the tests we run using the consumer devices. In addition, our overall methodology (including our patented OneScore ranking methodology which we introduced in 2016 and formalized over the next three years) closely aligns with the methodologies provided in the ETSI Technical Report regarding Benchmarking Best Practices (TR 103 559 – first published in 2019).

benchmarking

The consumer is king!

In terms of specific metrics, we start from the consumer side to determine what’s most important to them when using a mobile network. For example, we may evaluate in terms of coverage, reliability, speed, high-quality streaming, or some combination that factors in all the criteria that are most important to consumers.

Our methodology including our OneScore scoring approach provides a comprehensive framework for evaluating mobile network performance with input from real-world users. This unique approach integrates two key components: rigorous network testing and direct consumer feedback. By combining these elements, our methodology can deliver a holistic and accurate view of network performance that incorporates technical metrics that are relatable to consumer expectations.

GWS Sample Metrics Evaluated and Scored

GWS sample metrics evaluated and scored

Depending on the testing objective (for example, determining the best network experience, most reliable, fastest network, etc.), we will evaluate networks across critical categories such as data performance, network coverage, video performance, voice quality, and network reliability. These categories are carefully weighted based on consumer preferences and priorities, ensuring that the results reflect what matters most to users. This methodology allows GWS to provide actionable insights that not only highlight technical performance but also guide mobile network operators in delivering experiences that meet and exceed customer needs.

Depending on the testing objective (for example, determining the best network experience, most reliable, fastest network, etc.), we will evaluate networks across critical categories such as data performance, network coverage, video performance, voice quality, and network reliability. These categories are carefully weighted based on consumer preferences and priorities, ensuring that the results reflect what matters most to users. This methodology allows GWS to provide actionable insights that not only highlight technical performance but also guide mobile network operators in delivering experiences that meet and exceed customer needs.

How do we know what consumers want?

We conduct in-depth consumer research to understand how consumers are using their network and what they expect in terms of performance. Whether through nationwide polling (using a statistically significant number of respondents) or through actual focus groups, we dive deep into the mobile consumer’s psyche. For example, our consumer surveys ask respondents to rank the importance of tasks they perform on their mobile devices, as well as rank performance criteria that are most important to them. Our network test results are then weighted according to how consumers prioritize what’s important to them and evaluated as it relates to the network performance areas under study (whether it be related to voice, data, network reliability, network coverage, or some combination).

To sum it up

Our methodology involves the integration of controlled, scientific testing and in-depth nationwide consumer research. All of which results in the industry’s most comprehensive, realistic evaluation of network performance.

About Global Wireless Solutions, Inc. 

For most consumers today, their smart device is their life’s remote control. As an independent wireless insights consulting firm, GWS measures every aspect of how people live, work, and play via their mobile devices – as well as how mobile network performance affects them. 

Combining our market-leading network benchmarking technology and methodology with deep behavioral data, we help businesses and organizations drive innovation and deliver better customer experiences through wireless insights, anywhere in the world. Championing the needs of mobile and broadband users by understanding and testing the things that matter to them, we’ve evolved our technology and services in step with the needs of industry and consumers for 28 years.

Data Usage Policy

Using Our Data: All information, test results, performance metrics, or awards provided in any GWS press release or report may not used by an entity for marketing or advertising purposes (including, for example, promotion via the entity’s website or social media channels or paid-for promotion such as, but not limited to, print, online, radio, TV, and social media) without prior approval from GWS. Email marketing@w.bm.gwsolutions.com for further details.