Benchmarking is the process of comparing one operator’s network delivered quality against competitors, comparing network performance between markets or evaluating “before and after” performance following a critical change, such as a network enhancement program.
Benchmarking a wireless network can mean many different things to different people. We believe that benchmarking is not an optional process for operators – it is essential to the effective management of any mobile network.
Successful benchmarking should be repeatable and technically consistent; provide comparability of KPI’s for all networks; show trending over time and location; be statistically valid and technology independent; and be robust enough to withstand legal scrutiny of claims made. In addition to network operators, other entities such as regulators and equipment vendors are often interested in benchmarking. Some of the many reasons that influence all of these players to pursue a benchmarking program include:
- Seeking reassurance of market competitiveness
- Confirmation of strategic goals
- Evaluate results of infrastructure changes on performance
- Support ad-claims for marketing campaigns
- Compare network equipment vendor performance
- Confirm performance of outsourcing SLAs
- Identify opportunities for quality improvement
- Confirm operator performance towards license requirements
- Assess state of competition in the marketplace
- Evaluate emergency communications service readiness
- Ensure spectrum plan compliance
GWS has experience designing and implementing voice and data benchmarking campaigns to support all of these reasons why a company chooses to benchmark.
Benchmarking services require specialized collection platforms for vehicle or pedestrian testing. GWS has the experience and resources to conduct full voice and data benchmarking campaigns using collection equipment from any of the major commercially available test equipment vendors. Regardless of vendor, the benchmarking platform will be a single installation chassis housing multiple devices of multiple technologies and sharing a single navigation device. The platform will support external antennas to eliminate interference between devices, provide attenuators to simulate user performance and eliminate the device battery to allow automatic reset of the equipment.
The platform will have advanced call scripting that provides synchronized dialing across all devices for consistent call statistics and offer comprehensive call classification. Standard fixed side modules will provide quality data for uplink and downlink and collect all voice clips for post drive analysis. The platform should also allow the engineer to view displays during data collection for each tested handset, including navigation plots, tables, charts, and graphs for real-time monitoring of all handsets to provide the engineer with immediate, on-site data quality assurance.
GWS Data Collection and Quality Assurance
Our data collection engineers are highly skilled and receive ongoing training on our proven, comprehensive collection and quality procedures as well as hands on training for the equipment we field. Each project is monitored by a dedicated Project Manager who interfaces with the customer and the internal organization to make sure campaigns meet our quality expectations and customer business goals.
Back office administration support keeps teams on the road and functioning efficiently. All data is delivered to our quality assurance team which conducts validation of collected data integrity on a daily basis in addition to comprehensive review of the performance results on a market wide basis. Any questions or possible problems are escalated to quality assurance management and the project manager so they can be addressed while the collection team is still in the field. Only after benchmarking results have been fully reviewed and accepted are they forwarded for full post processing and reporting.
GWS Post Processing and Reporting
Post-processing is performed using the collection vendor’s processing software, Microsoft SQL Server 2005, and GWS’ proprietary in-house REMIS software applications. The data log files collected by the benchmarking platform are initially processed with the vendor’s software, then exported into an SQL database. GWS’ REMIS software then compresses the data into a flat file and exports it into a compact ASCII format. The exported data is analyzed by GWS’ data collection applications, where it is possible to quickly compare network performance across technologies.
GWS has developed comprehensive yet concise reports for any corporate audience – RF engineering, operations, marketing or senior management. GWS’ in-house software tools enable the mining of drive-test data, visualizing of RF problems, and presentation of results to the client. Our software tools and highly experienced engineering team allow us to offer both unparalleled, business critical voice and packet data benchmarking reports and Extended Analysis options which provide additional value added services using the collected data for any part of a customer’s organization.