CSRIC WG#3 Indoor Location Accuracy Test Bed Report

1. CSRIC WG#3 Indoor Location Accuracy Test Bed Report March6, 2013 Stephen J. Wisely & Richard Craig Co-Chairs Communications Security, Reliability and Interoperability Council CSR C

2. Work Group 3 Areas of Focus Outdoor Report Issued Work Effort Completed Indoor Report Issued Work Effort on Test Bed completed Feb. 2013 Work Effort Completed February 2013

3. In summary From September 12, 2012 Presentation • The WG addressed the FCC’s questions as best we could for now • We developed a testbed framework and have solicited interest from suppliers of current technology on their in interest in having their product go through the TB • We are currently working on the finer points of the testing process and logistics associated with TB program oversight • We anticipate conducting testing in two stages: • Stage I – Characterize performance of current technologies • Stage II – review of future technologies • We will provide a read out on Stage I efforts to the CSRIC committee at the September 2012 and March 2013 meetings • We believe the Testbed concept could survive beyond the current CSRIC charter as an independently supported entity for evaluating future technologies

4. Participants WG 3 Indoor Location Test Bed Subgroup consisted of the following members:

5. Indoor Test Bed –Stage 1 • Goal: Independent comparative testing that provides the FCC with meaningful, objective results on indoor wireless location performance • Constraints: Within time frame of CSRIC III; financially feasible framework • WG3 chose San Francisco Bay Area due to the wide range of wireless environments (morphologies) present • Invitation extended to a host of location technology vendors to participate • 7 vendors started the process and 3 eventually participated • Two existing and one new location technologies tested: • NextNav (wireless beacon technology) • Polaris Wireless (RF pattern matching/fingerprinting) • Qualcomm (AGPS/AFLT hybrid) • Four major wireless carriers participated: • AT&T, Sprint, T-Mobile, Verizon • Competent, experienced independent test house selected (TechnoCom) Communications Security, Reliability and Interoperability Council CSR C

6. Indoor Test bed Methodology • Based on sound statistical test methodology • Articulated in ATIS-0500013; received wide consensus • Multiple buildings (19) of different sizes & types across range of environments • Dense urban, urban, suburban, and rural morphologies • Multiple test points in each building • 2 to 6 test points depending on size and complexity of selected building • Representing typical use scenarios • Statistically significant, adequately large number of independent test calls at each stationary test point • 100 test calls per technology minimum target per test point • >13,400 test calls per technology placed at 74 test points, well over 100 avg. • Aggregated over different devices and networks used by the location technology under test at each test point • Indoor ground truth accuracy < 1 m (professional survey accuracy 2 cm) • Performance attributes tested: accuracy, yield, TTFF, uncertainty, scatter Communications Security, Reliability and Interoperability Council CSR C

7. Test Environments and Buildings Dense urban Downtown Financial District San Francisco (6 bldgs) Urban Downtown San Francisco and San Jose (5 bldgs) Suburban Santa Clara and Sunnyvale (6 bldgs) Rural San Benito County (2 bldgs) Communications Security, Reliability and Interoperability Council CSR C

8. Summary Results-- Accuracy Urban environment had the most challenged horizontal accuracy followed closely by dense urban NextNav technology provided vertical results Communications Security, Reliability and Interoperability Council CSR C

9. Summary Results: Yield, TTFF, and Uncertainty • Yield varied with the severity of the environment but was generally high • >90% for all technologies in the configurations tested with one minor exception (86% AGPS/AFLT yield in dense urban) • More yield variation for the technology actually integrated into the wireless network and has to exchange messages per the standard E911 implementation through severe RF fading • TTFF was stable and not an issue • Small variation about design values for non-network integrated implantations • 27 sec. for NextNav and 24 sec. for Polaris • More variation for AGPS/AFLT, which is actually integrated into the network • 28 sec. avg. with 33 sec. 90% in urban/dense urban vs. 24 sec avg. & 26 sec. 90% suburban/rural • Uncertainty is still a useful parameter and its reliability generally reflects how well a system is performing in a given environment Communications Security, Reliability and Interoperability Council CSR C

10. Location Scatter Relative to Building Sizes & Density • Traditional benchmarks from outdoor performance may not best suite the indoors, especially in denser cities • Even traditionally good performance may not meet the unmitigated expectations of some--education on indoor issues is important Test points in the 5-story building partially shadowed by tall bldg Test point 3 (T3) at center of 50/150 m circles Communications Security, Reliability and Interoperability Council CSR C

11. Lessons and Recommendations • The cooperatively funded indoor test bed is a viable framework, which yields very valuable results • It’s a model that should be repeated with some adjustments, based on the lessons learned • Key to its success are the experience & strong commitment of its participants • Create and maintain the CSRIC-type framework that fosters the active and balanced participation by the various stakeholders • There are a number of difficult challenges that need to be met repeatedly • Project setup and contractual arrangements are difficult and time consuming • Building access is the most vexing of challenges facing indoor testing • Allow for the long lead time of several months required to properly tackle these issues • Create a test management resource with contractual authority and some funding capability • Develop a more structured approach to dealing with building access and, as needed, use creative incentive mechanisms for buildings to participate • Deployed location technologies help with indoor performance and new technologies tested have promising performance, but there is a long way to go to meet expectations for reliable indoor use of wireless location for E911 • Continually test new technologies, including next generations of existing systems, technologies used for LBS (e.g., WiFi) & new technologies targeted to the latest air interfaces • The Bay Area has proven to be a good place for the indoor test bed • Leverage it fully in future test beds Communications Security, Reliability and Interoperability Council CSR C