Comments on Wireless Performance & Prediction Metrics

Comments on WirelessPerformance & PredictionMetrics Mark Kobayashi, Jason Trachewsky, Dalton Victor Broadcom Corp. Kobayashi Trachewsky Victor, Broadcom

Agenda • Objective • Background • Test Methods • Potential Metrics • Primary metrics • Secondary metrics • Problems with over simplification • Minimal draft requirements • Recommendations Kobayashi Trachewsky Victor, Broadcom

Topics • Objective • Background • Test Methods • Potential Metrics • Primary metrics • Secondary metrics • Problems with over-simplification • Minimal draft requirements • Recommendations Kobayashi Trachewsky Victor, Broadcom

Objective • From PAR and 5 criteria: • Scope: The scope of the 802.11T is to provide a set of performance metrics, measurement methodologies, and test conditions to enable measuring and predicting the performance of 802.11 WLAN devices and networks at the component and application level. (emphasis added) • Purpose: The purpose of the 802.11T is to enable testing, comparison, and deployment planning of 802.11 WLAN devices based on a common and accepted set of performance metrics, measurement methodologies and test conditions. Kobayashi Trachewsky Victor, Broadcom

Background • We should keep in mind various applications when creating test scenarios but we also need to consider the design of repeatable procedures. • To minimize the test plan size, we should abstract the traffic distributions while remaining faithful to the demands of the applications. • File transfer • VoIP • Others … Kobayashi Trachewsky Victor, Broadcom

Background • What makes a system great or something far less • Coverage (range) • Data rate • Voice quality (MOS, etc.) • Video quality • Users care about these most Kobayashi Trachewsky Victor, Broadcom

Background Data Apps • Application driven metrics are an important subset of the total space of metrics • However it is important to identify metrics outside of applications as well Streaming Apps Latency Sensitive Apps Metrics Application Driven Metrics Kobayashi Trachewsky Victor, Broadcom

Test methods • Uncontrolled over-the-air testing • Requires large number of relevant locations and requires statistical analysis. • The key is “large sample sizes” (many measurements at many locations). • Final outcome tells only whether system A “is better” than system B in statistically related environments Kobayashi Trachewsky Victor, Broadcom

Test Methods • An example showing the channel time evolution in one location • RX power and channel capacity can change rapidly in fading environment • Greater than 10dB change in received signal power in some bins over 60 msec. • People walking around in an adjacent room can cause changes. Kobayashi Trachewsky Victor, Broadcom

Test methods • Cabled testing • Can design tests that can be run several times yielding nearly identical results every time • Relative and absolute measurements of system performance can be made. • Can even construct frequency- and/or time-selective channels. • However, the disadvantage is no coverage of antennas, antenna-transceiver or antenna-chassis interactions. Kobayashi Trachewsky Victor, Broadcom

Test methods • The elusive middle ground? • It may be possible to reduce location-to-location variation and increase channel coherence time. • However, it is difficult to identify a set of conditions that sufficiently span the desired operational space. • Some possible techniques include the use of: • anechoic chambers • open-air test ranges • anechoic chambers with carefully placed and measured reflectors to introduce a specific multipath power-delay profile • If done well, relative and absolute full-system performance can be measured quickly. Kobayashi Trachewsky Victor, Broadcom

Primary metrics • Range (coverage) • Throughput (TCP) / File transfer time • Voice quality (MOS, etc.) • probably can be collapsed to: • frame error rate at a given PHY rate • latency at a given PHY rate • Video quality • ditto Kobayashi Trachewsky Victor, Broadcom

Secondary metrics • How are coverage/range and rate related to quantities precisely measurable at the component level? • Some secondary metrics: • Antenna gain patterns • Antenna port S11 • Receiver sensitivity • Output power • Tx EVM • Tx / Rx diversity system performance • Roam time under controlled conditions • Etc … Kobayashi Trachewsky Victor, Broadcom

Problems with over-simplification • RX power being equal in two locations does not mean that the performance of a system should be the same. (see comments on RF Performance Testing ) Kobayashi Trachewsky Victor, Broadcom

Over-simplification • Some of the secondary measurable quantities interact or cancel each other out, particularly in “small” tests. • E.g., good rx sensitivity may cancel out poor antenna gain in certain directions. • How does a component consumer define "goodness"? • Is it better to have an isotropic gain pattern or one with peaks in certain directions and nulls in others? (depends) Kobayashi Trachewsky Victor, Broadcom

Topics • Objective • Keep in mind • Test Methods • Potential Metrics • Primary metrics • Secondary metrics • Problems with over-simplification • Minimal draft requirements • Recommendations Kobayashi Trachewsky Victor, Broadcom

Minimal draft requirements • From PAR and 5 criteria: • Scope: The scope of the 802.11T is to provide a set of performance metrics, measurement methodologies, and test conditions to enablemeasuring and predicting the performance of 802.11 WLAN devices and networks at the component and application level. (emphasis added) • We should derive plans for testing wireless systems and predicting performance with all components integrated (system level). • We should derive plans for testing wireless components. • What do we define as a component? • Do we need a transceiver card test plan? • Do we need an antenna subsystem test plan? Kobayashi Trachewsky Victor, Broadcom

Minimal draft requirements • IEEE 802.11-2003 and 802.11g-2003 already specify limits for quantities such as: • receiver sensitivity • tx EVM • Antenna specifications are lacking other than specification of a "50-Ohm" port impedance. • There is no output power requirement (other than -3 dB for .11h) • Isn't this as much a function of other system requirements (battery life, cost)? • But, a higher-power transceiver will “outperform” a lower-power transceiver on a vanilla range/coverage test. Kobayashi Trachewsky Victor, Broadcom

Topics • Objective • Keep in mind • Test Methods • Potential Metrics • Primary metrics • Secondary metrics • Problems with over-simplification • Minimal draft requirements • Recommendations Kobayashi Trachewsky Victor, Broadcom

Recommendations • Create common framework for discussing particular metrics, and measurement methodologies. • See test template work in process. Kobayashi Trachewsky Victor, Broadcom

Recommendations • Begin a list of simplified metrics • Primary metrics • Secondary metrics • TGT really isn't in a position to collect enough data to map the secondary metrics to user experience through a large-scale field trial. • This is not a requirement. • However, it would be helpful to see contributions from member companies on any relevant trials. Kobayashi Trachewsky Victor, Broadcom

Recommendations • Aggregate set of metrics derived from applications combined with non-application driven metrics • Create Test Methdology that look at specific metrics with sub-tests that reflect the variations seen from aggregating the metrics • A single metric can be tested and then variations can be done where baseline test methodology would change very little Kobayashi Trachewsky Victor, Broadcom

Recommendations • The test environment should be as controlled as possible. • A valuable exercise for TGT is to attempt to create an over-the-air test plan that yields highly-repeatable results. • In parallel, a component test plan can be generated. • The test environment should not mandate (explicitly or implicitly) one vendor’s test equipment. Kobayashi Trachewsky Victor, Broadcom

Comments on Wireless Performance & Prediction Metrics