Urban Mesh Networks and Protocol Behavior under Diverse Operating Conditions

1. Urban Mesh Networks and Protocol Behaviorunder Diverse Operating Conditions Ed Knightly Rice University http://www.ece.rice.edu/~knightly Joint work with Joseph Camp, Omer Gurewitz, Vincenzo Mancuso, Jingpu Shi

2. Experimental Context: Large-Scale Multi-hop Networks • Technology For All Wireless Network • Research platform: programmable and observable • Wireless ISP for region since late 2004 • Over 4,000 users in 3 square kilometers • Multi-tier architecture

3. Access Tier • Clients (mobiles and residents) access mesh infrastructure • AP density: approximately 7 Mesh AP’s per km2

4. Backhaul Tier • Access nodes interconnected via backhaul tier • Access traffic forwarded to and from gateway • Omni directional 802.11 (b, g, or a)

5. Capacity Points • Gateways inject capacity into backhaul tier • Injects capacity for sufficient Mb/sec/km2 • Continued multi-hopping would be too many users over too many hops

6. TFA Network Architecture and Topology • 802.11b/g access and backhaul serving 4,000 users • Point to point capacity injection tier (802.11a and 900 MHz) • Multiple radios at gateway nodes, single radios elsewhere • Opportunistic and population-driven GW locations • 99% coverage over 4 km2 when complete • Future: WARP backhaul

7. Challenge of Diverse Operating Conditions • Protocol behavior typically narrowly understood • Study’s particular mathematical assumptions, ns configuration, testbed setup Challenges: • Design protocols that are robust to diverse: • Channel conditions • Asymmetries • Topologies • Traffic matrices • Understand protocol performance in diverse operating conditions

8. Explore robustness and unexplored cross-layer interactions driven by: Channel conditions Asymmetries Topologies Traffic matrices Two protocols TCP-like congestion control over multi-hop CSMA/CA Control traffic Two Case Studies

9. Node Down! 1 Mbps 5 Mbps Heterogeneous Channel Conditions & Traffic Types Heterogeneous Connectivity Set • High quality forwarding links (selected by routing protocol) • Diverse non-forwarding links (broadcast medium) Data and Control Planes • Large-sized data frames • Small-sized control frames • Link Establishment • Routing • Congestion Control • Network Management Homogeneous Topology Symmetric Topology

10. control data Experimental Finding • Heterogeneous connectivity matrix + heterogeneous traffic produces diverse cross-layer effects • Control frames force multiplicative degradation on data plane • Overhead traffic at rate r can reduce data throughput by up to 30 times r • Wireless Overhead Multiplier driven primarily by non-forwarding links

11. 1800 kbps 1100 kbps 800 kbps Diverse Overhead Effects • Experiment: measure data throughput with and without control overhead • Identical configuration • TX power 200 mW, RTS disabled, Autorate enabled • Overhead of 80 kbps (approx. 10 kbps/node) • Vastly different performance with and without overhead • 800 to 1800 kbps degradation • 10-20 times injected overhead

12. Wireless Overhead Multiplier Definition • Define WOM to quantify the effect of the bits of overhead • O is a set of OH-injecting nodes, where oO • Ois bits/sec of injected overhead from O • t s→r{s,r}is saturation throughput of tx (s) and rx (r)

13. data data How to Predict WOM? • If we know link characteristics, can we predict WOM? • Classify link between emitter of control traffic and data transmitter according to protocol-defined behavior • Decode Transmission • Detect Channel Activity (“carrier sense range”) • Unable to Detect Channel Activity (hidden) control control O O S R S R

14. “Carrier Sense Range” Does Not Exist • In-lab measurements show no carrier sense threshold • Set-up: 3 different cards (2 Mbps fixed modulation rate, UDP traffic) • Constant Noise • External 802.11 source heard only at transmitter (not shown) • Throughput degradation due to transmitter becoming deaf to ACK • Producing excessive backoff • Continues to transmit • MAC traces taken with Kismet Card at TX becomes deaf to ACK packets

15. WOM for Existing Link Classes • Measurements for GW’s neighbors • Injected overhead: 10 kbps, Autorate enabled, RTS off • Transmission Range (link o to s) • Overhead effectively sent at base rate (2 Mbps) • On average, quality of TFA links enables 11 Mbps operation • Out of Range (link o to s) • Average WOM: 10 (high variance) • What is causing the high variance in WOM? TCP data traffic (1500 byte), Autorate enabled, RTS off

16. Relative Link Quality of Competing Links • Same link behavior as defined by 802.11 (unable to carrier sense) but high variance - why? • Same injected overhead and non-forwarding links • Expect high WOM values (low variance) • Asymmetric WOM with forwarding link differences • 2 dB difference in link quality significantly alters performance physical layer capture DATA_s DATA_s OH OH link 1 < link 2 link 1 > link 2 UDP data traffic (1500 byte), Autorate disabled, RTS off

17. WOM and Diverse Link Characteristics • Control traffic can have widely assumed modest effect • Link asymmetry yields severe degradation • Overhead rate has up to 30 times impact on data rate • Strong nodes transmitting control traffic have wide reaching effects • In a realistic topology, a mix is inevitable

18. Explore robustness and unexplored cross-layer interactions driven by: Channel conditions Asymmetries Topologies Traffic matrices Two protocols TCP-like congestion control over multi-hop CSMA/CA Control traffic Two Case Studies

19. Congestion Control • Single TCP flow over multi-hop chain and 802.11 • Yields high utilization with an appropriate congestion window • Ideal window is a function of chain length TCP DATA A B GW TCP ACK

20. However… • Radically different behavior for a changed traffic matrix: • starvation arises even with fixed sliding-window flow control coupled with CSMA (including 1!) • individual or aggregate one-hop flows can starve multi-hop flows • This is the basic scenario for a mesh network • multi-hop and multi-flow TCP DATA A B GW TCP ACK

21. Severe Throughput Imbalance • Experiment of potential for starvation in operational mesh networks • inject traffic from A and B to GW • saturation conditions TCP DATA A B GW TCP ACK The two-hop node “starves” when contending with the one-hop node

22. Origins of Starvation • Compounding effect of three factors: • Collision avoidance in the medium access protocol induces bi-stability in which pairs of nodes symmetrically alternate in capturing system resources • Congestion control in the transport protocol induces asymmetry in the time spent in each state and favors the one-hop flow • High penalty due to cross-layer effects in terms of loss, delay, and consequently, throughput, in order to re-capture system resources

23. Due to lack of coordination: Bi-stable state: either A transmits and GW is in high backoff, or GW transmits and A is in high backoff Success state and fail state alternate Symmetric behavior CW=2CWmin CW=22CWmin CW=2CWmin CW=22CWmin CW=CWmin CW=2kCWmin CW=CWmin CW=CWmin CW=CWmin CW=CWmin Origins (I):Medium Access and Bi-Stability A B GW DATA Aggregate ACK

24. Multiple packet burst (GW,B) Multiple packet burst (A,B) GW traffic A traffic B traffic A B GW Middle Node Shares with Winner • B is in range of both A and GW (complete channel state) • B's packets interleave with A's and GW's packets

25. Origins (II):Asymmetry Induced by TCP DATA DATA Outer loop GW B A Inner loop ACK ACK DATA DATA GW B A DATA GW B A ACK • Two nested transport loops and sliding windows • Asymmetric impact of multipacket capture: transport loops change the duration of states • (A, B) burst: • the burst size is limited by: • TCP window size • (GW, B) burst: • self-sustaining loop: • TCP ACK are generated

26. TCP ACK received (Cumulated ACK) 335 TCP Congestion Window 330 MAC Packet drop (Max Retry Limit reached) 325 320 315 TCP sequence number [kB] TCP Timeouts 310 305 First time segment is transmitted TCP retransmissions 300 295 80 85 90 95 100 105 Time [sec] A B GW Origins (III): Severe State Transition Penalties • Asymmetric impact of multipacket capture • Node GW incurs small penalty: • short duration of fail state but long packet bursts • Node A incurs high penalty: • long duration of fail state and low offered load • high backoff & multiple TCP timeouts Node A

27. Analytical Model Objectives Isolate and capture the root cause of starvation Model one aspect of congestion control (sliding window), queues, and CSMA/CA Technique Embedded Markov chain model Queue state, congestion window, contention window, carrier sense

28. Evaluation: Model, Simulations, and TFA • Model • static sliding window congestion control • Simulation • fixed TCP congestion window + timeouts, cumulative ACKS, … • legacy TCP New Reno (dynamic congestion window) • Measurements at TFA • TCP New Reno+802.11 Model predicts starvation: experimental factors exacerbate

29. What Next? • Starvation exists • Understand origins through analysis and modeling of protocols • CSMA + sliding window are sufficient • Can we fix it?

30. QA QGW A B GW A B GW Solution: Redistributing Queues via MAC • Recall origins • MAC bi-stability • Flow-control induced asymmetry • Severe state transition penalty • Mitigate MAC bi-stability • Re-distribute the “distributed queue” • Decrease the steady state probability of system states where QA > 0 and QGW > 0 • Model-driven topology-based solution • Middle node should access medium less aggressively to shift queues to itself • Increase CWmin for node B

31. Conclusions • Urban-scale experiments preclude a narrow set of assumptions • Reality is diversity in channels, topologies, traffic matrices, … yielding vastly different protocol behaviors • Challenges • understanding protocols in diverse operating conditions • robust protocol design http://networks.rice.edu