SAMAN: Simulation Augmented by Measurement and Analysis for Networks

1. SAMAN: Simulation Augmented by Measurement and Analysis for Networks John Heidemann 28 September 2000 PIs: Heidemann, Deborah Estrin, Ramesh Govindan, Ashish GoelStudents: Kun-chan Lan, Xuan Chen, Debojyoti Dutta USC/ISI and UCLA NMS Albuquerque PI Meeting / Sep. 2000

2. SAMAN Challenge • Network robustness is a key challenge facing the Internet: • Understanding, predicting and avoiding failures • Understanding, predicting and avoiding cascading failures • Planning failure recovery strategies • SAMAN will apply network simulation to address these problems NMS Albuquerque PI Meeting / Sep. 2000

3. “the Internet” Example Scenario 1 • What if the blue link becomes overloaded? • Today: discover the symptom (high loss found through manual monitoring) • SAMAN will help identify the cause: • Change in C2 traffic mix? • Interactions between C1 and C2 traffic? • Need good traffic models C1 C2 Network Provider Clients NMS Albuquerque PI Meeting / Sep. 2000

4. “the Internet” Example Scenario 2 • What if the green router goes down? (DDoS?) • May produce cascading failure (blue link) • SAMAN will support prediction, understanding, and avoidance of cascading failures • Need to explore correct part of large space of simulations C1 C2 Network Provider Clients NMS Albuquerque PI Meeting / Sep. 2000

5. Specific Failure Conditions • Fail-stop failures due to external events • accidental (backhoes) or intentional • Traffic overload • Loss rates higher than p • Good ISPs consider p>1% serious • Loss rates map non-linearly into performance degradation and load • Benign (simple overload), unexpected (traffic shift), or malicious (DDoS) • Current challenge: failure propagation (cascades, delayed convergence, etc.) [Shaikh00a,Labovitz00a] NMS Albuquerque PI Meeting / Sep. 2000

6. goodput load Why Simulation? Answer “what if?” For protocols, scales, scenarios outside experimentation. (But depends on good models in interesting part of space.) NMS Albuquerque PI Meeting / Sep. 2000

7. Agenda • Challenges • SAMAN in NMS • Applications • Technologies • Early results • Potential collaborations NMS Albuquerque PI Meeting / Sep. 2000

8. SAMAN Applications • Failure prediction: • Understanding and reproducing protocol behavior under extreme conditions • Network early warning system • Tools to automatically generate models NMS Albuquerque PI Meeting / Sep. 2000

9. Protocol Robustness • Reliable networks demand reliable protocols • How do individual protocols behave near the edge of their operating limits: • What conditions are important to study? • Are simple protocol improvements possible? • How do protocols interact in extreme conditions: • How do individual and aggregate behavior relate? • When does individual failure trigger cascading failure? NMS Albuquerque PI Meeting / Sep. 2000

10. Network Early-Warning Systems • Tools to predict imminent network failures • Trigger preventive or corrective actions • Clear mappings from tools to specific failures • Many current tools do local measurements • Are measurements topologically or temporally related? • Minimize control loop • Performance, understandability, deployability… NMS Albuquerque PI Meeting / Sep. 2000

11. Model Generation Tools • Tools to automatically configure simulation models from network measurements • Integrate data from multiple network points • Serve as input to other portions of work • Validated across multiple time-scales • Build on library of validated simulation models NMS Albuquerque PI Meeting / Sep. 2000

12. Agenda • Challenges • SAMAN in NMS • Applications • Technologies • Early results • Potential collaborations NMS Albuquerque PI Meeting / Sep. 2000

13. SAMAN Technologies • Just-in-time model generation • Accurate traffic models • Analysis-informed simulation • Constrain parameter search space • In a robust simulation environment • Build on widely-used ns platform NMS Albuquerque PI Meeting / Sep. 2000

14. Model Generation • Application-driven (structural) models • Capture application-level dynamics (feedback, user behavior) • Validated, applicable across range of time-scales • Network measurements to parameterize models • Integrate data from multiple measurement points • Resulting in just-in-time models • Network admins can measure and parameterize models NMS Albuquerque PI Meeting / Sep. 2000

15. Analysis-Informed Simulation • Failure analysis spans huge parameter space • Most of space is uninteresting • Analysis-informed simulation • Rapid analytic pre-simulation pass categorizes scenario as uninteresting (clearly out of scope) or interesting • Focus detailed simulation on interesting scenarios NMS Albuquerque PI Meeting / Sep. 2000

16. Ns Simulation Environment • Builds on rich ns simulation environment • Wired and wireless (radio and satellite) • Robust protocol library: many TCP variants, multicast, … • Validation experience and test suite • 648 scenarios in 58 categories • Multiple levels of abstraction • packet-level and abstractions eliminating per-hop routing, multicast tree formation, mixed abstract/detailed sims, etc. • Emulation: mix real-world and virtual nodes • Broad community support and use • ns-users mailing list: >1000 hosts (~institutions), >8000 e-mail addresses (~users) NMS Albuquerque PI Meeting / Sep. 2000

17. Evaluating scalability in single dimension very risky many dimensions: nodes, users, multicast senders vs. recievers, protocol agents, traffic volume understanding is often bottleneck Parallelism sometimes key…if one simulation has the answer don’t ignore free parallelism if multiple simulations needed (ex. vary parameters, replicate results) Abstraction is critical to large and fast network sim: ns went from 100s to 1000s by tuning [on desktop hardware], but 1000s to 10000s with abstractions many abstractions: centralizing computations (unicast and multicast routing, etc.) packet delivery abstraction (trains, end2end delivery, fluid flow) protocols abstractions (FSA TCP, etc.) mixed abstract/detailed sims Large Simulations NMS Albuquerque PI Meeting / Sep. 2000

18. Agenda • Challenges • SAMAN in NMS • Applications • Technologies • Early results • Potential collaborations NMS Albuquerque PI Meeting / Sep. 2000

19. Early Results • Current focuses: • Reproducing failure scenarios in simulation • Multi-scale, application-driven traffic models • Pre-simulation scenario filtering NMS Albuquerque PI Meeting / Sep. 2000

20. Early Results • Current focuses: • Reproducing failure scenarios in simulation • Multi-scale, application-driven traffic models • Pre-simulation scenario filtering NMS Albuquerque PI Meeting / Sep. 2000

21. Modeling Real-Audio Traffic • Why real audio? • Example of a streaming media protocol • Very different from TCP • Possibly representative of future streaming media (certainly more representative than TCP) • Why now? • Help develop tools for multi-scale models • Modeling protocol effects without source code NMS Albuquerque PI Meeting / Sep. 2000

22. Constant bit-rate … or not? Basic R-Audio Behavior time-sequence plot of single flow mean and quartiles of 1200 flows (mean is smooth, quartiles at multiples of 1.8s) NMS Albuquerque PI Meeting / Sep. 2000

23. More complex internal structure Demonstrates importance of studying protocols at multiple time-scales Able to capture internal structure after iteration R-Audio Under the Microscope bursts 1.8s inter-burst interval NMS Albuquerque PI Meeting / Sep. 2000

24. R-Audio: Time-Variance Plot trace model noticeably less variance at key scales (1.8, 3.6, etc.) NMS Albuquerque PI Meeting / Sep. 2000

25. R-Audio: Scaling Plot trace model NMS Albuquerque PI Meeting / Sep. 2000

26. R-Audio Experiences and Plans • Currently validating model • stats seem promising • validation against additional traces in progress • Next steps: • Rapid model parameterization • Apply tools to complex models (mixed traffic) • Apply models to NMS challenge problem NMS Albuquerque PI Meeting / Sep. 2000

27. Early Results • Current focuses: • Reproducing failure scenarios in simulation • Multi-scale, application-driven traffic models • Pre-simulation scenario filtering NMS Albuquerque PI Meeting / Sep. 2000

28. Agenda • Challenges • SAMAN in NMS • Applications • Technologies • Early results • Potential collaborations NMS Albuquerque PI Meeting / Sep. 2000

29. Potential Collaborations • NMS can use models (ex. real audio) • In public ns releases now • Could be ported to other simulators • Model parameterization could use NMS measurement tools • Collaborative addition of NMS work into ns • Traffic, topology models • Simulation optimizations and abstractions • Non-NMS projects (STRESS, etc.) • Other opportunities? NMS Albuquerque PI Meeting / Sep. 2000

30. More information • http://www.isi.edu/saman/ NMS Albuquerque PI Meeting / Sep. 2000