Intro & Overview of RADS goals

1. Intro & Overview of RADS goals Armando Fox & Dave Patterson CS 444A/CS 294-6, Stanford/UC Berkeley Fall 2004

2. Administrivia • Course logistics & registration • Project expectations and other deliverables • Background and motivation for RADS • ROC and its relationship to RADS • Early case studies • Discussion: projects, research directions, etc.

3. Administrivia/goals • Stanford enrollment vs. Axess • SLT and CT tutorial VHS/DVD’s available to view • SLT and CT Lab/assignments grading policy • Stanford and Berkeley meeting/transportation logistics • Format of course

4. Background & motivation for RADS

5. RADS in One Slide • Philosophy of ROC: focus on lowering MTTR to improve overall availability • ROC achievements: two levels of lowering MTTR • “Microrecovery”: fine-grained generic recovery techniques recover only the failed part(s) of the system, at much lower cost than whole-system recovery • Undo: sophisticated tools to help human operators selectively back out destructive actions/changes to a system • General approach: use microrecovery as “first line of defense”; when it fails, provide support to human operators to avoid having to “reinstall the world” • RADS insight: can combine cheap recovery with statistical anomaly detection techniques

6. Hence, (at least) 2 parts to RADS • Investigating other microrecovery methods • Investigating analysis techniques • What to capture/represent in a model • Addressing fundamental open challenges • stability • systematic misdiagnosis • subversion by attackers • etc. • General insight: “different is bad” • “law of large numbers” arguments support this for large services

7. Why RADS • Motivation • 5 9’s availability => 5 down-minutes/year => must recover from (or mask) most failures without human intervention • a principled way to design “self-*” systems • Technology • High-traffic large-scale distributed/replicated services => large datasets • Analysis is CPU-intensive => a way to trade extra CPU cycles for dependability • Large logs/datasets for models => storage is cheap and getting cheaper • RADS addresses a clear need while exploiting demonstrated technology trends

8. Cheap Recovery

9. Complex systems of black boxes • “...our ability to analyze and predict the performance of the enormously complex software systems that lies at the core of our economy is painfully inadequate.” (Choudhury & Weikum, 2000 PITAC Report) • Networked services too complex and rapidly-changing to test exhaustively: “collections of black boxes” • Weekly or biweekly code drops not uncommon • Market activities lead to integration of whole systems • Need to get humans out of loop for at least some monitoring/recovery loops • hence interest in “autonomic” approaches • fast detection is often at odds with false alarms

10. Consequences • Complexity breeds increased bug counts and bug impact • Heisenbugs, race conditions, environment-dependent and hard-to-reproduce bugs still account for majority of SW bugs in live systems • up to 80% of bugs found in production are those for which a fix is not yet available* • some application-level failures result in user-visible bad behavior before they are detected by site monitors • Tellme Networks: up to 75% of downtime is “detection” (sometimes by user complaints), followed by localization • Amazon, Yahoo: gross metrics track second-order effect of bugs, but lags actual bug by minutes or tens of minutes • Result: downtime and increased management costs * A.P. Wood, Software reliability from the customer view, IEEE Computer, Aug. 2003

11. “Always adapting, always recovering” • Build statistical models of “acceptable” operating envelope by measurement & analysis on live system • Control theory, statistical correlation, anomaly detection... • Detect runtime deviations from model • typical tradeoff is between detection rate & false positive rate • Rely on external control using inexpensive and simple mechanisms that respect the black box, to keep system within its acceptable operating envelope • invariant: attempting recovery won’t make things worse • makes inevitable false positives tolerable • can then reduce false negatives by “tuning” algo’s to be more aggressive and/or deploying multiple detectors Systems that are “always adapting, always recovering”

12. Toward recovery management invariants • Observation: instrumentation and analysis • collect and analyze data from running systems • rely on “most systems work most of the time” to automatically derive baseline models • Analysis: detect and localize anomalous behavior • Action: close loop automatically with “micro-recovery” • “Salubrious”: returns some part of system to known state • Reclaim resources (memory, DB conns, sockets, DHCP lease...), throw away corrupt transient state, setup to retry operation if appropriate • Safe: no effect on correctness, minimal effect on performance • Localized: parts not being microrecovered aren’t affected • Fast recovery simplifies failure detection and recovery management.

13. Non-goals/complementary work All of the following are being capably studied by others, and directly compose with our own efforts... • Byzantine fault tolerance • In-place repair of persistent data structures • Hard-real-time response guarantees • Adding checkpointing to legacy non-componentized applications • Source code bug finding • Advancing the state of the art in SLT (analysis algorithms)

14. Outline • Micro-recoverable systems • Concept of microrecovery • A microrecoverable application server & session state store • Application-generic SLT-based failure detection • Path and component analysis and localization for appserver • Simple time series analyses for purpose-built state store • Combining SLT detection with microrecoverable systems • Discussion, related work, implications & conclusions

15. Microrebooting: one kind of microrecovery • 60+% of software failures in the field* are reboot-curable, even if root cause is unknown... why? • Rebooting discards bad temporary data (corrupted data structures that can be rebuilt) and (usually) reclaims used resources • reestablishes control flow in a predictable way (breaks deadlocks/livelocks, returns thread or process to its start state) • To avoid imperiling correctness, we must... • Separate data recovery from process recovery • Safeguard the data • Reclaim resources with high confidence • Goal: get same benefits of rebooting but at much finer grain (hence faster and less disruptive) - microrebooting * D. Oppenheimer et al., Why do Internet services fail and what can be done about it? , USITS 2003

16. Write example: “Write to Many, Wait for Few” AppServer STUB Try to write to W random bricks, W = 4Must wait for WQ bricks to reply, WQ = 2 Brick 1 Brick 2 Browser Brick 3 Brick 4 Brick 5

17. Write example: “Write to Many, Wait for Few” AppServer STUB Try to write to W random bricks, W = 4Must wait for WQ bricks to reply, WQ = 2 Brick 1 Brick 2 Browser Brick 3 Brick 4 Brick 5

18. Write example: “Write to Many, Wait for Few” AppServer STUB Try to write to W random bricks, W = 4Must wait for WQ bricks to reply, WQ = 2 Brick 1 Brick 2 Browser Brick 3 Brick 4 Brick 5

19. Write example: “Write to Many, Wait for Few” AppServer STUB Try to write to W random bricks, W = 4Must wait for WQ bricks to reply, WQ = 2 Brick 1 Brick 2 Browser Brick 3 Brick 4 Brick 5

20. Write example: “Write to Many, Wait for Few” AppServer STUB Crashed? Slow? Try to write to W random bricks, W = 4Must wait for WQ bricks to reply, WQ = 2 Brick 1 Brick 2 Browser 14 Brick 3 Brick 4 Cookie holds metadata Brick 5

21. Read example: AppServer STUB Try to read from Bricks 1, 4 Brick 1 14 Brick 2 Browser Brick 3 Brick 4 Brick 5

22. Read example: AppServer STUB 14 Brick 1 Brick 2 Browser Brick 3 Brick 4 Brick 5

23. Read example: AppServer STUB Brick 1 crashes Brick 1 Brick 2 Browser Brick 3 Brick 4 Brick 5

24. Read example: AppServer STUB Brick 2 Browser Brick 3 Brick 4 Brick 5

25. SSM: Failure and Recovery • Failure of single node • No data loss, WQ-1 remain • State is available for R/W during failure • Recovery • Restart – No special case recovery code • State is available for R/W during brick restart • Session state is self-recovering • User’s access pattern causes data to be rewritten

26. Backpressure and Admission Control AppServer AppServer STUB STUB Brick 1 Brick 2 Drop Requests Brick 3 Brick 4 Brick 5 Heavy flow to Brick 3

27. Statistical Monitoring Pinpoint Pinpoint Statistics Statistics NumElementsMemoryUsedInboxSizeNumDroppedNumReadsNumWrites Brick 1 Brick 2 Brick 3 Brick 4 Brick 5

28. SSM Monitoring • N replicated bricks handle read/write requests • Cannot do structural anomaly detection! • Alternative features (performance, mem usage, etc) • Activity statistics: How often did a brick do something? • Msgs received/sec, dropped/sec, etc. • Same across all peers, assuming balanced workload • Use anomalies as likely failures • State statistics: Current state of system • Memory usage, queue length, etc. • Similar pattern across peers, but may not be in phase • Look for patterns in time-series; differences in patterns indicate failure at a node.

29. Detecting Anomalous Conditions • Metrics compared against those of “peer” bricks • Basic idea: Changes in workload tend to affect all bricks equally • Underlying (weak) assumption: “Most bricks are doing mostly the right thing most of the time” • Anomaly in 6 or more (out of 9) metrics => reboot brick • Use different techniques for different stats • “Activity” – absolute median deviation • “State” – Tarzan time-series analysis

30. Network Fault – 70% packet loss in SAN Brick restarts Fault detectedBrick killed Network fault injected

31. J2EE as a platform for uRB-based recovery • Java 2 Enterprise Edition, a component framework for Internet request-reply style apps • App is a collection of components (“EJBs”) created by subclassing a managed container class • application server provides component creation, thread management, naming/directory services, abstractions for database and HTTP sessions, etc. • Web pages with embedded servlets and Java Server Pages invoke EJB methods • potential to improve all apps by modifying the appserver • J2EE has a strong following, encourages modular programming, and there are open source appservers

32. Separating data recovery from process recovery • For HTTP workloads, session state  app checkpoint • Store session state in a microrebootable session state subsystem (NSDI’04) • Recovery==non-state-preserving process restart, redundancy gives probabilistic durability • Response time cost of externalizing session state: ~25% • SSM, an N-way RAM-based state replication [NSDI 04] behind existing J2EE API • Microreboot EJB’s: • destroy all instances of EJB and associated threads • releases appserver-level resources (DB connections, etc) • discards appserver metadata about EJB’s • session state preserved across uRB

33. JBoss+uRB’s+SSM + fault injection Fault injection: null refs, deadlocks/infinite loop, corruption of volatile EJB metadata, resource leaks, Java runtime errors/exc RUBiS: online auction app (132K items, 1.5M bids, 100K subscribers) 150 simulated users/node35-45 req/sec/node Workload mix based on a commercial auction site Client-based failure detection

34. uRB vs. full RB - action weighted goodput • Example: corrupt JNDI database entry, RuntimeException, Java error; measure G_aw in 1-second buckets • Localization is crude: static analysis to associate failed URL with set of EJB’s, incrementing an EJB’s score whenever it’s implicated • With uRB’s, 89% reduction in failed requests and 9% more successful requests compared to full RB, despite 6 false positives

35. Performance overhead of JAGR • 150 clients/node: latency=38 msec (3 -> 7 nodes) • Human-perceptible delay: 100-200 msec • Real auction site: 41 req/sec, 33-300 msec latency

36. Improving availability from user’s point of view • uRB improves user-perceived availability vs. full reboot • uRB complements failover • (a) Initially, excess load on 2nd node brought it down immediately after failover • (b) uRB results in some failed requests (96% fewer) from temporary overload • (c,d) Full reboot vs. uRB without failover • For small clusters, should always try uRB first

37. uRB Tolerates Lax Failure Detection • Tolerates lag in detection latency (up to 53s in our microbenchmark) and high false positive rates • Our naive detection algorithm had up to 60% false positive rate in terms of what to uRB • we injected 97% false positives before reduction in overall availability equaled cost of full RB • Always safe to use as “first line of defense”, even when failover is possible • cost(uRB+other recovery)  cost(other recovery) • success rate of uRB on reboot-curable failures is comparable to whole-appserver reboot

38. Performance penalties • Baseline workload mix modeled on commercial site • 150 simulated clients per node, ~40-45 reqs/sec per node • system at ~70% utilization • Throughput ~1% worse due to instrumentation • worst-case response latency increases from 800 to 1200ms • Average case: 45ms to 80ms; compare to 35-300ms for commercial service • Well within “human tolerance” thresholds • Entirely due to factoring out of session state • Performance penalty is tolerable & worth it

39. Recovery and maintenance

40. Microrecovery for Maintenance Operations • Capacity discovery in SSM • TCP-inspired flow control keeps system from falling off a cliff • “OK to say no” is essential for this backpressure to work • Microrejuvenation in JAGR (proactively microreboot to fix localized memory leaks) • Splitting/coalescing in Dstore • Split = failure + reappearance of failed node • Same safe/non-disruptive recovery mechanisms are used to lazily repair inconsistencies after new node appears • Consequently, performance impact small enough to do this as an online operation

41. Using microrecovery for maintenance • Capacity discovery in SSM • redundancy mechanism used for recovery (“write many, wait few”) also used to “say no” while gracefully degrading performance

42. Full rejuvenation vs. microrejuvenation 76%

43. Splitting/coalescing in Dstore • Splitting/coalescing in Dstore • Split = failure + reappearance of failed node • Same mechanisms used to lazily repair inconsistencies

44. Summary: microrecoverable systems • Separation of data from process recovery • Special-purpose data stores can be made microrecoverable • OK to initiate microrecovery anytime for any reason • no loss of correctness, tolerable loss of performance • likely (but not guaranteed) to fix an important class of transients • won’t make things worse; can always try “full” recovery afterward • inexpensive enough to tolerate “sloppy” fault detection • low-cost first line of defense • some “maintenance” ops can be cast as microrecovery • due to low cost, “proactive” maintenance can be done online • can often convert unplanned long downtime into planned shorter performance hit

45. Anomaly detection as failure detection

46. Example: Anomaly Finding Techniques Question: does anomaly == bug? * Includes design time and build time ** Includes both offline (invasive) and online detection techniques

47. Examples of Badness Inference • Sometimes can detect badness by looking for inconsistencies in runtime behavior • We can observe program-specific properties (though using automated methods) as well as program-generic properties • Often, we must be able to first observe program operating “normally” • Eraser: detecting data races [Savage et al. 2000] • Observe lock/unlock patterns around shared variables • If a variable usually protected by lock/unlock or mutex is observed to have interleaved reads, report a violation • DIDUCE: inferring invariants, then detecting violations [Hangal & Lam 2002] • Start with strict invariant (“x is always =3”) • Relax it as other values seen (“x is in [0,10]”) • Increase confidence in invariant as more observations seen • Report violations of invariants that have threshold confidence

48. Generic runtime monitoring techniques • What conditions are we monitoring for? • Fail-stop vs. Fail-silent vs. Fail-stutter • Byzantine failures • Generic methods • Heartbeats (what does loss of heartbeat mean? Who monitors them?) • Resource monitoring (what is “abnormal”?) • Application-specific monitoring: ask a question you know the answer to • Fault model enforcement • coerce all observed faults to an “expected faults” subset • if necessary, take additional actions to completely “induce” the fault • Simplifies recovery since fewer distinct cases • Avoids potential misdiagnosis of faults that have common symptoms • Note, may sometimes appear to make things “worse” (coerce a less-severe fault to a more-severe fault) • Doesn’t exercise all parts of the system

49. Internet performance failure detection • Various approaches, all of which exploit the law of large numbers and (sort of) Central Limit Theorem (which is?) • Establish “baseline” of quantity to be monitored • Take observations, factor out data from known failures • Normalize to workload? • Look for “significant” deviations from baseline • What to measure? • Coarse-grain: number of reqs/sec • Finer-grain: Number of TCP connections in Established, Syn_sent, Syn_rcvd state • Even finer: additional internal request “milestones” • Hard to do in an application-generic way...but frameworks can save us

50. Example 1: Detection & recovery in SSM • 9 “State” statistics collected per second from each replica • Tarzan time series analysis* compares relative frequencies of substrings corresponding to discretized time series • “anomalous” => at least 6 stats “anomalous”; works for aperiodic or irregular-period signals • robust against workload changes that affect all replicas equally and against highly-correlated metrics *Keogh et al., Finding surprising patterns in a time series database in linear time and space, SIGKDD 2002