1 / 51

Yongle Zhang, Serguei Makarov, Xiang Ren, David Lion, Ding Yuan

Pensieve : Non-Intrusive Failure Reproduction for Distributed Systems using the Event Chaining Approach. Yongle Zhang, Serguei Makarov, Xiang Ren, David Lion, Ding Yuan. Failure Reproduction is Time-Consuming. Understand bug Verify fix. $156 Billion i n 2013.

zada
Download Presentation

Yongle Zhang, Serguei Makarov, Xiang Ren, David Lion, Ding Yuan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pensieve: Non-Intrusive Failure Reproduction for Distributed Systems using the Event Chaining Approach Yongle Zhang, Serguei Makarov, Xiang Ren, David Lion, Ding Yuan

  2. Failure Reproduction is Time-Consuming • Understand bug • Verify fix $156 Billion in 2013

  3. A Real-World Debugging Procedure Provided error description, log & reproduction steps. Can’t reproduce. Added some reproduction steps. Still cannot reproduce. Could you upload fsimage? Fsimage uploaded. Revised reproduction steps. Reporter Developer Still cannot reproduce. HDFS-6130

  4. A Real-World Debugging Procedure Provided error description, log & reproduction steps. Can’t reproduce. Added some reproduction steps. Still cannot reproduce. Could you upload fsimage? Fsimage uploaded. Revised reproduction steps. Reporter Developer Still cannot reproduce. . . . [5 days, 29 discussions] Reproduced… HDFS-6130 [after another 8 minutes] Posted the working patch.

  5. Pensieve: Non-Intrusive Failure Reproduction • Analyzes Java bytecode & log • Non-intrusive • Works on JVM based system

  6. Pensieve: Non-Intrusive Failure Reproduction • Analyzes Java bytecode & log • Non-intrusive • Works on JVM based system Command Line Input (HDFS-4022) ./pensieve -jar ./hadoop-hdfs-2.0.0-alpha.jar // Java bytecode -log ./HDFS-logs/ // failure logs -error ./HDFS-logs/datanode-2.log#800 // symptoms

  7. Pensieve: Non-Intrusive Failure Reproduction • Analyzes Java bytecode & log • Non-intrusive • Works on JVM based system Command Line Input (HDFS-4022) ./pensieve -jar ./hadoop-hdfs-2.0.0-alpha.jar // Java bytecode -log ./HDFS-logs/ // failure logs -error ./HDFS-logs/datanode-2.log#800 // symptoms . . 800: ERROR: invalid block . datanode-2.log

  8. Pensieve: Non-Intrusive Failure Reproduction • Analyzes Java bytecode & log • Non-intrusive • Works on JVM based system Output Unit Test (HDFS-4022) initCluster(1); // start 1-machine cluster create(“a.txt”,2); // create 2-replica file append(“a.txt”,“X”); // append to file addDataNode(); // add a datanode on the fly

  9. Existing Solutions Are Limited • Record-and-replay (deterministic replay) • Intrusive: modifies existing software stack • Incurs performance overhead • Symbolic Execution • E.g., ESD [Zamfir EuroSys’10], SherLog [Yuan ASPLOS’10]. • Pros: precise & non-intrusive • Cons: hard to scale to large systems

  10. Scalability of Symbolic Execution 1.for(inti=0;i<blocks.length;i++){ 2. if(blocks[i].genStamp!=VALID_GS) 3. log("invalid block. . ."); 4.} Simplified code snippet for HDFS-4022 • Enumerates every possible execution path

  11. Scalability of Symbolic Execution 1.for(inti=0;i<blocks.length;i++){ 2. if(blocks[i].genStamp!=VALID_GS) 3. log("invalid block. . ."); 4.} Simplified code snippet for HDFS-4022 • Enumerates every possible execution path blocks[i].genStamp!=VALID_GS

  12. Scalability of Symbolic Execution 1.for(inti=0;i<blocks.length;i++){ 2. if(blocks[i].genStamp!=VALID_GS) 3. log("invalid block. . ."); 4.} Simplified code snippet for HDFS-4022 • Enumerates every possible execution path blocks[0].genStamp!=VALID_GS blocks[i].genStamp!=VALID_GS blocks[1].genStamp!=VALID_GS && blocks[0].genStamp==VALID_GS . . .

  13. Scalability of Symbolic Execution 1.for(inti=0;i<blocks.length;i++){ 2. if(blocks[i].genStamp!=VALID_GS) 3. log("invalid block. . ."); 4.} Simplified code snippet for HDFS-4022 • Enumerates every possible execution path blocks[0].genStamp!=VALID_GS OR blocks[i].genStamp!=VALID_GS blocks[1].genStamp!=VALID_GS && blocks[0].genStamp==VALID_GS OR . . .

  14. Scalability of Symbolic Execution 1.for(inti=0;i<blocks.length;i++){ 2. if(blocks[i].genStamp!=VALID_GS) 3. log("invalid block. . ."); 4.} Simplified code snippet for HDFS-4022 • Enumerates every possible execution path

  15. Core Idea – Partial Trace Observation Developers almost never debug a failure by reconstructing its complete execution path. Instead, they construct a simplified trace which only contains events that are likely to be causally relevant to the failure.

  16. How do developers debug HDFS-4022? for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . .");

  17. How do developers debug HDFS-4022? if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . ."); void rollLog(. . .){ b.genStamp=logGS; }

  18. How do developers debug HDFS-4022? if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . ."); void rollLog(. . .){ b.genStamp=logGS; }

  19. How do developers debug HDFS-4022? if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . .");

  20. How do developers debug HDFS-4022? if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . .");

  21. How do developers debug HDFS-4022? Client.java void append(. . .){ stage=APPEND; . . . } if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . .");

  22. How do developers debug HDFS-4022? Network de-serialization serialization DataNode.java Client.java void append(. . .){ stage=APPEND; . . . } if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . .");

  23. How do developers debug HDFS-4022? Network de-serialization serialization DataNode.java Client.java void append(. . .){ stage=APPEND; . . . } if(this.stage==APPEND){ log("Appending" + b); b.genStamp=newGS; } Got one user command (append) by looking at 8 instructions! for(inti=0;i<blocks.length;i++) if(blocks[i].genStamp!=VALID_GS) log("invalid block. . .");

  24. Event Chaining Approach • Event – a point in time during execution • Location event – a program location reached • Condition event – a condition holds • Invocation event – a function invoked

  25. Event Chaining Approach Location event Condition event Invocation event • An event is explained by other events • Location event  path conditions 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); 888: ERROR: invalid block… datanode-2.log e1:line2(L2)

  26. Event Chaining Approach Location event Condition event Invocation event • An event is explained by other events • Location event  path conditions 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); 888: ERROR: invalid block… e2:blocks[i].genStamp !=VALID_GS@L1 datanode-2.log e1:L2

  27. Event Chaining Approach Location event Condition event Invocation event • An event is explained by other events • Condition event  definitions 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); e3:L5 888: ERROR: invalid block… e2:blocks[i].genStamp !=VALID_GS@L1 datanode-2.log e1:L2

  28. Event Chaining Approach Location event Condition event Invocation event • An event is explained by other events 6.void append(. . .){ 7. stage=APPEND; … . . . … } 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } e5:L7 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); e4:this.stage==APPEND@L3 e3:L5 888: ERROR: invalid block… e2:blocks[i].genStamp !=VALID_GS@L1 datanode-2.log e1:L2

  29. Event Chaining Approach Location event Condition event Invocation event • An event is explained by other events • Location event  function invocation 6.void append(. . .){ 7. stage=APPEND; … . . . … } 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } e6:append() e5:L7 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); e4:this.stage==APPEND@L3 e3:L5 888: ERROR: invalid block… e2:blocks[i].genStamp !=VALID_GS@L1 datanode-2.log e1:L2

  30. Event Chaining Approach Location event Condition event Invocation event • Captures dependency on shared variables 6.void append(. . .){ 7. stage=APPEND; … . . . … } 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } Thread 1 e6:append() Thread 2 e5:L7 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); e4:this.stage==APPEND@L3 e3:L5 888: ERROR: invalid block… e2:blocks[i].genStamp !=VALID_GS@L1 datanode-2.log e1:L2

  31. Forking for Multiple Possibilities e2:blocks[i].genStamp !=VALID_GS@L1 e1:L2 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . .");

  32. Forking for Multiple Possibilities e7:L8 e3:L5 e2 e2:blocks[i].genStamp !=VALID_GS@L1 fork e1:L2 e1 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); … void rollLog(. . .){ 8. b.genStamp=logGS; … }

  33. Priority based scheduling P:1000 e7:L8 P:500 P:500 e3:L5 e2 e2:blocks[i].genStamp !=VALID_GS@L1 fork e1:L2 e1

  34. Priority based scheduling e7:L8 P:500 P:500 e3:L5 e2:blocks[i].genStamp !=VALID_GS@L1 e2 e1:L2 e1 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); … void rollLog(. . .){ 8. b.genStamp=logGS; … }

  35. Priority based scheduling e7:L8 P:0 P:1000 e3:L5 e2:blocks[i].genStamp !=VALID_GS@L1 e2 e1:L2 e1 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } s . . Appending… . . log files 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . ."); … void rollLog(. . .){ 8. b.genStamp=logGS; … }

  36. Priority based scheduling • Favors event chains with most matched logs • Favors simpler reproduction paths

  37. Eliminating Infeasible Event Chains path conditions • Path conditions • Variable substitution • Logical conjunction blocks[i].genStamp!=VALID_GS@L1 e2:blocks[i].genStamp !=VALID_GS@L1 e1:L2 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . .");

  38. Eliminating Infeasible Event Chains path conditions • Path conditions • Variable substitution • Logical conjunction blocks[i].genStamp!=VALID_GS@L1 newGS!=VALID_GS@L5 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } e3:L5 e2:blocks[i].genStamp !=VALID_GS@L1 e1:L2 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . .");

  39. Skip Less-Relevant Loops • Skips loops when there’s no loop carried dependency • 77% of randomly sampled loops in HDFS • Follows loop iterations otherwise 3.if(this.stage==APPEND){ 4. log("Appending" + b); 5. b.genStamp=newGS; … } e3:L5 e2:blocks[i].genStamp !=VALID_GS@L1 e1:L2 0.for(inti=0;i<blocks.length;i++) 1. if(blocks[i].genStamp!=VALID_GS) 2. log("invalid block. . .");

  40. Verification & Testcase Refinement • Sacrifices precision for efficiency Event Chain e3: i=2; e2: i>1 e1: log(“ERROR”);

  41. Verification & Testcase Refinement • Sacrifices precision for efficiency Event Chain Execution e3: i=2; e2: i>1 e1: log(“ERROR”);

  42. Verification & Testcase Refinement • Sacrifices precision for efficiency Event Chain Execution e3: i=2; e2: i>1 X Diverged! e1: log(“ERROR”); …

  43. Verification & Testcase Refinement • Sacrifices precision for efficiency e3: i=2 Event Chain Execution e5: N>0 e4: i=0; e2: i>1 X Diverged! e1: log(“ERROR”); …

  44. Verification & Testcase Refinement • Sacrifices precision for efficiency e3: i=2 Event Chain e3: i=2 Execution e5: N>0 e6: N<=0 e4: i=0; e2: i>1 e2: i>1 e1: log(“ERROR”); X Diverged! e1: log(“ERROR”); …

  45. Verification & Testcase Refinement • Variable modified in a different thread (in paper) e3: i=2 Event Chain e3: i=2 Execution e5: N>0 e6: N<=0 e4: i=0; e2: i>1 e2: i>1 e1: log(“ERROR”); X Diverged! e1: log(“ERROR”); …

  46. Evaluation • Evaluated on 18 cases from JVM distributed systems • HDFS, HBase, ZooKeeper, Cassandra • with noisy logs generated from manual reproduction • Overall Result • Successfully reproduces 72% • Finishes analysis within 10 min

  47. Case Study: HDFS-6130 path conditions inferred • Useful for hard bugs fsimage.layoutVersion!=TXID_LAYOUT // Use old fsimage initCluster(UPGRADE); restartCluster(); initCluster(UPGRADE); restartCluster(); Pensieve’s reproduction Developers’ reproduction

  48. Case Study: HDFS-4022 • Finds different reproduction than developers’ initCluster(3); setConfig(“policy”, “ALWAYS”); create(“a.txt”,2); stopDataNode(1); append(“a.txt”,“data”); initCluster(4); create(“a.txt”,3); stopDataNode(3); append(“a.txt”,“data”); Pensieve’s reproduction –fewer nodes! Developers’ reproduction

  49. Limitations • Error is not logged (e.g., silent data loss) • Bugs involving resource exhaustion • Systems need to have clearly defined input events. • E.g., not for compilers.

  50. Related Work • Static program slicing [Weiser81] • Obtains static trace but not dynamic partial trace • Symbolic execution based approach • ESD, SherLog. • Record-and-replay based approach • BugRedux [Wei ICSE’12], etc.

More Related