1 / 17

Building User-defined Runtime Adaptation Routines for Stream Processing Applications

Building User-defined Runtime Adaptation Routines for Stream Processing Applications. Gabriela Jacques-Silva, Bugra Gedik, Rohit Wagle, Kun-Lung Wu, Vibhore Kumar. Outline. Motivation Challenge for orchestration Building a management policy Use case.

booker
Download Presentation

Building User-defined Runtime Adaptation Routines for Stream Processing Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building User-defined Runtime Adaptation Routines forStream Processing Applications Gabriela Jacques-Silva, Bugra Gedik, Rohit Wagle, Kun-Lung Wu, Vibhore Kumar

  2. Outline • Motivation • Challenge for orchestration • Building a management policy • Use case

  3. In System S, users build streaming applications as data flow graphs composite operator stream connection processing element (PE)

  4. Streaming applications have single deployment mode submit application Application Manager start PEs

  5. Users want adaptability, but currently this requires ad-hoc solutions submit application Application Manager start PEs sleep, cat, grep, cut, …

  6. Orchestrator controls the life-cycle of an application via event handling onStart: submitJob(MyApp); onOperatorMetric: … onPEFailure: … onUserEvent: … submit orchestrator Application Manager orchestrator submit application • Use cases • Recovery coordination on operator failures • Operator adaptation due to metric measurement values • Incremental job deployment based on runtime learning

  7. Events and actions occur at the physical level, but users are familiar with logical view Logical view Physical view Host A PE 3 op4’ op4’ op3’ op3’ op6’ op6’ Host B op1 op1 op5’ op5’ PE 4 PE 1 composite1 op2 op2 op4’’ op4’’ op2 op6’’ op2 op6’’ op3’’ op3’’ op5’’ op5’’ PE 2 composite2 Orchestrator maintains an in-memory application representation to disambiguate logical and physical views when necessary.

  8. Developers register events of interest based on the logical view of the application void MyOrca::handleOrcaStart( const OrcaStartContext & context) { PEFailureScope greenS(“green”); PEFailureScope blueS(“blue”); greenS.addCompositeInstanceFilter(“sample::greenComp”); _orca->registerEventScope(greenS); blueS.addOperatorInstanceFilter(“sample::blueOp”); _orca->registerEventScope(blueS); id = submitJob(sample::MyMainComp); }

  9. On event delivery, we provide a context to understand logical and physical views void MyOrca::handlePEFailure( const PEFailureContext & context) if(context.hasScope(“green”)) { list<PEID> pes = new list<PEID>(); _orca->getPEsWithCompositeInstance( pes, “sample::greenComp”); _orca->restartPEs(pes); } else { _orca->executeCommand(“./myRecoveryHelper”); _orca->restartPE(context.getPEId()); } } PE failure context: - Matched scopes - Application name - Job ID - PE ID - Reason

  10. Orchestrator allows management of multiple streaming applications • Dependency relations • Uptime requirements • Garbage collectable

  11. Developer configures application requirements and specifies its dependency 10.0 T, 60 void MyOrca::handleOrcaStart( const OrcaStartContext & context) { AppConfig fb(“Facebook”, “sample::Facebook”, true, 60.0); AppConfig tw(“Twitter”, “sample::Twitter”, false); AppConfig sc(“Social”, “sample::SNetwork”, true, 60.0); _orca->registerDependency(sc, fb, 10.0); _orca->registerDependency(sc, tw, 10.0); _orca->submitJob(sc); } T, 60 F, 0 10.0

  12. Orchestrator starts applications according to their dependencies and uptime requirements 10.0 T, 60 T, 60 void MyOrca::handleOrcaStart( const OrcaStartContext & context) { … _orca->registerDependency(sc, fb, 10.0); _orca->registerDependency(sc, tw, 10.0); _orca->submitJob(sc); } F, 0 10.0 • Orchestrator log • 10:00:00 AM – “Facebook” submitted • 10:00:02 AM – “Twitter” submitted • 10:00:12 AM – “Social” submitted

  13. Unused applications are automatically cancelled according to their configurations and dependencies 10.0 T, 60 T, 60 void MyOrca::handleSomeEvent( const SomeEventContext & context) { … _orca->cancelJob(sc); } F, 0 10.0 • Orchestrator log • 11:00:00 AM – “Social” cancelled • 11:01:00 AM – “Facebook” cancelled

  14. Example demonstrator uses custom operator metrics to adapt a Sentiment Analysis applications Sentiment Analysis Model computation

  15. Orchestrator helped streaming application to adapt to the incoming data distribution Actuation threshold MapReduce job submitted New model loaded new stream of tweets injected

  16. Summary • System S hides the complexity of parallel and distributed systems • Orchestrator must open this abstraction to allow appropriate event handling and actuation • APIs for event registration based on logical view • Event delivery includes context to understand physical view • Future work • Add orchestrator concepts to SPL • Use static analysis to automatically infer event scope

  17. Building User-defined Runtime Adaptation Routines forStream Processing Applications Gabriela Jacques-Silva, Bugra Gedik, Rohit Wagle, Kun-Lung Wu, Vibhore Kumar

More Related