1 / 33

Elicitation and Utilization of Application-level Utility Functions

Elicitation and Utilization of Application-level Utility Functions. Paul de Grandis and Giuseppe Valetto Drexel University - Dept. of Computer Science. Outline. Motivation Goal Approach Engineering considerations Experiments Conclusions. Motivation.

viveca
Download Presentation

Elicitation and Utilization of Application-level Utility Functions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Elicitation and Utilization of Application-level Utility Functions Paul de Grandis and Giuseppe Valetto Drexel University - Dept. of Computer Science

  2. Outline • Motivation • Goal • Approach • Engineering considerations • Experiments • Conclusions

  3. Motivation • Autonomic provisions are often costly and complex • Many times, those provisions target only specific classes of problems • expected or likely conditions • It’s not only a matter of requirements … • … it’s also hard to develop complete system models that enable general-purpose adaptations

  4. The challenge • Unexpected conditions are often the most critical • Typically occur only on the field • By definition, they have not been accounted for by designers, programmer, testers etc. • Potentially devastating for the system • One could argue that autonomic provisions fulfill their promise when they can handle the unexpected

  5. Designing for the unexpected

  6. Our goal • Enable non-analytic self-assessment of SW applications • Self-assessment: map the state space of the application to some abstraction of value • Recognize where the system on in that map • Non-analytic: do not require a complete model of the system in order to enable self-assessment

  7. “bad” state “good” states Two-steps strategy Self-adaptation Self-assessment ? State space

  8. Approach: application-based utility

  9. Utility of an application • What is the app. value for, e.g., its client? • A single scalar: Uapp [0,1] • Utility function: Uapp = f(V[s0,…,sn]) • s0,…,snare state variables • Network measures • System measures • Application specific measures • User-observable measures

  10. Our vision • Elicitation: construct Uapp for a given application • Often, for a given task or feature • Embedding: insert runtime code to monitor state variables and compute Uapp • Utilization: continuously evaluate Uapp • We are interested in a method that can handle legacy applications as well as new ones

  11. Our process • Identify utility from the client perspective • Instrument application and environment • Run experiments and observe V[s0, …,sn] • Synthesize Uapp(V[]) • Substitute Uappin the instrumented code • Manual, expert knowledge • Automated • Automated • Off-line, statistical correlation • Automated

  12. Observation and synthesis Statistical correlation Application Environment • We operate at multiple distinct levels • Typically, application- vs. environment-level • Collect lots of lower-level monitoring data from the running environment … • … and relate it statistically to some externally visible manifestation of application goals

  13. Example – FTP service • Simple protocol-based application • Utility is straightforward: • For an FTP client, utility = client throughput • Relevant environment (socket-level) parameters: • Latency • Bandwidth • Loss rate • …

  14. Eliciting application utility U Applicationlevel t Environmentlevel

  15. Embedding application utility U Applicationlevel t Environmentlevel

  16. Engineering approach

  17. Elicitation and embedding framework • Aupy: “Application Utility in Python” • Framework leveraging meta-programming capabilities of Python to instrument applications • Two flavors: • Python annotations (legacy code-friendly) • Python contexts (granular and flexible) • Open-sourced at: http://www.aupy.org

  18. Aupy annotations • Added to method definitions • @monitor(foo) bar triggers evaluation of utility function “foo” when “bar” is executed • u.f.method declarations: • @pre-utility foo • @post-utility foo • @utility foo • @concurrent-utility foo • foo() can contain monitoring code (elicitation) or code computing Uapp (embedding)

  19. Aupy contexts • Contexts are Python scoping constructs • “with” keyword establishes a custom scope • Aupy uses them to associate utility functions to arbitrary blocks of code • Finer-grained control • You can pass to a context utility function, arguments, configuration data, callback information …

  20. Benefits of contexts • Contexts (and hence u.f.) can be composed • Multi-step u.f. • Utility of a u.f. • Support of selection logic among multiple alternative u.f.

  21. Experiments

  22. FTP experiment set-up • In-the-lab data collection • 125 test runs using dummynet to throttle network • create a uniform map of environmental conditions • Two download scenarios: • Bulk-ftp: transfer one large file • Small-ftp: transfer multiple small files • Off-line data analysis and u.f. synthesis

  23. FTP results • X=inter-arrival time • Y= socket throughput • Very different u.f. for the 2 scenarios • We plugged in elicited u.f. • kept monitoring client throughput online and compared

  24. FTP-bulk results Estimated Actual

  25. Another experiment – log explosion • Real-world case study • Logging component of a Python-powered popular WWW site gone awry • Explosive growth of log would block the WWW site in a few seconds • Even prevent remote login from sysadmins • Could we apply non-analytic self-assessment to this problem?

  26. Log explosion experiment • Phase 1: catching log explosions • Utility of WWW site drops almost immediately to 0 upon log explosion • need to quickly capture acceleration peaks in disk writes per process • before disk resource is exhausted • Phase 2: preventing log explosions: • hypothesis: utility is Uapp ≈ f(1-pnboom) • pboom : probability that a log explosion is going to happen “some time soon”

  27. Log explosion results • We were able to catch log explosions • Captured log explosion dynamics for disk writes • Write speed raises 2 orders of magnitude each second • Uapp looks like a step function • Collapse occurs in a few seconds • But we could detect the front and turn off logging in time • The prevention problem is more complex • Would require very frequent sampling of disk write speed • That interferes too much with normal operation • We cannot really “see” preliminary signs and estimate “pboom”

  28. Conclusions

  29. Lessons learned • We have understood the elicitation process • Repeatable • Requires only a modicum of domain knowledge • We have acquired know-how on the mechanisms for elicitation and embedding • Can be made minimally intrusive • Can be applied to legacy systems • Layered approach is generally valid

  30. Lessons learned • We have insight on the task-based nature of application utility • Need an array of u.f. that can be selected opportunistically • On a per-feature basis • Also based on different tasks • Idea applies especially well to: • Protocol-based applications • Infrastructure-level services

  31. Research directions • Towards in-the-field data collection • Deploy Aupy-instrumented application • Multiple features are exercised • Assign clusters in the monitored space to exercised features • Elicit all feature-specific u.f. from the clustered areas

  32. Research directions • Towards automated utility definition • Use cooperative negotiation and dependency discovery to figure which application characteristics to consider as utility • Leverage utility map to guide adaptations • Reinforcement learning to compute trajectories back from bad utility to good utility states • Actionable trajectories: can be traversed by tuning some state variables • Utility delta as RL “reward”

  33. Questions?

More Related