1 / 45

From C IEL to Firmament & DIOS a heavenly tale of not just clouds

From C IEL to Firmament & DIOS a heavenly tale of not just clouds. Joint work with:. Steven Hand Anil Madhavapeddy Chris Smowton Steven Smith Derek Murray (MSR-SVC). Disclaimer . Recap: C IEL. [NSDI 2011]. A. B. G. M. M. M. R. R. R. Dynamic task graphs. G. a. M. M. T. x.

kobe
Download Presentation

From C IEL to Firmament & DIOS a heavenly tale of not just clouds

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From CIEL to Firmament & DIOSa heavenly tale of not just clouds Joint work with: Steven HandAnil MadhavapeddyChris SmowtonSteven Smith Derek Murray (MSR-SVC)

  2. Disclaimer 

  3. Recap: CIEL [NSDI 2011]

  4. A B G M M M R R R

  5. Dynamic task graphs G a M M T x b R R Allow tasks to spawn more tasks

  6. Experiment from D. Murray, A distributed execution engine supporting data-dependent control flow. PhD thesis, University of Cambridge, 2011.

  7. [interlude] polyglot CIEL

  8. polyglot CIEL [unpublished]

  9. CIEL Saving state – options heavyweight lightweight VM migration BLCR (processcheckpoint.) Serializablecontinuations Haskellmonads hardware / OS level application level

  10. JavaScalaHaskellStackless PythonOCaml(C with BLCR)  no need for Skywriting!

  11. Binomial options pricing Experiment from D. Murray, C. Smowton, M. Schwarzkopf, S. Smith, A. Madhavapeddy. A polyglot approach to cloud programming. Unpublished, 2011.

  12. What‘s next?! many-core clustersheterogeneity

  13. timespin on unmodified CIEL 41.6x rel. overhead 5.1x less is better 1.3x seconds 1.04x number of cores

  14. [Data-Intensive Operating System] EnterFirmamentandDIOS

  15. ... Host OS ... Hardware User code CIEL Programming Model 1st class exec. 2nd Skywriting Execution Engine Master W0 W1 Wn ... ...

  16. DIOS ... User code CIEL Programming Model 1st class exec. 2nd Skywriting Firmament: Coordination Engine ... Hardware

  17. Firmament multi-scaleheterogeneity-aware

  18. How much heterogeneity is there?

  19. Google trace, machine platforms

  20. Google trace, machine specs CPU cores (normalized) Total RAM (normalized)

  21. Google trace, platforms + specs

  22. Google trace, machine attributes

  23. Firmament Cluster knowledge base • historic task resource usage historic task performance info machine information Efficient runtime [Storage? Networking? Transfer management?]

  24. Firmament It’s real! • ~2k LOC, basic tests run ToDo (aka WIP): • knowledge base design & impl. scheduling algorithms interface to CIEL

  25. DIOS ... User code CIEL Programming Model 1st class exec. 2nd Skywriting Firmament: Coordination Engine ... Hardware

  26. DIOS topology-awareinterference-awarelightweight OS

  27. Heterogeneity [again!] Many-core => intra-machine communication = important!

  28. Intel Core i7-2600K @ 3.40GHz (native) 48-core AMD Opteron 6168 (native) (Xen) Joint work with Steven Smith, Anil Madhavapeddy, and Chris Smowton; cf. “The case for reconfigurable I/O“ (RESoLVE 2012)

  29. Intel Xeon E5620 @ 2.40GHz (native) Different physicalcore Hyperthread Joint work with Steven Smith, Anil Madhavapeddy, and Chris Smowton; cf http://fable.io

  30. Intel Core i7-2600K @ 3.40GHz (native) Different physicalcore Hyperthread Joint work with Steven Smith, Anil Madhavapeddy, and Chris Smowton; cf http://fable.io

  31. AMD Opteron 6168@ 1.9 GHz (native) Same MCM, same socket Different MCM,different socket,2-hop Hypertransport Joint work with Steven Smith, Anil Madhavapeddy, and Chris Smowton; cf http://fable.io

  32. Topology-awareness OS responsibility? Yes. General case = hard! Workload-awareness helps!

  33. hwloc

  34. Interference #include <results>

  35. Lightweight Make the OS do exactly (and just) what is needed. Dedicate resources instead of sharing them.

  36. Resourcemultiplexing Process mgmt Isolation I/O mgmt Concurrency primitives Multi-threading Pre-emption Locking Filesystem Shell Standard libs

  37. Scheduling a T x b

  38. DIOS ... Scheduling Firmament: Coordination Engine ... Program

  39. DIOS Pieces exist • currently combining ;-) WIP: • interference experiments related work reading group starting point? (Linux or Xen?)

  40. BACKUP SLIDES

  41. Binomial options pricing 800k (EC2) 800k (MC) 400k (EC2) 400k (MC) 200k (EC2) higheris better 200k (MC)

  42. Redis example Numbers and experiment by Sören Bleikertz: http://openfoo.org/blog/redis-native-xen.html

More Related