1 / 70

Concurrent Revisions And Cloud Types

Concurrent Revisions And Cloud Types. Sebastian Burckhardt In collaboration with Daan Leijen, Manuel Fähndrich , Alexandro Baldassin, Benjamin Wood, Mooly Sagiv, Yuelu Duan, Alexey Gotsman, Hongseok Yang. Overview. Part I: Concurrent Revisions Summary of prior work(What led me here)

galeno
Download Presentation

Concurrent Revisions And Cloud Types

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Concurrent RevisionsAnd Cloud Types Sebastian Burckhardt In collaboration with Daan Leijen, Manuel Fähndrich, Alexandro Baldassin, Benjamin Wood, Mooly Sagiv, Yuelu Duan, Alexey Gotsman, Hongseok Yang

  2. Overview Part I: Concurrent Revisions Summary of prior work(What led me here) Part II: Concurrent Revisions and Distributed Systems Motivation: • Programming Apps for Mobile+Cloud Proposed Solution: • Revision Consistency • Cloud Types Part III: How to make it real: TouchDevelop

  3. Part I Concurrent Revisions

  4. Parallel tasks: Pick any two Parallel Performance FrequentConflicts Serializability

  5. Our pick Parallel Performance FrequentConflicts Serializability Revisions

  6. Concurrent Revisions 101 fork • When forking a task, state is copied as well. • Task operates on its copy of the data in isolation. • When task is joined, changes are merged. • The merge is fully defined by the data type declarations. • some types may include custom merge functions • there is no failure , rollback, or retry fork B fork A C D join join

  7. Good for Parallel Programming • On multicore : efficient thanks to copy-on-write • Studied game application [OOPSLA 2010]: revisions provide more parallelization, better performance • Programming model has good properties: Deterministic Parallel Programming [ESOP 2010] • Can be extended to express both parallel and incremental computation: “Two for the price of one” [OOPSLA 2011], Distinguished Paper Award

  8. Application Example: SpaceWars3D Game Revisions helped with these challenges: • Need stable snapshot for rendering task • Need to parallelize tasks that may write to same data(physics, collisions, network) • Need to allow slow background tasks (e.g. autosave) to work on snapshot

  9. Application Example: SpaceWars3D Game Revision Diagram of Parallelized Game Loop Collision Detection part 1 Render Physics network part 4 part 3 part 2 autosave (long running)

  10. Application Example: SpaceWars3D Game Eliminated Read-Write Conflicts Collision Detection part 1 Render Physics network part 4 part 3 part 2 autosave (long running) All tasks see stable snapshot

  11. Application Example: SpaceWars3D Game Eliminated Write-Write Conflicts Collision Detection part 1 Render Physics network part 4 part 3 part 2 autosave (long running) Network after CD after Physics

  12. A : integer = 0; B : integer = 0; Understanding Concurrent RevisionsOperation-Based Interpretation • Current state determined by update sequence along path from root. • Tip of arrow (arrow = end of a revision) count as the aggregate of all operations along the revision A.Set(2) B.Set(2) A.Get() -> 0 A.Set(1) A.Get() -> 1 B.Get() -> 2

  13. Set(A,0) Set(B,0) Sees only the initialization operation A.Set(2) B.Set(2) • Current state determined by update sequence along path from root. • Tip of arrow (arrow = end of a revision) count as the aggregate of all operations along the revision A.Get() -> 0 A.Set(1) A.Get() -> 1 B.Get() -> 2

  14. A.Set(0) B.Set(0) • Current state determined by update sequence along path from root. • Tip of arrow (arrow = end of a revision) count as the aggregate of all operations along the revision A.Set(2) B.Set(2) A.Get() -> 0 A.Set(1) A.Set(2) B.Set(2) A.Set(1) A.Get() -> 1 B.Get() -> 2

  15. A : integer = 0 Puzzle 1 A.Add(1) A.Add(2) A.Get() -> ?

  16. A : integer = 0 Puzzle 1 A.Set(0) Answer: Updates along path: A.Set(0) A.Add(1) A.Add(2) Result: 3 A.Add(1) A.Add(2) A.Add(1) A.Add(2) A.Get() -> 3

  17. Puzzle 2 A : integer = 0 A.Add(1) A.Set(1) A.Add(1) A.Get() -> ?

  18. Puzzle 2 A.Set(0) Answer Updates along path: A.Set(0) A.Add(1) A.Set(1) A.Add(1) Result: 2 A.Add(1) A.Set(1) A.Add(1) A.Set(1) A.Add(1) A.Get() -> 2

  19. S : string = “” Puzzle 3 S.Append(“1”) S.Append(“2”) S.Append(“3”) S.Get() -> ? S.Append(“4”) S.Get() -> ?

  20. S.Set(“”) Puzzle 3 S.Append(“1”) Answer 1 Updates along path: S.Set(“”) S.Append(“1”) S.Append(“3”) Result: “13” S.Append(“2”) S.Append(“3”) S.Get()->“13” S.Append(“4”) S.Get() -> ?

  21. S.Set(“”) Puzzle 3 S.Append(“1”) Answer 2 Updates along path: S.Set(“”) S.Append(“1”) S.Append(“2”) S.Append(“3”) S.Append(“4”) Result: “1234” S.Append(“2”) S.Append(“3”) S.Append(“1”) S.Append(“2”) S.Get()->“13” S.Append(“3”) S.Append(“4”) S.Append(“3”) S.Append(“4”) S.Get() -> “1234”

  22. Visibility & Arbitration in Revision Diagrams • Visibilitywho can see what updates?= Reachabilityis there a (directed) path? • Arbitrationwhose update goes first?= Cactus Walk 1 2 7 4 3 6 5 8 9

  23. Not everything is a revision diagram: The join condition Revision diagrams are subject to the join condition:A revision can only be joined into vertices that are reachable from the fork. Invalid join, no path from fork.

  24. Without join condition, causality may be violated. • B sees updates of A • C sees updates of B • But C does not see updates of A • Without join condition, visibility is not transitive. • We prove in paper: enforcing join condition is sufficient to guarantee transitive visibility. A B C

  25. Conclusion of Part I • Revision Diagrams • Make replication explicit • Provide a principled way to understand and define the effect of concurrent conflicting updates • Concurrent Revisions • Can use revisions as a programming model to achieve better performance, or to express incremental + parallel algorithms

  26. Part II Revisions & Distributed Systems

  27. Revisions + Distributed Systems • Can think of many applications. • Revision pattern is commonly used for: • Source control systems(data structured as file systems, with per-file merge operations) • Classic web applicationsLoad HTML form – edit locally – submit – server does merge • Modern web applicationsread REST object – javascript modifies locally – write REST object • We are currently focusing on this programming domain:Apps for Mobile + Cloud.

  28. Revisions + Distributed Systems • Can think of many applications. • Revision pattern is commonly used for: • Source control systems(data structured as file systems, with per-file merge operations) • Classic web applicationsLoad HTML form – edit locally – submit – server does merge • Modern web applicationsread REST object – javascript modifies locally – write REST object • We are currently focusing on this programming domain:Apps for Mobile + Cloud.

  29. Part IIa Motivation

  30. Why apps communicate Personal Publishing Sync and Backup Data Collection Blog Facebook Wall Website Music Video SkyDrive Surveys High Scores Collaboration Games Transactions Remote Control OneNote Shared Lists Shared Calendar Shared Spreadsheet Real-time Turn-based Store Auction Matchmaking Home Control Robotics Media Player

  31. Requirements • Persistence • Data is not deleted when we:quit the app, lose connection to server, take the battery out, crash due to bug, close the browser, replace the phone • Reliability • Process is not lost (resume at last stable point) • Data integrity is protected • Offline support • App continues to work without connection to cloud • Security • Control who can do what • Scalability • Support many users and/or large databases at low cost

  32. Requirements Our focus. Implies: - Need replicas on client - must support eventual consistency • Persistence • Data is not deleted when we:quit the app, lose connection to server, take the battery out, crash due to bug, close the browser, replace the phone • Reliability • Process is not lost (resume at last stable point) • Data integrity is protected • Offline support • App continues to work without connection to cloud • Security • Control who can do what • Scalability • Support many users and/or large databases at low cost

  33. grocery list cilantro sardines guava milk bread eggs

  34. Implementation Architecture? • Peer-to-Peer • Program runs on clients only, no server • Popular with researchers • Not all thatcommon in practice • Client-Server, or more recently Client-Service • Very common these days • Service typically hosted on virtualized infrastructure (cloud)-> makes “economy of scale” accessible to everybody

  35. Distributed Systems Basic Using Cloud Infrastructure Node Node Node Client Client Client Client Client Client Compute Compute Compute Storage Storage Storage Compute Compute Compute Storage Storage Storage Storage

  36. How to program this machine? Layer Client Not physically secure Unreliable Cannot detect failures Potentially many Client Client Client Client Client Client Cloud Compute Physically secure, not so many Not reliable: no persistent state Can detect failures somewhat Relatively Expensive Compute Compute Compute Cloud Storage Secure Reliable Can be very cheap Storage Storage Storage Storage

  37. Extensive Replication Cache Coherence? Consistency Model? App Compute Layer Compute Layer Compute Layer Messages GUI State Local State Compute Layer Binding Compute Layer Compute Layer Compute Layer save/restore save/restore LocalStorage Messages Messages App GUI State Local State Storage Backend Binding save/restore save/restore LocalStorage Replica StorageBackend sync

  38. grocery list

  39. App programmers should not have to think that much. All this stuff is all about the implementation, not about the problem domain. App Compute Layer Compute Layer Compute Layer Messages GUI State Local State Compute Layer Binding • Program runs on server, client is just a view • Example: Classic HTML approach • Client clicks link/button to submit, gets next page • Program runs on client, uses server as a resource • Client issues webrequests (e.g. REST) • Program runs on client and on server • Example: websockets are full-duplex • Client and server can send messages, causing event handlers to launch at other end • Peer-to-Peer • Program runs on clients only, no server • Rare for apps, as far as I know Compute Layer Compute Layer Compute Layer save/restore save/restore LocalStorage Messages Messages App GUI State Local State Storage Backend Binding save/restore save/restore LocalStorage Replica StorageBackend sync

  40. Abstractions, please. We propose: • - Revision Consistency • Cloud Types • Papers: • Eventually Consistent Transaction (ESOP 2011) • Cloud Types (ECOOP 2012)

  41. Closely Related Work • CRDTs (Conflict-Free Replicated Data Types) • [Shapiro, Preguica, Baquero, Zawirski] • Similar motivation and similar techniques • Bayou • user-defined conflict resolution (merge fcts.)

  42. Part IIb Revision Consistency

  43. Revision Consistency device 1 cloud device 2 • Client code: • Declare data types • read/update data • yield (=polite sync) • flush (=forced sync) • Under the hood: • Revision diagram rules

  44. Implicit Transactions device 1 cloud device 2 • At yieldRuntime has permission to send or receive updates. Call this frequently, e.g. automatically “on idle”. • In between yieldsRuntime is not allowed to send or receive updates • Implies: all client code executes in a (eventually consistent) transaction yield yield … … yield yield … … yield yield … … yield yield …

  45. On-Demand Stronger Consistency • flush primitive blocks until local state has reached main revision and result has come back to device • Sufficient to implement strong consistency • Flush blocks –times out if server connection is not available. flush(blocks) (continue)

  46. Client 1 Revision consistency Client 2 B yield A • Global state evolves as a Revision diagram • Main revision (center) in reliable cloud storage • Seamless offline support • Never blocks, except when client issues fence C yield yield D F flush yield G yield E

  47. Client 1 Revision consistency Client 2 B yield A • Global state evolves as a Revision diagram • Main revision (center) in reliable cloud storage • Seamless offline support • Never blocks, except when client issues fence C yield yield D F fence yield G yield E

  48. Nice things about Revision Consistency Strong guarantees: • Guarantees causal eventual consistency • Supports eventually consistent transactions • Supports on-demand stronger consistency Opportunities for efficient implementation • Naturally supports storage hierarchies • Consistent with full-duplex pull & push updates between service and client (e.g. websockets) • Can be combined with “log reduction” techniques

  49. Client 1 Related to Log-BasedImplementations Client 2 B A • Main Revision = Master Log • Suggested Implementation:- Main Log in Cloud Storage- Scalable read & write • It is possible to scale reading/writing of log • Log Reduction B C D F AD E G C F G

  50. 0 0 0 1 0 [FlushPush] Can build layered service [SyncPush] [YieldPush] 1 [SyncPush] 1 [YieldPull] [SyncPull] 1 2 [YieldPush] 2 [SyncPush] 3 3 [YieldPull] [SyncPull] [FlushPull] ComputeLayer Reliable Storage ComputeLayer 4 2 [SyncPull] Client Client 1 4

More Related