Orca internals 101

1. Orca internals 101 Jeff Chase Orcafest 5/28/09

2. Summary of Earlier Talks • Factor actors/roles along the right boundaries. • stakeholders, innovation, tussle • Open contracts with delegation • resource leases • Orca as control plane for GENI: • Aggregates are first-class (“authority”) • Slice controllers are first-class (“SM”) • Clearinghouse brokers enable policies under GSC direction

3. For more… • For more on all that, see the slides tacked onto the ends of the presentation.

4. Orca Internals 101 • Leasing core: “shirako” • Plugins and control flow • Actor concurrency model • Lease state machines • Resource representations

5. Slice Controller Actors: The Big Picture Broker request ticket delegate redeem Authority lease

6. Slice Controller Actors: The Big Picture Broker Integrate experiment control tools here (e.g., Gush and DieselNet tools by XMLRPC to generic slice controller) request Integrate substrate here with authority-side handler plugins ticket delegate redeem Authority lease These are inter-actor RPC calls made automatically: you should not have to mess with them.

7. Terminology • Slice controller == slice manager == service manager == guest controller • Blur the distinction between the “actor” and the controller module that it runs. • Broker == clearinghouse (or a service within a clearinghouse) • == “agent” (in SHARP, and in some code) • Authority == aggregate manager • Controls some portion of substrate for a site or domain under a Management Authority

8. Experiment Manager Slice Controller Slice Controllers • Separate application/environment demand management from resource arbitration Slice controller monitors the guest and obtains/renews leases to meet demand Site Authority • Aggregate/authorities monitor • resources, arbitrate access, • and perform placement of guest • requests onto resources Experiment control tools (e.g., Gush and DieselNet tools) e.g. with XMLRPC to generic slice controller.

9. ProtoGENI? • Possibility: export ProtoGENI XMLRPC from a generic slice controller • > > (One thing to consider is how close it is to protogeni, which is not that • > > complicated, something like:) • > From poster on protogeni.net. (We should be able to support it. ) • > > - GetCredential() from slice authority • > > - CreateSlice() goes to slice authority • > > - Register() slice with clearinghouse • > > - ListComponents() goes to CH and returns list of AMs and CMs • > > - DiscoverResources() to AM or CM returns rspecs • > > - RequestTicket goes straight to an AM • > > - RedeemTicket also goes to the AM • > > - StartSliver to the AM after redeem: "bring sliver to a running state"

10. Authority Slice Controller Brokers and Ticketing • Sites delegate control of resources to a broker • Intermediary/middleman • Factor allocation policy out of the site • Broker arbitrates resources under its control • Sites retain placement policy • “Federation" • Site autonomy • Coordinated provisioning Broker request ticket delegate redeem lease SHARP [SOSP 2003] w/ Vahdat, Schwab

11. Actor structure: symmetry • Actors are RPC clients and servers • Recoverable: commit state changes • Pluggable DB layer (e.g., mysql) • Common structures/classes in all actors • A set of slices each with a set of leases (ReservationSet) in various states. • Different “*Reservation*” classes with different state machine transitions. • Generic resource encapsulator (ResourceSet and IConcreteSet) • Common kernel: Shirako leasing core

12. Actors and containers • Actors run within containers • JVM, e.g., Tomcat • Per-container actor registry and keystore • Per-container mysql binding • Actor management interface • Useful for GMOC and portals • Not yet remoteable • Portal attaches to container • tabs for different actor “roles” • Dynamic loadable controllers and views (“Automat”)

13. “Automat” Portal

14. Shirako Kernel Snippet from “developer setup guide” on the web. The paths changed in RENCI code base: prefix with core/trunk

15. Shirako Kernel Events The actor kernel (“core”) maintains state and processes events: • Local initiate: start request from local actor • E.g., from portal command, or a policy • Incoming request from remote actor • Management API • E.g., from portal or GMOC • Timer tick • Other notifications come through tick or protocol APIs

16. Leasing Core instantiate guests monitoring state storage/recovery negotiate contract terms event handling lease groups Configure resources Policy Modules “Controllers” Resource Handlers and Drivers Pluggable Resources and Policies

17. Kernel control flow • All ops come through KernelWrapper and Kernel (.java) • Wrapper: validate request and access • Most operations pertain to a single lease state machine (FSM) • But many access global state, e.g., alloc resources from shared substrate. • Kernel: execute op with a global “core” lock • Nonblocking core, at least in principle

18. Kernel control flow • Acquire core lock • Invoke *Reservation* class to transition lease FSM • Release core lock • Commit new state to actor DB • Execute async tasks, e.g. “service*” methods to invoke plugins, handlers • Ticks probe for completions of pending async tasks.

19. Lease State Machine Broker policy selects resource types and sites, and sizes unit quantities. original lease term Broker Ticketed Extending Guest may continue to extend lease by mutual agreement. request ticket extend update ticket request ticket return ticket Reservation may change size (“flex”) on extend. Resources join guest application. Service Manager Active Ticketed Extending Active Closed Nascent Joining Ticketed Active Guest uses resources. form resource request request lease extend return lease update lease redeem ticket close handshake Site Authority Priming Extending Active Time Teardown/reclaim resources after lease expires, or on guest-initiated close. Site policy assigns concrete resources to match ticket. Initialize resources when lease begins (e.g., install nodes).

20. Handlers • Invocation upcalls through ResourceSet/ConcreteSet on relevant lease transitions. • Authority: setup/teardown • Slice controller: join/leave • Unlocked “async task” upcalls • Relevant property sets are available to these handler calls • For resource type, configuration, local/unit • Properties ignored by the core • ConcreteSet associated with ShirakoPlugin • e.g., COD manages “nodes” with IP addresses, invokes handlers in Ant

21. Drivers • Note that a handler runs within the actor • So how to run setup/teardown code on the component itself? • How to run join/leave code on the sliver? • Option 1: handler invokes management interfaces, e.g., XMLRPC, SNMP, ssh • Option 2: invoke custom driver in a NodeAgent with secure SOAP

22. Example: VM instantiation drivers handlers

23. TL1 Driver Framework • General TL1 (command line) framework • Substrate component command/response • What to “expect”? • XML file

24. Authority Guest Broker Orca: Actors and Protocols formulate requests redeem tickets [1] [5] core calendar [2] [6] ticket extendTicket redeem extendLease updateTicket updateLease core core [4] [8] inventory resource pools assign redeem ticket allocate extend extend [3] [7]

25. Site Authority Broker Policy Plugin Points plug-in broker policies for resource selection and provisioning Broker service interface Negotiation between policy plugins over allocation and configuration Properties used to guide negotiation Guest application Host site (resource pool) Service Manager application resource request policy leasing API leasing service interface assignment policy join/leave handler for service lease event interface lease status notify setup/ teardown handlers for resources

26. Site Authority Broker Property Lists Examples: FCFS,priority,economic Broker service interface Request properties Resource Properties elastic,deferrable machine.memory machine.clockspeed Guest application Host site (resource pool) Service Manager application resource request policy leasing API leasing service interface assignment policy Configuration properties image.id,public.key join/leave handler for service lease event interface lease status notify setup/ teardown handlers for resources Unit properties host.ip,host.key

27. Messaging Model • Proxies maintained in actor registry • Asynchronous RPC • Upcall to kernel for incoming ops • Downcall from lease FSM for outgoing • Local, SOAP w/ WS-Security, etc. • WSDL protocols, but incomplete • Integration (e.g., XMLRPC) • Experiment manager calls into slice controller module • Substrate ops through authority-side handler

28. The end, for now • Presentation trails off…. • Follows are other slides from previous presentations dealing more with the concepts and rationale of Orca, and its use for GENI.

29. Embedding NSF GENI Initiative Experiments (Guests occupying slices) Sliverable GENI Substrate (Contributing domains/Aggregates) Observatory Wind tunnel Petri dish

30. Optical Switch Fiber ID ρ Aggregate Mgmt Authority Switch Port Channel Band Dreaming of GENI NSF GENI clearinghouse 3. MA (because it has sufficient credentials) registers name, GID, URIs and some descriptive info. http://groups.geni.net/ GID Component Registry 4. MA delegates rights to NSF GENI so that NSF GENI users can create slices. • Notes: • Identity and authorization are decoupled in this architecture. GIDs are used for identification only. Credentials are used for authorization. I.e., the GID says only who the component is and nothing about what it can do or who can access it. • Assuming aggregate MA already has credentials permitting access to component registry 2. CM sends GID to MA; out of band methods are used to validate MA is willing to vouch for component. CM delegates MA the ability to create slices. Usage Policy Engine 1. CM self-generates GID: public and private keys Aaron Falk, GPO BBN

31. Slivers and Slices Aaron Falk, GPO BBN

32. GENI as a Programmable Substrate • Diverse and evolving collection of substrate components. • Different owners, capabilities, and interfaces • A programmable substrate is an essential platform for R/D in network architecture at higher layers. • Secure and accountable routing plane • Authenticated traffic control (e.g., free of DOS and spam) • Mobile social networking w/ “volunteer” resources • Utility networking • Deep introspection and adaptivity • Virtual tunnels and bandwidth-provisioned paths

33. Some Observations • The Classic Internet is “just an overlay”. • GENI is underlay architecture (“underware”). • Incorporate edge resources: “cloud computing” + sliverable network • Multiple domains (MAD): not a “Grid”, but something like dynamic peering contracts • Decouple services from substrate; manage the substrate; let the services manage themselves. • Requires predictable (or at least “discoverable”) allocations for reproducibility • QoS at the bottom or not at all?

34. Breakable Experimental Network (BEN) • BEN is an experimental fiber facility • Supports experimentation at metro scale • Distributed applications researchers • Networking researchers • Enabling disruptive technologies • Not a production network • Shared by the researchers at the three Triangle Universities • Coarse-grained time sharing is the primary mode for usage • Assumes some experiments must be granted exclusive access to the infrastructure

35. Open Resource Control Architecture (Orca) Cloud Apps Services Other Guests Middleware Resource Control Plane Node Node VM VM VM VM Hardware Hardware Resource Control Plane • Contract model for resource peering/sharing/management • Programmatic interfaces and protocols • Automated lease-based allocation and assignment • Share substrate among dynamic “guest” environments • http://www.cs.duke.edu/nicl/

36. → 16 rc=(4,4) c → rb=(4,8) → b ra=(8,4) a The GENI Control Plane • Programmable substrate elements • Dynamic end-to-end sliver allocation + control • Delegation of authority etc. • Instrumentation (feedback) • Resource representation and exchange • Defining the capabilities of slivers • “network virtual resource” • Foundation for discovery • Of resources, paths, topology bandwidth shares CPU shares

37. Define: Control Plane GGF+GLIF: "Infrastructure and distributed intelligence that controls the establishment and maintenance of connections in the network, including protocols and mechanisms to disseminate this information; and algorithms for automatic delivery and on-demand provisioning of an optimal path between end points.” s/connections/slices/ s/optimal path/embedded slices provisioning += and programmed instantiation

38. Key Questions • Who are the entities (actors)? • What are their roles and powers? • Whom do they represent? • Who says what to whom? • What innovation is possible within each entity, or across entities? Control plane defines “the set of entities that interact to establish, maintain, and release resources and provide…[connection,slice] control functions”.

39. Design Tensions • Governance vs. freedom • Coordination vs. autonomy • Diversity vs. coherence • Assurance vs. robustness • Predictability vs. efficiency • Quick vs. right • Inclusion vs. entanglement • Etc. etc. …

40. Design Tensions • What is standardized vs. what is open to innovation? • How can GENI be open to innovation in components/management/control? • We want it to last a long time. • Innovation is what GENI is for. • Standardization vs. innovation • Lingua Franca vs. Tower of Babel

41. Who Are the Actors? • Principle #1: Entities (actors) in the architecture represent the primary stakeholders. • Resource owners/providers (site or domain) • Slice owners/controllers (guests) • The facility itself, or resource scheduling services acting on its behalf. Others (e.g., institutions) are primarily endorsing entities in the trust chains.

42. Network Service Etc. Cloud Service Brokering intermediaries (ClearingHouse) Control Plane • Plug guests, resources, and management policies into the “cloud”. Resources Infrastructure providers

43. Contracts • Principle #2: provide pathways for contracts among actors. • Accountability [SHARP, SOSP 2003] • Be open with respect to what promises an actor is permitted to make. • Open innovation for contract languages and tools • Yes, need at least one LCD • Rspec > HTML 1.0 • Lingua Franca vs. Tower of Babel • Resource contracts are easier than service/interface contracts.

44. Rules for Resource Contracts • Don’t make promises you can’t keep…but don’t hide power. [Lampson] • There are no guarantees, ever. • Have a backup plan for what happens if “assurances” are not kept. • Provide sufficient power to represent what promises the actor is explicitly NOT making. • E.g., temporary donation of resources • Best effort, probabilistic overbooking, etc. • Incorporate time: start/expiration time • Resource contracts are leases (or tickets).

45. request Guest grant Provider Site Leases • Foundational abstraction: resource leases • Contract between provider (site) and guest • Bind a set of resource units from a site to a guest • Specified term (time interval) • Automatic extends (“meter feeding”) • Various attributes <lease> <issuer> Site’s public key </issuer> <signed_part> <holder> Guest’s public key </holder> <rset> resource description </rset> <start_time> … </start_time> <end_time> … </end_time> <sn> unique ID at Site </sn> </signed_part> <signature> Site’s signature </signature> </lease>

46. Network Description Language? <ndl:Interfacerdf:about="#tdm3.amsterdam1.netherlight.net:501/3"> <ndl:name>tdm3.amsterdam1.netherlight.net:501/3</ndl:name> <ndl:connectedTo rdf:resource="http://networks.internet2.edu/manlan/manlan.rdf#manlan:if1"/> <ndl:capacity rdf:datatype="http://www.w3.org/2001/XMLSchema#float">1.244E+9</ndl:capacity> </ndl:Interface> <ndl:Interfacerdf:about="http://networks.internet2.edu/manlan/manlan.rdf#manlan:if1"> <rdfs:seeAlsordf:resource="http://networks.internet2.edu/manlan/manlan.rdf"/> <?xml version="1.0" encoding="UTF-8"?> <rdf:RDFxmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#” xmlns:ndl="http://www.science.uva.nl/research/sne/ndl#” xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#"> <!-- Description of Netherlight --> <ndl:Locationrdf:about="#Amsterdam1.netherlight.net"> <ndl:name>Netherlight Optical Exchange</ndl:name> <geo:lat>52.3561</geo:lat> <geo:long>4.9527</geo:long> </ndl:Location> <!-- TDM3.amsterdam1.netherlight.net --> <ndl:Devicerdf:about="#tdm3.amsterdam1.netherlight.net"> <ndl:name>tdm3.amsterdam1.netherlight.net</ndl:name> <ndl:locatedAtrdf:resource="#Amsterdam1.netherlight.net"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:501/1"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:501/2"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:501/3"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:501/4"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:502/1"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:502/2"/> <ndl:hasInterfacerdf:resource="#tdm3.amsterdam1.netherlight.net:502/3"/> >

47. Delegation • Principle #3: Contracts enable delegation of powers. • Delegation is voluntary and provisional. • It is a building block for creating useful concentrations of power. • Creates a potential for governance • Calendar scheduling, reservation • Double-edged sword? • Facility can Just Say No

48. Aggregation • Principle #4: aggregate the resources for a site or domain. • Primary interface is domain/site authority • Abstraction/innovation boundary • Keep components simple • Placement/configuration flexibility for owner • Mask unscheduled outages by substitution • Leverage investment in technologies for site/domain management

49. BEN fiberplant • Combination of NCNI fiber and campus fiber • Possible fiber topologies:

50. Infinera DTN • PIC-bases solution • 100Gbps DLM (digital line module) • Circuits provisioned at 2.5G granularity • Automatic optical layer signal management (gain control etc.) • GMPLS-based control plane • Optical express • All-optical node bypass