GPO System Engineer: Harry Mussman July 22, 2010 geni

1. GENI Instrumentation and Measurement WG:2nd I&M Workshop and Priority Architecture Topics GENI Engineering Conference 8UC San Diego GPO System Engineer: Harry MussmanJuly 22, 2010 www.geni.net

2. 2nd I&M Workshop • Objective: • Gather contributors from key I&M prototyping projects that have already implemented pieces of I&M functionality in a manner consistent with GENI goals. • Define priority pieces of the I&M architecture by consensus • Assemble teams to complete the discussion and documentation • Draft a roadmap for implementations during Spirals 2 and 3. • See http://groups.geni.net/geni/wiki/2ndInstMeasWork#References • Initial set of priority topics: • GENI I&M use cases • GENI I&M services • GENI measurement plane • Interfaces, protocols and schema for measurement data in GENI

3. (continued) • Approach: • Each priority topic was discussed in a structured manner • The organizers outlined an approach or solution, and representatives from the key projects discussed it • The group identified where they had a consensus, and where there were issues that needed further discussion • One new topic was identified: how to discover, specify, authorize and assign all of the various types of resources required for I&M capabilities • The interfaces topic was split into four topics, divided upon function • Finally, teams were assembled to discuss each priority topic, come to a consensus, and write a revised section for the architecture document • Initial set of priority topics: • GENI I&M use cases • GENI I&M services • GENI measurement plane • Interfaces, protocols and schema for measurement data in GENI

4. (continued) • Organizing Committee: • Paul Barford - University of Wisconsin – Madison • Bruce Maggs – Duke University and Akamai • Harry Mussman – BBN/GPO • Vic Thomas - BBN/GPO • Evan Zhang – BBN/GPO • Key projects who sent representatives: • OML (ORBIT Measurement Library) OMF (ORBIT Management Framework) • Instrumentation Tools • perfSONAR • Scalable Sensing Service (S3) • OnTimeMeasure for network measurements • GENI Meta-Operations Center and NetKArma • Virtual Machine Introspection (VMI) • Experiment Management Service – Digital Object Registry

5. (continued) • Final set of priority topics: • Topic 1: GENI I&M Use Cases • Topic 2: GENI I&M Services • Topic 3: GENI I&M Resources • Topic 4: GENI I&M Measurement Plane and Interfaces • Topic 5: GENI I&M Interfaces and Messages/Flows/APIs : Manage Services • Topic 6: GENI I&M Interfaces and Messages/Flows/APIs : Data Flows and Data File Transfers • Topic 7: GENI I&M Interfaces and Messages/Flows/APIs : Service Registration and Discovery • Topic 8: GENI I&M Interfaces and Messages/Flows/APIs : GUIs • Topic 9: GENI Measurement Data Schema

6. Topic 1: GENI I&M Use Cases • Team members: • Paul Barford - University of Wisconsin – Madison • Jim Griffioen - Univ Kentucky • Prasad Calyam* - Ohio Supercomputing Ctr • Camilo Viecco - Indiana Univ • Brian Hay – Univ Alaska *agreed to organize first writing and discussion • Identify all user groups, and provide basic use cases for each user group: • Experiment researchers • Experiment (opt-in) users • Central (i.e., GMOC) operators • Cluster and aggregate providers and operators • Archive service providers and operators • Researchers that use archived measurement data

7. Topic 2: GENI I&M Services • Team members: • Harry Mussman* and Evan Zhang – BBN/GPO • Giridhar Manepalli – CNRI • Chris Small and Beth Plale - Indiana Univ *agreed to organize first writing and discussion • Summarize current view of services • Measurement Orchestration (MO) Service • Measurement Point (MP) Service • Measurement Information (MI) Service • Measurement Collection (MC) Service • Measurement Analysis and Presentation (MAP) Service • Measurement Data Archive (MDA) Service • Need basic definition of a Measurement Data Archive (MDA) service

8. (continued) • Have identified three different types of services: • Type 1: Dedicated service platform for customized Information. (Completely dedicated to an experiment) • Type 2: Common service platform with dedicated slivers for customized Information. (Common portion, plus parts associated with different experiments) • Type 3: Common service for common or customized information. (Common service, with data provided to multiple experiments)

9. GENI I&M Architecture:Services and Interfaces

10. Topic 3: GENI I&M Resources • Team members: • Vic Thomas - BBN/GPO • Jim Griffioen* - Univ Kentucky • Martin Swany - Univ Delaware • Camilo Viecco - Indiana Univ • Brian Hay – Univ Alaska • Giridhar Manepalli - CNRI *agreed to organize first writing and discussion • How are each of these resources discovered, specified, authorized and assigned: • Hosts, VMs, etc. • Network connectivity • Software, e.g., I&M software that can be included in an experiment • I&M services • I&M data flows and file transfers • I&M data files stored in archives

11. (continued) • Options: • Always by mechanisms provided by the CF? • With CF plus additional mechanisms? • Consider example of LS in perfSONAR • Consider example of data file stored in archive, owned by an experimenter • Need to understand interop with CF for each option • Does CF setup secondary authorization mechanisms in some cases? If so, how? • For each item, consider • Unique name? • Unique and persistent identifier? • Ownership • What sort of policies the owner may want to apply • How to create • How to name • How to register and discover • How to authorize and assign

12. (continued) • Chart:

13. Topic 4: GENI I&M Measurement Plane and Interfaces • Team members: • Harry Mussman* – BBN/GPO • Ezra Kissel – Univ Delaware • Chris Small - Indiana Univ *agreed to organize first writing and discussion • Assume: Use IP backbone network (with Internet access) for most measurement traffic (like control traffic) • Consider: Use Layer 2 (VLAN) connections only when necessary • Issues with measurement traffic: • Which protocols are active? minimize number to simplify? • Need access to resources in aggregates, even when resources are in private address space, via proxies (i.e., gateways) • Need to provide authentication and authorization, without and with proxies • Need to be able to reserve bandwidth (provide QoS) to protect measurement traffic from other traffic, and vice-versa • Which of these requires the use of Layer 2 (VLAN) connections?

14. GENI Measurement Traffic Flows

15. GENI I&M Interfaces and Messages/Flows/APIs • 1) Discover Resources and Assign Slivers • EC srvc uses CF to discover resources, and then assign slivers to slice/researcher for I&M srvc’s • 2) Configure and Program Slivers • EC srvc uses CF and/or ssh to load std or customized software images for I&M srvc’s • Note: 1) and 2) are not specific to I&M services • 3) Manage Services • EC srvc and MO srvc use CF and/or https to check status of I&M srvcs, receive event notifications, and execute functions such as start, stop, reset, reboot, and checkpoint • 4) Measurement Data Flows and Data File Transfers • Measurement data flows and data file transfers between I&M srvcs. Options: Pull, Push, Pub/Sub. • 5) Register I&M Service • Operator configures I&M srvc to register with Lookup Srvc, advertising name, location, and available metadata • 6) Discover I&M Service and Establish Meas Data Flow • ECS or I&M srvc discovers I&M srvc advertisement, and establishes data flow • 7) Conduct and Observe Experiment • Researcher uses browser to interact with and observe services via web portals (GUIs)

16. GENI I&M Architecture:Services and Interfaces

17. Topic 5: GENI I&M Interfaces and Messages/Flows/APIs: Manage Services • Team members: • Vic Thomas - BBN/GPO • Ivan Seskar – Rutgers WINLAB • Max Ott – NICTA • Sonia Fahmy* – Purdue • Giridhar Manepalli – CNRI *agreed to organize first discussion and writing • Define an approach based on OMF/OML and S3 • Consider: • HTTP(S) • REST vs SOAP • Authorization by credentials or ? If credentials, how to revoke? • Pass XML fragments • Define basic API

18. Topic 6: GENI I&M Interfaces and Messages/Flows/APIs: Data Flows and Data File Transfers • Team members: • Harry Mussman* – BBN/GPO • Ivan Seskar – Rutgers WINLAB • Max Ott – NICTA • Ezra Kissel – Univ Delaware • Prasad Calyam - Ohio Supercomputing Ctr • Michael Zink - UMass Amherst *agreed to organize first writing and discussion • Consider data flows and data file transfers between all services • Fundamental to I&M, and not yet defined in GENI • Linked to measurement data schema, which is primary

19. (continued) • Consensus to allow a wide range of options: • Both data flows and data files transfers • Types: Pull, Push, Pub/Sub • Protocols: SNMP, SCP, FTP, gridFTP, HTTP, XMPP, TCP, SCTP • Consider: Naming, Discovery, Connectivity , Authentication and authorization mechanisms • Goal is to define an initial set: • Minimum set required for GENI • Mapped to current projects • Permit later additions

20. Topic 7: GENI I&M Interfaces and Messages/Flows/APIs: Registration and Discovery of Serviceswith Available Measurement Data • Team members: • Jason Zurawski* – Internet2 (yes) • Prasad Calyam - Ohio Supercomputing Ctr (yes) *agreed to organize first writing and discussion • Consider approach used in perfSONAR • Summarize for: • Services with available data flows • Also sources of file transfers? • Also GUIs?

21. Topic 8: GENI I&M Interfaces and Messages/Flows/APIs: GUIs • Team members: • Jeremy Reed and Zongming Fei - Univ Kentucky (yes) • Guilherme Fernandes* – Univ Delaware (yes) • Puneet Sharma - HP Labs (yes) • *agreed to organize team • Consider all types of GUIs: • Control experiments • Display I&M results • Report status • View archive service • Define overall goals for GENI GUIs • Consider portal, for access to multiple GUIs • Consider need for authentication and authorization

22. Topic 9 GENI Measurement Data Schema • Team members: • Bruce Maggs – Duke University and Akamai • Max Ott – NICTA • Ivan Seskar – Rutgers WINLAB • Martin Swany* - Univ Delaware • Camilo Viecco - Indiana Univ • Michael Zink - UMass Amherst • Jim French – CNRI *agreed to organize first discussion and writing • Consider: • Measurement data schema • Metadata schema • Metadata contents

23. (continued) • Goal is to define an initial set: • Minimum set required for GENI • Mapped to current projects • Permit later additions • Consider measurement data schema and/or metadata schema from: • perfSONAR • GMOC-provided • Current OML approach • Proposed OML approach using IPFIX • NetCDF (as used by DI Cloud) • Provide overall template for GENI metadata: • Which items in GENI metadata template are required? • Invariant?

24. GENI I&M Architecture Priority Topics:Current Status

25. GENI I&M Architecture Document:Current Status • Status: • By GEC7 : v0.1 DRAFT completed, by GPO; see http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurementsArchitecture • By GEC8: v0.5 DRAFT completed, by GPO, with contributions from WG; see • Plan: • By GEC9: v1.0 draft, reviewed by WG

26. GENI I&M Architecture Priority Topics and Document: Next Steps • Complete group discussion and first writing for each priority topic: • When? • Add members to each group? • Review each priority topic with WG, and rewrite: • How? • When? • Assemble and review revised Architecture Document as a whole • How? • Before GEC9 • Draft a roadmap for implementations during Spiral 3.