1 / 27

GPO System Engineer: Harry Mussman March 15, 2011 geni

GENI I&M Update: Architecture Overview and Current Status GENI Engineering Conference 10 San Juan, PR. GPO System Engineer: Harry Mussman March 15, 2011 www.geni.net. GENI I&M Update. Goals Architecture Overview Process Functional Services Measurement Data Flows/Transfers

kelly-collins
Download Presentation

GPO System Engineer: Harry Mussman March 15, 2011 geni

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GENI I&M Update:Architecture Overview and Current Status GENI Engineering Conference 10San Juan, PR GPO System Engineer: Harry MussmanMarch 15, 2011 www.geni.net

  2. GENI I&M Update • Goals • Architecture Overview • Process • Functional Services • Measurement Data Flows/Transfers • Measurement Traffic Flows • Principal Use Cases • Types of Services • Control Framework Implications • Measurement Data Objects and Descriptors • Controlling Access to Measurement Data Objects • Current Status • Current I&M Design and Prototyping Efforts • I&M Service Deployments • References: • I&M work in progress: http://groups.geni.net/geni/wiki/GeniInstMeas • I&M capabilities catalog: http://groups.geni.net/geni/wiki/GENIIandMCAPCAT • I&M architecture document: http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurementsArchitecture • Available I&M tools: http://groups.geni.net/geni/wiki/GeniInstMeas#AvailableGENIIMTools

  3. GENI I&M Goals • Provide broad data gathering, analysis and archival capability that is sufficient for: • Experiments • Maintaining the infrastructure • GENI operations • Sharing data with researchers outside of GENI • Support the experimenter: • Remove the burden on the experimenter to become a system and network measurement infrastructure expert, so that they can better focus on the science in the experiments • Scale to support: • Small-scale experiments • Long-running future-internet experiments • Operator monitoring their aggregate (or service) • GMOC monitoring all of GENI, 24x7 • Interoperable protocols, schemas and APIs: • Allows efficient development of services by reusing interface software modules • Allows different users to utilize same services

  4. GENI I&M Architecture Overview:Process Workshop 1 Workshop 2 Workshop 3 GEC10 GEC9 GEC7 GEC6 GEC8 Discuss 3 key topics Survey use cases, systems, issues Vision Review projects Next: Discuss 6 key topics Analyze current projects Identify 9 key topics Discuss 6 key topics Review projects and 4 key topics Identify functional services based on early view of arch Tutorials for experimenters Draft ver1.0 arch doc WG ends Soon: Issue ver1.0 arch doc Draft capabilities catalog WG starts

  5. GENI I&M Architecture Overview:Functional Services • GENI I&M services have been defined by function: • Measurement Point (MP) service • Measurement Collection (MC) service • Measurement Analysis and Presentation (MAP) service • Measurement Orchestration (MO) service • Measurement Information (MI) service • User Workspace (UW) service • Digital Object Archive (MDA) service

  6. GENI I&M Architecture Overview:Measurement Data (MD) Flows/Transfers • MD packet flows and file transfers are carried from one I&M service to the next • “Stitching” is done as I&M services are setup and configured • Authorization mechanisms are required to protect MD • Flows/transfers follow a wide range of protocols and schema, including both pull and push methods

  7. Measurement Data (MD) Flows/Transfers:Current Protocols and Schema Opt 3 Opt 1 • Current range of protocols and schema: • Option 1: Bytes of MD via SNMP • Option 2: File of MD via SCP, FTP or 3rd-party FTP • Option 3: XML-formatted MD via HTTP (e.g., perfSONAR) • Option 4: Tuples of MD via Custom OML Protocol • Option 5: Tuples of MD via IPFIX over SCTP • Option 6: NetCDF formatted files of data using LDM over TCP Opt 5 Opt 3 Opt 5 Opt 2 Opt 5 Opt 4 Opt 2 Opt 2

  8. GENI I&M Architecture Overview:Measurement Traffic Flows • Expect most measurement traffic to be carried with control traffic, via Internet and/or IP backbone • Many servers will have a Private (non-reachable) IP Address and proxies (or other mechanisms) are required to reach these servers.

  9. GENI I&M Architecture Overview:Principal Use Cases • Use Case 1: Experimenter gathering measurement data (MD) from their slice • Use Case 2: Operator (or Service Provider) gathering measurement data (MD) from GENI infrastructure • Use Case 3: Experimenter gathering measurement data (MD) from their slice and from GENI infrastructure • Use Case 4: Experimenter (or Operator) moving measurement data (MD) to an archive, with an option to share with other Researchers

  10. Use Case 1: Experimenter gathering MD from their slice • Each I&M service is setup as part of the Experimenter’s slice • GENI I&M project examples: • Instrumentation Tools (for use with protoGENI) • OMF/OML • LAMP (uses perfSONAR technology, for use with protoGENI) • OnTimeMeasure (for use with protoGENI or PlanetLab) • Includes: • Type 1 service: dedicated to a slice • Type 4 service: with portal to share MD

  11. Type 1 I&M Service: Dedicated to a slice • Each Type 1 I&M service: • Dedicated to a slice • Assembled by Slice Owner (e.g., Experimenter) using Control Framework mechanisms • Configured by Slice Owner (e.g., Experimenter) • Managed by Slice Owner (e.g., Experimenter) • I&M service management interface and API required on service

  12. Type 4 I&M Service: With portal to share MD • Each Type 4 I&M service: • Has web portal to observe and share MD • Web portal authorization mechanism required to admit only specified users; should reuse accepted certificate/credential arrangements • Example: Presentation service that is part of Instrumentation Tools • Example: Digital Object Archive (DOA) service prototype from CNRI • DOA service required to follow sharing policy within Descriptor associated with a Digital Object

  13. Use Case 2: Operator (or Service Provider) Gathering MD from GENI infrastructure • An Operator (or Service Provider) can gather MD from GENI infrastructure into their slice • For their own use • To share with other operators, e.g., share with GMOC • Includes: • Type 2 service: common service with multiple slivers • Type 3 service: common service with MD to multiple slices

  14. Type 2 I&M Service:Common service with multiple slivers • Each Type 2 I&M service: • Common service, assembled, configured and managed by a Service Provider (or Operator) in their sliver using Control Framework (CF) mechanisms • Service Provider runs an AggMgr interface and offers slivers (e.g., MP service slivers, or UW service slivers, as shown above) to other Operators (or Experimenters) via CF mechanisms • Each sliver is a resource with an rspec • Example: Measurement System for packet capture from Univ Wisconsin • Example: User Workspace (UW) service prototype from CNRI

  15. Type 3 I&M Service:Common service with MD to multiple slices • Each Type 3 I&M service: • Common service assembled, configured and managed by a Service Provider (or Operator) in their sliver using Control Framework (CF) mechanisms • Service Provider (or Operator) registers Descriptors of available MD Objects (e.g., MD flows, as shown above) with Measurement Information (MI) service • Example: perfSONAR MP and MI services • Operator (or Experimenter) finds and requests a flow from the I&M service • Authorization mechanism required to make flow available to selected operators/experimenters.

  16. Use Case 3: Experimenter gathering MD from their slice and from GENI infrastructure • An Experimenter can gather MD from their slice and from GENI infrastructure • Combines Use Case 1 with Use Case 2 • Includes Type 2 service: common service with multiple slivers • Policy required to make sliver available to selected experimenters • Includes Type 3 service: common service with MD to multiple slices • Access mechanism required to make flow available to selected experimenters • Includes Type 1 service: dedicated to a slice • Configured to stitch flow into experimenter’s I&M service

  17. Use Case 4: Experimenter (or Operator) moving MD to an archive, with an option to share with other Researchers • An Experimenter (or Operator) can move MD Objects (directories or files) to (and from) a Type 2 User Workspace (UW) service • Always includes associated MD Object Descriptor (“metadata”) • Identifiers and annotation in Descriptors allow Experimenter (or Operator) to manage thousands of files gathered during an experiment, or when monitoring infrastructure • The Experimenter (or Operator) can move selected MD Objects (directories or files) from UWservice to the Type 4 Digital Object Archive (DOA) service, to become persistent Digital Objects • Digital Object receives a persistent global identifier • Digital Object includes an associated Descriptor • Descriptor includes sharing policy, that DOA service is obligated to follow

  18. GENI I&M Architecture Overview:Functional Services • GENI I&M services have been defined by function: • Measurement Point (MP) service • Measurement Collection (MC) service • Measurement Analysis and Presentation (MAP) service • Measurement Orchestration (MO) service • Measurement Information (MI) service • User Workspace (UW) service • Digital Object Archive (MDA) service

  19. GENI I&M Architecture Overview:Types of Services • Type 1 Service: dedicated to a slice • Type 2 service: common service with multiple slivers • Type 3 service: common service with measurement data (MD) to multiple slices • Type 4 Service: with portal to share measurement data (MD)

  20. GENI I&M Architecture Overview:Control Framework (CF) Implications • I&M services are built by Experimenters/Operators/Service Providers using CF functions • I&M service is a resource • An I&M service (type 2) can offer slivers as resources via CF • I&M service must support an AggMgr interface • I&M sliver as a resource requires an rspec • Requires policy for selected experimenters and/or operators • I&M service has a standardized management interface/API linked to Measurement Orchestration (MO) service via CF mechanisms • For example, keys are installed that are used to sign messages • I&M service requires proxies (or other methods) to permit measurement traffic flows (data and management) to reach servers in private address space

  21. GENI I&M Architecture Overview:Measurement Data (MD) Objects • GENI MD Objects include: • Packet flows and streams • Directories and files • Persistent digital objects • Data structures accessed by querying a data base • Data structures accessed through a web portal GUI • Each MD Object has an associated MD Object Descriptor (MDOD) • Commonly called “metadata”

  22. Measurement Data Object Descriptor (MDOD) • Each MD Object Descriptor follows one common data model : • Elements and values for data model under study • Some will be Mandatory and some will be Optional • Information from data model must be translated into appropriate schema for use in different parts of GENI • Mandatory elements in a MD Object Descriptor are expected to include: • Identifier and Locator elements, for finding the MD Object. • Object Type and Measurement Data Schema elements, for interpreting the MD Object by programmatic means. • Other descriptive elements, e.g., Subject, to facilitate searching for MD Objects. • Annotation elements, added to provide “standardized lab notes”. • Owner/creator, previous holder(s), current holder • Policy elements, to specify how the MD Object can be shared or disposed. • The associated MD Object Descriptor is created at the same time an MD Object is created within an Experimenter’s or Operator’s slice, • The Experimenter or Operator that collects the MD “owns” the MD, and sets the policy for how it can be stored or disposed

  23. GENI I&M Architecture Overview: Controlling Access to Measurement Data (MD) Objects • MD Objects have associated Measurement Data Object Descriptors (MDODs, commonly known as “metadata”) • Each MDOD typically includes policy elements, to specify how the MD Object can be shared or disposed. • The I&M service holding the MD Object is obligated to follow the policy for sharing • Measurement data flows and transfers between Type 1 services within a slice must be “stitched” as slice is setup • An access control mechanisms is required to prevent others from getting the MD • A Type 3 service can register measurement data for others to discover and share MD via a standardized interface • An access control mechanism is required to admit selected GENI experimenters and operators • It should reuse accepted certificate/credential arrangements • A Type 4 service can share measurement data via a web portal • An access control mechanism is required to admit selected GENI experimenters and operators • It should reuse accepted certificate/credential arrangements • A Digital Object Archive (DOA) service is required to make digital object available to selected users within GENI, selected users outside of GENI, and/or all others • It must follow policy in MDOD associated with Digital Object

  24. GENI I&M Architecture Process:Current Status Workshop 1 Workshop 2 Workshop 3 • With strong support from GENI community, ver1.0 architecture nearing completion! • Next working meeting to discuss key topics: GEC10, Thursday, 1pm – 4pm • Requires CF-related functions, MD access control mechanisms, coordinated with other GENI efforts • Will guide current D&P projects, future Solicitation 3 projects and ongoing deployments GEC10 GEC9 GEC7 GEC6 GEC8 Discuss 3 key topics Survey use cases, systems, issues Vision Review projects Next: Discuss 6 key topics Analyze current projects Identify 9 key topics Discuss 6 key topics Review projects and 4 key topics Identify functional services based on early view of arch Tutorials for experimenters Draft ver1.0 arch doc WG ends Soon: Issue ver1.0 arch doc Draft capabilities catalog WG starts

  25. Current GENI I&M Design and Prototyping Efforts • I&M services for Experimenters, including: • Instrumentation Tools (for use with protoGENI) • OMF/OML • LAMP (uses perfSONAR technology, for use with protoGENI) • OnTimeMeasure (for use with protoGENI or PlanetLab) • I&M services for monitoring infrastructure, including: • Measurement System, for packet capture • Shadownet, for router monitoring • Scalable Sensing Service • Measurement Data Archive (MDA) service • First prototype by CNRI, including User Workspace (UW) service and Digital Object Archive (DOA) service • Management of an I&M service, from Measurement Orchestration (MO) service: • First prototype by IMF and ERM projects, following OMF approach • Measurement Data Objects, and Descriptors (“metadata”): • Defining common data model, then schemas • First prototype software by ? • Measurement traffic flows, proxies/mechanisms) to reach servers in private address space • First prototype of ssh access by ORCA project

  26. GENI I&M Service Deployments • Current I&M services for Experimenters, including: • Instrumentation Tools (for use with protoGENI) • OMF/OML • LAMP (uses perfSONAR technology, for use with protoGENI) • OnTimeMeasure (for use with protoGENI or PlanetLab) • Introduce I&M services for monitoring infrastructure, including: • perfSONAR components, with Measurement Information (MI) service, and first use of Measurement Data Object Descriptors (MDOD’s)? • Measurement System for packet capture? • Scalable Sensing Service? • Introduce Measurement Data Archive (MDA) service: • Include User Workspace (UW) service • Include Digital Object Archive (DOA) service • Who are first users? • Are there other uses, e.g., software downloads? • When ready to support Researchers?

  27. GENI I&M Update • Goals • Architecture Overview • Process • Functional Services • Measurement Data Flows/Transfers • Measurement Traffic Flows • Principal Use Cases • Types of Services • Control Framework Implications • Measurement Data Objects and Descriptors • Controlling Access to Measurement Data Objects • Current Status • Current I&M Design and Prototyping Efforts • I&M Service Deployments • References: • I&M work in progress: http://groups.geni.net/geni/wiki/GeniInstMeas • I&M capabilities catalog: http://groups.geni.net/geni/wiki/GENIIandMCAPCAT • I&M architecture document: http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurementsArchitecture • Available I&M tools: http://groups.geni.net/geni/wiki/GeniInstMeas#AvailableGENIIMTools

More Related