1 / 13

Cloudspace: virtual environments in the VO

T HE US N ATIONAL V IRTUAL O BSERVATORY. Cloudspace: virtual environments in the VO. Matthew J. Graham (Caltech) Roy Williams (Caltech). Scenario . For the good of the community, I expose my super data mining algorithm as a service It runs on the local cluster in the basement.

james-conner
Download Presentation

Cloudspace: virtual environments in the VO

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. THE US NATIONAL VIRTUAL OBSERVATORY Cloudspace: virtual environments in the VO Matthew J. Graham (Caltech) Roy Williams (Caltech) ADASS XVII : London, UK

  2. Scenario • For the good of the community, I expose my super data mining algorithm as a service • It runs on the local cluster in the basement Performance efficiency Time ADASS XVII : London, UK

  3. The problem: utility computing • Dynamic allocation of computing resource to meet demand: • more users • more data • more jobs • Traditionally throwing more hardware at a problem requires prior software porting: • complexity of applications • consistency of the environment • Cannot enforce Quality of Service ADASS XVII : London, UK

  4. The solution: virtualization • Virtual workspaces • An abstraction of an execution environment that can be made dynamically available to authorized clients by using well-defined protocols • resource quota (e.g. CPU, memory share) • software configuration (e.g. O/S, provided services). • Implement on Virtual Machines (VMs) • abstraction of a physical host machine • hypervisor intercepts and emulates instructions from VMs and allows management of VMs • VMWare, Xen, etc. ADASS XVII : London, UK

  5. Existing Technologies • Amazon EC2 - “computing in the cloud” • Use Xen VMs • Images, Instances and Security • FlexiScale from XCalibre/Virtual Iron - “cooler than EC2” • Google/Microsoft? • Problem becomes availability of the right type of hypervisor ADASS XVII : London, UK

  6. Metadata instance Create() Authorize & instantiate Workspace Factory Service Deployment request Inspect & manage Workspace Resource Instance Workspace Service Notify Contains GLOBUS Deployment RP Globus Workspace • WSRF • Back-end implementation on UChicago Teraport cluster and EC2 ADASS XVII : London, UK

  7. Cloudspace • Based on VOSpace • IVOA standard interface for distributed data storage • Lightweight abstraction layer (Façade pattern) • Sits on top of proven solutions such as http and SRB • Resource oriented • Data and services are both first-class entities • Universal Worker Service interface to manage resources ADASS XVII : London, UK

  8. Resource-oriented computing • A resource is an abstract set of information • Each resource may be identified by one or more logical identifiers • A logical identifier may be resolved within an information-context to a physical resource representation • Computation is the reification of a resource to a physical resource representation • Resource representations are immutable • Transreption is the isomorphic lossless transformation of one resource representation to another • Computational results are resources and are identified within an address space ADASS XVII : London, UK

  9. Logical identifiers • URI: • [scheme-name]://[scheme-specific-address] • Cloudspace scheme is csp - why not ivo://? • URI mapping (links) • Different URI schemes return different representations (http, ftp)cf. HTTP MIME type and headers • Data objects • use VOSpace syntax:vos://nvo.caltech!vospace/myTable1 • Services • use regular IVOA identifiers to address image:ivo://nvo.caltech/service/paramSweeper • csp to address instantiations:csp://nvo.caltech/service/paramSweeper/1234 ADASS XVII : London, UK

  10. Resource representations • Data objects use VOSpace <node> • Arbitrary metadata (properties, tags) • Transreption formats (views) • Service image uses VOResource mechanism • Extension schemata to handle virtualization metadata (Globus equivalent) • Service instantiation uses UWS representation ADASS XVII : London, UK

  11. Universal Worker Service • URIs: • /(jobs): Job List • /(jobs)/(job-id) : Phase, Termination Time, Quote, Results List • /(jobs)/(job-id)/phase: Phase • /(jobs)/(job-id)/termination: Termination Time • /(job)/(job-id)/quote: Quote • /(job)/(job-id)/results: Results List ADASS XVII : London, UK

  12. Computing in the cloud • Can use standard endpoint of service • http://some.service.endpoint • Define computation in terms of result URI • comp:// scheme refers to computations:comp://nvo.caltech/service/paramSweeper/1234+data@vos://nvo!caltech/vospace/myTable1+params@vos://nvo!caltech/vospace/myParam1 • Map to vos:// data object:vos://nvo!caltech/vospace/myResult1 • Memoization: data caching a function call ADASS XVII : London, UK

  13. Not just blue-sky thinking • Proof-of-concept system • NetKernel • Ruby script to interface with EC2 as backend • Other packaged software • War files ADASS XVII : London, UK

More Related