NeSC, University of Edinburgh

1. Grids, Clouds and the Community.Cloud Technology and the NGSSteve ThornEdinburgh UniversityMatteo Turilli, Oxford UniversityPresented by David Fergusson

2. NeSC, University of Edinburgh • National eScience Centre • Support and develop IT supported advanced research • University of Edinburgh • College of Science and Engineering • Coordination and cooperation to make best use of leading research across research.

3. NGS • JISC supported national service to: • Provide a framework for institutions to cooperate and share services. • Help promote access to JISC services and synthesis for researchers.

4. Views of the Clouds • Infrastructure as a Service • Platform as a Service • Service as a Service • Software as a Service • Private Clouds • Public Clouds • Hybrid Clouds • Cloud vs Web 2.0 (?)

5. Views of the Clouds • Infrastructure as a Service • Platform as a Service √ • Service as a Service • Software as a Service • Private Clouds • Public Clouds • Hybrid Clouds • Cloud vs Web 2.0 (?)

6. Cloud computing for NGS/NeSC (incomplete list) • Dynamic provision of services • Individualisation • Peaky demand • Extending virtualisation • Packaging • Service configuration • Provenance for research • New publishing models

7. Head in the clouds? • Dynamic (service) provisioning • How is it applicable to the NGS? • Training • Rapidly deploy NGS services for training • Isolate training from production • Other • Specialised research environments • Rapid deployment • Identify use cases and gather requirements

8. NGS 3 EWP2 • “NGS Agile Deployment Environments” • EPSRC funded, 2 years • People • Matteo Turilli (OeRC, Oxford) [0.75 FTE] • Steve Thorn (NeSC, Edinburgh) [0.5 FTE] • David Fergussion (NeSC, Edinburgh) [WP Leader]

9. Overview • Agile service deployment • Virtualization vs. Cloud? • Use cases and requirements gathering • Training • Identify other (scientific) communities • Create images • NGS Services. Which ones?

10. Overview (cont.)‏ • Realistic usage • Training event on virtualized infrastructure • Hosting infrastructure? • Amazon EC2 compatible • De facto standard currently, with open source implementation • Ease of deployment • Eucalyptus, Nimbus and others • Hardware • Edinburgh: 8 cores ⇒ 16+ dual cores => 64 cores • Oxford: 64 cores (older)‏, new cluster => 64 cores

11. Eucalyptus • “Elastic Utility Computing Architecture Linking Your Programs To Useful Systems” • Open source and Commercial • Amazon Web Services API compatible • EC2, storage - S3, Elastic Block Store (EBS)‏ • Easy to install • Xen and KVM hypervisors • Commercial version supports others (inc. VMWare)‏

12. In the past • We have worked with Xen in the past to have Live CDs • Virtualisation • Works, but • Issues with security setups • networking

13. Eucalyptus networking challenges • Eucalyptus 'Managed' networking • Different networking modes • 'Managed' is the most flexible and feature rich – more complex • Allows elastic IP pool and image isolation. • VMs have private and public IPs • Introspection issues • Elastic IP • User assignable (may be somewhat different from Amazon)‏ • X509 Service certificates (NGS Host)‏ • Switch configurations

14. cont..... • Security Groups (EC2) • Implemented in Eucalyptus • isolate VMs • VM public traffic routed through Cluster controller • Instance doesn't have knowledge of its public IP • Bit like a NAT • Implications for GSI: $GLOBUS_HOSTNAME

15. Eucalyptus architecture • Storage controller (Walrus)‏ • implements Amazon’s S3 interface • Cloud controller • Entry point • Gathers information • Cluster controller • Schedules VM execution • Manages virtual network • Node controller • Controls VM execution • (Xen running on node)‏

16. cont..... • Security Groups (EC2) • Implemented in Eucalyptus • isolate VMs • VM public traffic routed through Cluster controller • Instance doesn't have knowledge of its public IP • Bit like a NAT • Implications for GSI: $GLOBUS_HOSTNAME

17. Clouds vs Virtualisation • Similar security and networking issues in Clouds and Virtualised instances • Virtualisation – virtualise instance • Clouds – virtualise the network (and other things) too • All arise from the requirements for rapid, automated, dynamic, reliable, reproducible, robust, provisioning

18. Progress • Started with Eucalypus 1.4.2 • Eucalyptus 1.6.2 deployed at Oxford and Edinburgh • Existing Images: • GSI-OpenSSH server • 'Single node cluster': torque/maui + Globus GRAM & GridFTP • Next step – some real world testing of phase1 images. • Image snapshots – not straightforward

19. Further work • Re-evaluate hosting infrastructure • Develop more images • Distributed torque/maui cluster + GRAM & GridFTP • Condor & GRAM? • 'Core site' • Training event in near future • Identify pilot community & gather requirements • Deploy fledgling cloud infrastructure

20. Further work 2 • Can snap shotting in Clouds side step packaging issues (configuration) in middleware? • By automating the copying and re-deployment of successful server installs. • Not just having a machine but a set of services which can be copied and deployed directly

21. Statement 1 • NRENs are still able to provide services that are generally better or more economic than those from commercial services providers. • Commercial offerings are currently shaped to particular modes of usage (due to their “by-product” heritage. This may limit utility for some users. EBI have reported issues due to virtual network sharing & unpredictability. • Data out costs are high • Commercial providers can offer “infinite” resource rapidly – not easy for NRENs • Should NRENs own large computing centres? • Payment models in institutions & research – FEC in the UK

22. Statement 2 • NRENs generally operate as a network for a closed group of users who have advanced requirements to support their research and education users. • There are certainly major issues with current commercial provision in regards to data confidentiality, isolation of users, service/data guarantees, internal/external access. • Recent data losses from commercial services • Commercial providers are almost certainly at an advantage in providing simple generic computing • NGS/UoE sees need to provide more complex research services.

23. Statement 3 • The NRENs do not compete with commercial ISPs, but offer a different level of service in parallel with them. • This is necessarily so. NRENs and other academic providers should find “blue water” for services. • Trust networks and security for public clouds ? • Service discovery ? • Data services ? • Accounting, monitoring ? • ............. > • Commercial providers will have no interest in some research services, cf. current research computing software provision.