Grids: The Future of Computing

1. Grids: The Future of Computing Jacques P. Sauvé – UFCG 2004

2. Agenda • Problems • Solutions • The State of the Art • Challenges for the Future

3. Problems • Remember that business needs drive technology • However, technology itself can also change enough to upset the status quo • So let’s look at: • Business drivers that lead to grids • Technology drivers that lead to grids

4. “Business” needs in Science and Engineering Domains • Scientists and engineers can always use up as much computing power as you can throw at them! • They generate lots of data (petabytes) and always need cycles, storage, etc. • One computer, even a supercomputer, isn’t enough • New needs • Much more collaboration between remote sites • Lots of disperse data

5. Business Drivers in the Business Domain • E-Commerce and E-Business have led to large distributed systems • Need to interconnect with partners and cross administrative domains • Demands on the CIO • Improve availability • Increase business relevance • Reduce complexity • Support rapid change • Drive costs down • Grids will try to address items in bold

6. The Mission in Technological Terms • The aggregate effect is that qualities of service traditionally associated with mainframe host-centric computing are now essential to the effective conduct of e-business across distributed compute resources, inside as well as outside the enterprise. • For example, enterprises must provide consistent response times to customers, despite workloads with significant deviations between average and peak utilization

7. Why it’s difficult and expensive • Four main problems • How to handle scale and large demand variations while maintaining QoS • How to cross administrative domains while maintaining security • How to deal with high cost of operating computers • How to deal with the appearance of service providers (eUtilities)

8. Problem 1 Scale, Demand Variation and QoS • Should a business have resources to handle the peak (to maintain QoS)? • Peak demand may be 10 times normal demand • Systems have to handle tens of thousands of users • We would not accept a situation in which every home and business had to operate its own power plant, library, printing press and water reservoir. Why should we do so for computers?

9. Problem 2Crossing Admin Domains, Security • Very tough technical problem to solve • Solutions are labor-intensive and prone to error • Trustis inherently non-scalable because it cannot be automated!

10. Problem 3High Cost of Operating Computers • Total cost of ownership is always going up • We are now using very complex distributed systems • Responding to change leads to frequent reconfigurations with very high maintenance costs • Businesses are screaming for a way to reduce all this complexity • Outsourcing is a trend that highlights this

11. Problem 4e-Utilities • Specialized e-Utilities leverage economies of scale to drive costs down for certain services • How to factor them into a business’ IT architecture?

12. Solutions: Grids • What? • How? • Definitions

13. Solutions: What we want • Decoupling production and consumption • Enables specialized functions and common services to be turned into commodities • Demand-driven access to (decoupled) computational resources • Transparency

14. Computational Grid (Resources) What we want: A Picture • Looks like the Web?? • Inside the cloud are resources, not information

15. Solutions: How is this achieved? • Virtualization • Hide implementation behind an interface • User receives a virtual computer • Looks like a “utility grid” whence the name “computing grid” • Provisioning of computational resources (compute-power, storage, network bandwidth), on a per-need basis

16. Solutions: How is this achieved? • Service Orientation • If resources are not “here”, how do I get to them? • Access a service • You promise a behavior: natural way to virtualize • Since services are distributed, use Web Services to access middleware functions • With some modifications for grids, this was turned into Web Services Resource Framework (old OGSI)

17. Grids: A Definition • Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations • Not subject to centralized control • Based on standards • Delivers non-trivial QoS

18. State of The Art: Applications • Distributed Supercomputing • Distributed supercomputing applications use grids to aggregate substantial computational resources in order to tackle problems that cannot be solved on a single system • High-Throughput Computing (Scavenging grids) • Harness many otherwise idle resources to increase aggregate throughput

19. State of the Art: Applications • Data-Intensive Computing • Focus is on synthesizing new information from data that is maintained in geographically distributed repositories, digital libraries, and databases • On-Demand Computing • Use grid capabilities to meet short-term requirements for resources that can not be cost-effectively or conveniently located locally • Collaborative Computing • Enabling and enhancing human-to-human interactions

20. Commercial Grids • Closer to last 2 (on-demand, collaboration) but there are other differences • Business applications run continuously • Over long periods of time ... • Have more stringent requirements on reliability, security and accountability • Virtualizing IT assets • Phrases used (marketing!) • “utility computing,” “e-business on demand,” “planetary computing,” “autonomic computing,” “enterprise grids”

21. Infrastructure for Computational Grids • State of the Art: we are still working on building a robust infrastructure • Still trying to standardize basic pieces • Ex.: OGSI just changed to WSRF to make Web Services people happy • No planetary deployment yet • Largest grid: China Educational and Research Grid • 500 TeraBytes, 6 TeraFLOPS (30 MFLOPS/student)

22. State of the Art: Grid Components • 1. User view: a portal

23. State of the Art: Grid Components • 2. Security (GSI – Grid Security Infra)

24. State of the Art: Grid Components • 3. Broker Service (MDS = Monitoring and Discovery Service)

25. State of the Art: Grid Components • 4. Scheduler

26. State of the Art: Grid Components • 5. Data Management (GASS = Grid Access to Secondary Storage)

27. State of the Art: Grid Components • 6. Job Submission (GRAM = Grid Resource Allocation Manager)

28. State of the Art: HP • Utility Data Center (UDC) • Wire-once, programmatically reconfigurable, virtualized data center, with fined-grained allocation, security and control of every resource • UDC can be a node in a larger grid (virtual data center with many UDCs) • Go to Planetary scale: planetary scale computing • “The” Grid

29. State of the Art: IBM

30. Challenges for the Future • We’re at the very beginning • Lots of marketing and hype • “The” grid doesn’t exist • Standardization still going on • Where are the challenges ...?

31. Challenges • The Nature of Applications • Given a grid, what is possible in terms of applications in scientific and engineering domains and in other areas such as business, art, and entertainment?

32. Challenges • Programming models and tools • Need new thinking about this • System architecture • Many conflicting requirements • Infrastructure support • Resource management (allocation, coordination, …) • Security, defining and managing sharing relationships and other policies

33. Challenges • Grid management • Grid configuration and discovery • Fault handling • Accounting • Instrumentation • How to handle planetary scale?

34. Thank You. http://jacques.dsc.ufcg.edu.br/palestras