Top Ten Grid Questions

1. Top Ten Grid Questions Written by: Jennifer M. Schopf Bill Nitzberg Presented by: Allen Lee

2. Also presenting… • “10 Things We Hate About the Grid” • Also by Jennifer M. Schopf • Powerpoint Presentation • http://www.mcs.anl.gov/~jms/Pubs/TopTenTalk.ppt

3. Topics Covered • How is the Grid different? • Why isn’t the Grid a reality?

4. The Grid isn’t a new concept • Using multiple distributed resources to cooperatively work on a single application has been around for decades • Networked OS (70s) • Distributed OS (80s and 90s) • Heterogeneous Computing http://www.acis.ufl.edu/hcw2006/ • Parallel Distributed Computing • Metacomputing http://www.cnds.jhu.edu/research/metacomputing/

5. How is a grid different? • Grids focus on site autonomy • Grids involve heterogeneity • Grids involve more resources than just computers and networks • Grids focus on the user

6. Grid Criticisms… • “The Grid is a solution looking for a problem.”

7. Grid Criticisms… • “We tried to install Globus and found out that it was too hard to do. So we decided to just write our own...”

8. #1 Why don’t Grids have basic functionality yet? • Grids aren’t being used for what they were “meant to be” • “Application developers aren’t using multiple machines at geographically distant sites to coordinate and solve a single application.” • Used to solve embarrassingly parallel applications such as Seti@home and Entropia

9. #1 Why don’t Grids have basic functionality yet? • What is functionality? • The ability to run an application in a straightforward manner using single commands for a user to just run.

10. #1 Why don’t Grids have basic functionality yet? • Running a process isn’t simple • Steps: • Determine what software is on the machine and install it, then check it • Make sure all security credentials are setup (certificates, keys, accounts, etc.) • Determine availabile resources • Submit to a Grid computer (set up environment) • Run • Cleanup

11. #1 Why don’t Grids have basic functionality yet? • “The few current users have had to go through heroic measures to achieve any functionality at all.” • Complicated due to socio-political problems • It is difficult setting up and maintaining a piece of new software on one system let alone many systems • Resistance from users who would have to learn a new environment

12. Moral #1: Before we can have a successful Grid, we must have a functional Grid

13. Grid Criticisms… • “Cynics reckon that the Grid is merely an excuse by computer scientists to milk the political system for more research grants so they can write yet more lines of useless code” – Economist, June 2001

14. #2 Why aren’t there more Grid application developers and users? • There is a need for user input to guide research, but users are hard to find and input is hard to receive despite large communities at NSF supercomputer centers • Groups are running specialized code targeted to specific platforms • Even when they use the Grid, they use it “improperly” they use it to gain access to specific machines, not as the Grid was envisioned.

15. #2 Why aren’t there more Grid application developers and users? • Making apps Grid enabled is seen by some as a distraction from getting real science done • Grid work is a buzzword for funding, but takes away from other software development work

16. Moral #2: Without users there can be no Grid • Better software tools to ease transition to a new environment • Standards for uniform access to Grid services • Deployment of Grid software must be made easier

17. #3 Where are the Grid software tools to aid application developers? • Parallel programming was difficult so few people attempted it until debuggers, fast compilers and other flexible tools were available • There is active work on a set of software tools for the Grid such as Globus, however it has a steep learning curve

18. #3 Where are the Grid software tools to aid application developers? • This is due to socio-political problems • Most projects are academic • Little or no funding funding for hardening code • Chicken and Egg problem • To make tools, you need use cases • Without tools to use, you don’t know what to ask for • Constructing a reasonable use case is difficult, it needs specific details without too much overwhelming detail, especially if you don’t know what is possible

19. #3 Where are the Grid software tools to aid application developers? • Different language/communication styles • Computer Science-Centric projects • Application-Centric

20. Moral #3: The Grid needs tools that facilitate the use of a Grid

21. #4 How do we make Grids secure? • For many systems, the mandatory “security is important” paragraph is all the depth problem receives • GSI is accepted security infrastructure and is used in things such as CVS, OpenSSH • However, that only resolves Authentication

22. #4 How do we make Grids secure? • Local accounts are still needed and the issue of authorization and usage policy is not address uniformly

23. Moral #4: Without a security infrastructure(including higher-level tools) users will not take advantage of the Grid

24. #5 How can we define standard interfaces and definitions for the Grid? • No simple answer for the following questions: • How do I run a job on the Grid? • What sort of monitoring is available? • Where do I get information about the Grid resources? • How do I make sure this operation is secure?

25. #5 How can we define standard interfaces and definitions for the Grid? • A lingua franca needs to be created to promote greater interoperability between current tools • Lack of agreement between developers over simple terms such as: • Job • Resource

26. #5 How can we define standard interfaces and definitions for the Grid? • Example • Globus Metacomputing Directory Service (MDS) and Grid Monitoring Architecture (GMA) • MDS concentrates on resource discovery (2 protocols) • GMA concentrates on the provision of data (3 implementations) • However, rather than interoperate to work off each other’s strengths, there is little collaboration between these standards, interfaces and protocols

27. #5 How can we define standard interfaces and definitions for the Grid? • The Global Grid Forum has standardization as a goal • But so does Peer-to-Peer Working Group and the New Productivity Initiative for example • Who will standardize between the standardization bodies?

28. Moral #5: The lack of standards continues to hinder the interoperability needed for the Grid

29. #6 How can we manage variance on the Grid? • Grid resource behavior is unpredictable • Unsure of who else is using a resource and in what way • Users want not only fast execution times from their applications, but unpredictable behavior, and would be willing to sacrifice performance in order ot have reliable run times.

30. Moral #6: Varying behavior is a fact in the Grid and must be addressed • Teach users how variance can affect their performance • Scheduling algorithms have to be made to incorporate variance • Information services have to incorporate the handling of the additional variance information

31. #7 How can deployment be made easier? • Deploying a test bed is difficult • Firewalls are often a major stumbling block • Accounting methodology • Accounts have to be made for resources that never had policies for Grid access in mind

32. Moral #7: If the deployment and setup of software isn’t seamless, getting users to adapt their systems to be Grid compatible will not happen • Halfway solutions can be worse than no solution

33. #8 Where are the benefits to encourage sharing on the Grid? • Grid computing has high appeal to the end user and to the top-level policy makers • End Users: access to resources spread out geographically and across several domains • Top-Level: Enterprise-wide organization policy (NASA could direct 90% of its resources to deflecting near-Earth collision)

34. #8 Where are the benefits to encourage sharing on the Grid? • Tragedy of the Commons • Users don’t play nice; they will act in their own interests and degrade and overuse any shared resource • Middle-Level resource owners and managers are caught in the middle • Users will exploit Grid technology increasingly to consume more resources • Grid accounting can’t provide the accounting or control needed so ssers end up getting a free ride • How can Middle-Level managers prove their value to policymakers while controlling user consumption?

35. Moral #8: Benefits to middle-level resource ownersk must be made clear in order fo the Grids to be deployed

36. #9 How can we fund the work needed for a functioning Grid? • Work to coordinate sites and resources are seen as not new enough so it doesn’t get funding • Fund groups to work on the same problem but using different methods. Then fund to combine them. • Funding several groups is seen as wasteful • Tensions caused by fight between Best Approach and Standard

37. Moral #9: Without support for basic work in functionality, standards and software engineering, the Grid will not live up to its potential • Funding agencies need to be educated on the needs of Grid researchers • Money should be made for mundane tasks to get software beyond beta versions, standardization, setting up test beds, functionality

38. #10 Where are the performance metrics for success? • Success can not be determined without metrics to compare a project against

39. Moral #10: There isn’t a moral • Proposed Success Metrics • No more “Grid” papers • Instead, papers with the following footnote, “This work was achieved using the Grid” • When supercomputer centers don’t give a user the choice of using their machines or the Grid • They will just use the Grid

40. Moral #10: There isn’t a moral • Proposed Success Metrics (Cont.) • When a SuperComputing demo can be run any time of the year • The SuperComputing demo runs every year since 1988 in November for one week • “For one week every year, it’s the world’s fastest nework, providing direct connectivity to a multitude of worldwide research networks.” Source: SC|05 http://sc05.supercomputing.org

41. Agnostic 1 • In the introduction, the authors point out some differences between old distributed systems and today’s Grids. Are these all the major differences? If not, can you suggests others grid distinctive characteristics?

42. Agnostic 2 - “1. Why don’t Grids have basic functionality yet?” • According to the authors, grids do not have basic functionality because to run a grid application one has to follow a sequence of intricate steps, like installing software, setting security, finding the appropriate resource and submitting a job, which look daunting and extremely complex. Do you think this is a fair description as for users running applications on the grid? How is the Service Oriented Architecture helping improve that scenario?

43. Agnostic 3 - “3. Where are the Grid software tools to aid application developers?” • Do you agree with the statement that the difficulty of developing useful tools for the grid is because there are not well defined use cases? Is it really a chicken and egg problem?

44. Agnostic 4 - “4. How do we make Grids secure?” • It is mentioned that the authorization issue has not been fully addressed, and typically imply using a local account to access a particular resource. This would difficult tasks like auditing the usage of a certain resource. Could you provide some information as for the advances in this matter? • Another aspect of grid security is reliance on the grid services. A service can mimic a real application and obtain sensitive information from the users. What work has been done in this respect?

45. Agnostic 5 - “5. How can we define standard interfaces and definitions for the Grid?” • It is mentioned during the introduction that none of the standards defined at the time of writing the paper (2002) were standing still. Is this assertion still true? • Could you briefly talk about the current status of the Global Grid Forum and the Enterprise Grid Alliance groups as for grid standards?

46. Agnostic 6 -“8. Where are the benefits to encourage sharing on the Grid?” • Could you suggest some ways to enforce the user to “play nicely”? In particular, how to manage multiple user applications that can potentially exhaust the grid resources? • Also, can you think of good reasons to convince managers for sharing resources?

47. Agnostic 7 -“10. Where are the performance metrics for success?” • The second success metric says that users are not given a choice of using or not the grid, but just use it. What is your view of this statement? Aren’t the authors assuming that all problems are suitable to get solved in a grid environment?