Future GRID Strategy Beijing 7th of September 2004 (CNES, J.P.Antikidis)

1. Future GRID StrategyBeijing 7th of September 2004(CNES, J.P.Antikidis) • Reminders • - GRID evolution and upcoming concepts • Technological trends • Report on WGISS action 17.4 • The WGISS WAG experiments • Further on activities Beijing 6th Sept 2004, CNES/J.P.Antikidis

2. Exemple of distributed knowledge and process environment Beijing 6th Sept 2004, CNES/J.P.Antikidis

3. IMAGINE…. That you could plug your computer into the wall and have direct access to huge computing resources immediately, just as you plug-in a lamp to get instant light. This is the idea the Grid project is about to make into reality Let’s quote Ian Foster and Carl Kesselman's book (The Grid): "A computational Grid is a hardware and software infrastructure that provides dependable, consistent , pervasive and inexpensive access to high-end computational capabilities". Beijing 6th Sept 2004, CNES/J.P.Antikidis

4. “The Power of Grids” All data systems technologies are concerned : * Networks * Operating systems * Large scale data processing * Protected architectures * Systems creation and programming

5. Classical GRID based GRID Beijing 6th Sept 2004, CNES/J.P.Antikidis

6. Virtual machine defined by Middleware primitives « Pseudo-machine » Beijing 6th Sept 2004, CNES/J.P.Antikidis

7. GRID based activities GRID: Distributed machines organized by an Operating System making individual hardware unvisible GRID is not an end by itself and covers #GRID techno developments #GRID based system concept Present paradox with Middleware evolution: Make GRID invisible from users, so Who are the users (historical trend) ?: # End users directly connected to GRIDs # System developers directly connected to GRID # System developers willing to make use of GRID high level services (Upperware High level O/S that may compile or emulate needed systems) Strategic usersthinking "differently" because of the GRID advent

8. The GRID Cook-book Direct mapping Block module or API primitives "High level O/S: Upperware Beijing 6th Sept 2004, CNES/J.P.Antikidis

9. GRID evolution Creation of "Virtual machines" (Upperwares and HEAVEN) Exchange power against flexibiliy (Upcoming upper-wares) Create "Virtual machines at will"=> Virtual Factories Several approaches: Data flow virtual descriptions "Virtual machine and processing Emulation" "Virtual layer of JAVA machines Creation of "transparent" system connectivity Why not leave Telecom and data transfers under O/S control ? "Distance is not anymore relevant=> Wide Aera Grids

10. From Real => to « Virtual » Blueprint plans (1850) CATIA (1985) (virtual mockup) Formal programming (1948) WINDOWS (1990) MINITEL Physical links(1970) INTERNET (1990) Electronic circuitry (1914) SPICE simulators(1985) Processing systems (1960) GRID+UpperWare (2002) Methodology trends Beijing 6th Sept 2004, CNES/J.P.Antikidis

11. "Hosting of Emulated Applications in a Virtual Environnement" If it is possible to “instruct” a GRID to mimic (or simulate) a computer system, Shouldn’t be possible to use this simulation as a replacement for the real system by directly running the application software within the simulation itself? In short: If one can draw a computer system (that could be a large scale and widely distributed implementation) on a piece of paper , will it be possible to create it with the help of a suitable high level operating system (called Upperware) In such a way it behave like a real system If feasible , the consequences on our way to create operational and distributed collaborative system may change drastically and lead to a conceptual change comparable to Internet revolution ...

12. Instead of creating complex processing system, perhaps better describe them in a powerfull and fully flexible computing topology and make the resulting emulation running as a replacement for the real system. The HEAVEN paradigm "The result produced by running an "emulator" is equivalent to the result produced by the real system." HEAVEN can "mimic" any required system The difficulties experienced by the KRELS are properly solved in a pure mathematic computer world !!!

13. Beijing 6th Sept 2004, CNES/J.P.Antikidis

14. Virtual enabling facilities • Most recent approach are connected to: • Compilers and mapping systems able to "tranlate" user requirement into GRIDbasic primitives (efficient but little flexibility and conventional way to treat the inormation • Emulation of systems embedded into a general purpose supporting GRID creating " virtual machines" • For example the Emulated type approach allows to inject conventional SW and play with system without changing any users'habits (in therory no-one can make the difference in between a Virtual based applicaion and a conventional implementation, but the development cost and effort is several order of magnitude lower Beijing 6th Sept 2004, CNES/J.P.Antikidis

15. From stone-age to modern time Computers (1950)+ network (1960)=> distributed processing + Packet transmission(1980) + UNIX/WINDOWS O/S(1980)+Mosaic high level interface (1990) =Internet revolution Computers(1950)+networks+middleware (1990)= Grid (1995+Upperware (2002)= Virtual defintion and running of processing systems (2005 ….)==> Full virtual data management factories(evolution or revolution ?) Advanced Upperwares like HEAVEN is to GRID the equivalent of an advanced O/S like Windows vis à vis a PC-BIOS (inside the machine everything is 8086 protocols but nobody cares anymore).To day using GRID on a direct fashion (trough middleware primitives) is equivalent to code an applicaion in machine language. Beijing 6th Sept 2004, CNES/J.P.Antikidis

16. Technology trends and goals • Machines running and software implementaion as general purpose services fully shared on an invisible fashion among all operators • Information, processing, organization of system fully virtual (you just pay for GRID use and access to data sets) • Full analogy with Telecomm:You pay for time use of a system, not anymore for individual components uses. • Data use will not anymore associated to an image or a data set, it could be associated to its use • Applications systems created by the users themselves Beijing 6th Sept 2004, CNES/J.P.Antikidis

17. "The WAG Experiments" Beijing 6th Sept 2004, CNES/J.P.Antikidis

18. Response to action 17.25 "J.P.Antikidis with input from J.Hielkema and Yonsock to prepare a paper on the system process for Wide Aera Grid (WAG)" Meeting with FAO (end of August) Coordination with ESA/ESRIN-> MME (Meris-Modis Experiment) CNES organized effort on GRID experiment =>> Proposal to hold demonstrative test based on "real size case" illustrating the capability of WAG

19. Virtual handling of information GRIDs main characteristics: Flexible topology Data flow handling Computing Power Distributed processing =>>Let's add an upper layer "upperware" able to submit user's jobs to the system= Virtual handling of information

20. An illustration of Virtual storage handling Conventional: Locate the information (catalogues etc..) Get hold of the data (media, access costs, transmission & delays) Process the information Transfert the result to suitable place WAG based concept: Create a "Light" Grid connecting a few centers Make their archive accessible from this GRID Enable the creation of jobs submitted to this GRID =>>Then could be demontrated the feasibility of a virtual factory able to dematerialize the handling of information

21. GRIDS species ALL GRID ARE NO EQUIVALENT FROM THE EO-SCIENCE PERSPECTIVE - Macro-Grids (ganging of processors) (dynamic hanling of HW resources, eg multi-CPU machines) ~ meters (efficiency) - Local Grids (Networked computers) with Grapes of computers (local GRIDS, eg:CERN machines) ~decameters (power) Wide Aera Grids (Widely distributed systems) worldwide basis (integration of services) "WAG" concept Beijing 6th Sept 2004, CNES/J.P.Antikidis

22. Upperware(high level O/S) Middleware handling(Basic O/S) “Virtual layout” CAD “like” design(Virtual factory) “Real interface” Virtual Systems internal mechanism Physical layout (general purpose) Beijing 6th Sept 2004, CNES/J.P.Antikidis

23. Induced conceptual changes • No more knowledge of information or/and data-bank physical location (an archive is just known to be accessible trough its naming) • No more need to describe the physical connections in between different locations (since fully supported by the underlying GRID) making in theory any world-wide distributed systems easy to implement. • No more physical handling of information sets. This could trigger completly new merchandising concepts (for ex. come close to Telecomm recharge) • No need to know the mapping of user ’software enabling virtual service to be easealy created and offered to any-one on an autonomous fashion (virtual services) Beijing 6th Sept 2004, CNES/J.P.Antikidis

24. WAG Data providers User Descr. Data User data W.A.GRID based "Virtual elements User agent Real flow System designer GRID Beijing 6th Sept 2004, CNES/J.P.Antikidis

25. A start : The “Meta Computing” Process localisation is not anymore relevant Job control is not fully machine independant and special care is still needed wih respect to Application mappingl

26. A “Proto”-WAG experiment:GRID5000 Grid'5000 is a research effort developping a large scale nation wide infrastructure for Grid research. The current plans are to assemble a physical platform featuring 8 clusters, each with 100 to a thousand PCs, connected by the Renater Education and Research Network. 7 sites are already selected. A 8th one sould be selected soon. All clusters will be connected to Renater at 1 Gb/s (10 Gb/s is expected in near future). This high collaborative research effort is funded by the French ministry of Education and Research, INRIA, CNRS, the Universities of all sites and some regional councils. 10 laboratories are involved, nation wide, in the objective of providing the community of Grid researchers a test-bed allowing experiments in all the software layers between the network protocols up to the applications:

27. WGISS WAG Experiments • Archives are not "physically" but logically" described • Processes are described as "virtual actions" • The underlying HW layout is managed by an underlying Middleware/Upperware fMODIS MERIS Experiment (MME) Virtual Factory Experiment(VFE) …… Beijing 6th Sept 2004, CNES/J.P.Antikidis

28. Near future plans: “Earth Science Grid on Demand” • Within CEOS: Share data and resources from the different agencies • Provide secure and controlled access to the different data storage elements • Share computer resources specific modules • Test and evaluate new processing data fusion algorithms Specific case: MERIS and MODIS vegetation products Beijing 6th Sept 2004, CNES/J.P.Antikidis

29. WAG Experiment The Virtual Factory Experiment Connect several resource center in a WAG Make Process server available Create a virtual distribution scheme Allow the projection by users of needed process Potential partners: Data and GRID providers (Europe+others), Remote job submitter and users (DC representatives like FAO and UNEP )

30. WAG Experiments & WGISS layout Identification of a potential "task" called WAG experiment covering several demonstratives "test case" Task group composed of test case participants: ESA,CNES,NASA,FAO,UNEP etc… - Activity being blessed by WGISS18 ? - Consolidated proposals with respect to the two first experiments headed by ESA (MME) and CNES (VFE) to be established during WGISS19/20 time frame

31. That'all folks! Beijing 6th Sept 2004, CNES/J.P.Antikidis