A Lightweight Platform for Integration of Resource Limited Devices into Pervasive Grids

1. A Lightweight Platform for Integration of Resource Limited Devices into Pervasive Grids Stavros Isaiadis and Vladimir Getov University of Westminster http://www.coregrid.net s.isaiadis@wmin.ac.uk, v.s.getov@wmin.ac.uk

2. Outline • Current Grid Systems • Motivation • Project Overview • Research Issues • Operational Overview • Conclusion European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

3. Current Grid Systems • Grids were initially developed to support large (mainly) scientific data-centric applications • Resources were in abundance and Grids orchestrated sharing in large scale and/or harnessing of idle resources • Current Grid systems assume: • Plenty of resources • Relatively reliable and static networks • Traditionally, Grid systems are not concerned with the platform’s footprint: platform is “heavy” and cannot be deployed in resource limited devices European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

4. Grids and “Small” Devices • But networks have evolved. The “edge” of networks increasingly consists of resource limited and/or mobile devices • The Grid, however, hasn’t yet allowed the integration of these devices • While access to the Grid and job submission is generally supported (e.g. through portals), contribution of resources is very difficult • Resource constraints make installation of Grid middleware prohibiting European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

5. Outline • Current Grid Systems • Motivation • Project Overview • Research Issues • Operational Overview • Conclusion European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

6. But Why Integrate Resource Limited Devices? • Future Grid systems should be truly pervasive and ubiquitous • Mobile and pervasive computing attract a big part of current research and industry funds and hold a big share of the market pie • Small and/or mobile devices increasingly offer functionality not available in traditional Grid nodes –multimedia equipment, sensors, global positioning systems etc. European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

7. Outline • Current Grid Systems • Motivation • Project Overview • Research Issues • Operational Overview • Conclusion European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

8. A Lightweight Grid Platform • A lightweight platform will provide the foundations for our project: • Should be based on the CoreGRID component model • Will consist of a small well defined set of core components • Extra functionality through pluggable components • Small footprint and communication overhead so that it won’t drain resources European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

9. Outline • Current Grid Systems • Motivation • Project Overview • Research Issues • Operational Overview • Conclusion European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

10. Dealing with Device Dynamicity • Increased dynamicity is not acceptable in complex and heavily loaded Grid systems • Solution: • Hide devices behind a proxy and delegate finer control (scheduling, monitoring, recovery etc) to a local community system • Devices now form a virtual “cluster” and are presented as a single entity to the Grid. European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

11. Virtual Aggregator Services • Similar services (that implement the same interface) are aggregated and published through a single interface at the proxy • The aggregator service implementation orchestrates access to the services it represents (by coordinating the various components like schedulers, indexes etc.) • A single consistent interface to all aggregated resources is presented regardless of the internal state of the “cluster” European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

12. Making an Unstable Environment Virtually Stable • When aggregating services that provide controlled access to raw resources (like cpu cycles, memory, storage etc.), we don’t present the total aggregate but only a fraction • This way we can mask internal failures by reallocating resources from within the cluster without notifying (and thus placing extra burden on) higher level Grid components European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

13. Dealing with Failures • Even behind a cluster, small devices still suffer from a high number of failures and we must provide the best possible quality of service • Solution: • A monitoring and failure recovery framework to predict, detect, diagnose and recover from failures • Forecasting components to support the community scheduling system providing recommendations on the resource usage plans European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

14. The Monitoring Framework • A set of small monitoring agents installed in the devices. They collect state information and forward them to the proxy • A set of monitoring components at the proxy that collect and analyze the information • A monitoring “gateway” makes all decisions and orchestrates the other components • A forecasting component provides predictions on the best possible usage plan (according to usage and failure history) • Failure recovery components decide on the recovery plans based on predefined scenarios European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

15. Outline • Current Grid Systems • Motivation • Project Overview • Research Issues • Operational Overview • Conclusion European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

16. Component Prototypes • Currently using Globus Toolkit 3.2 middleware • Will port to GT4 in the very near future • Prototyping components using Java • Prototypes are quite restrictive at this stage but they will: • Help us get a better insight • Help us identify possible obstacles • Provide the skeleton for the functional final platform European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

17. Add/Update Interface Update Index Discovery and Interface Identification Registration Operational Overview: Registration of Services AGGREGATED SERVICE INTERFACES RESOURCE META-DATA REPOSITORY LIVE RESOURCE INDEX European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

18. Job Submission Forward App/ Code Job Requirements Consult Application Tasks Distribution Operational Overview: Job Submission AGGREGATED SERVICE INTERFACES TASK EXTRACTION ENGINE SERVICE IMPLEMENTATION COMMUNITY SCHEDULING SYSTEM LIVE RESOURCE INDEX European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

19. Event notification + status info diagnosis forward plan update Event notification + status info Operational Overview: Failure Recovery JOB META-DATA REPOSITORY FAILURE DIAGNOSIS FORECASTING COMPONENT MONITORING GATEWAY FAILURE RECOVERY LIVE RESOURCE INDEX PLAN DEPLOYMENT European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

20. Outline • Current Grid Systems • Motivation • Project Overview • Research Issues • Operational Overview • Conclusion European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

21. Conclusion • The lightweight platform could provide the foundations for truly ubiquitous Grids • Coupled with a lightweight monitoring framework, it is a big first step towards full and efficient integration of resource limited devices into the Grid • The early prototyping phase will help us gain a better insight on the issues that might arise and will provide the skeleton for the final deliverable platform European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

22. Questions? …or comments, suggestions, ideas… Contact: www.isaiadis.com s.isaiadis@wmin.ac.uk