Dr. Rajkumar Buyya

1. WW Grid SensorGrid: A new Cyberinfrastructure Integrating Sensor Network and Grid Computing for e-Science Applications Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.buyya.com Dr. Rajkumar Buyya

2. WW Grid SensorGrid: A new Cyberinfrastructure for Linking the Physical World with the Digital World Fellow of Grid Computing Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australiagridbus.org/~raj/tut/gridbus.zip Dr. Rajkumar Buyya

3. GRIDS Lab @ Melbourne R & D Education • The youngest and one of the largest research labs in the CSSE Dept: • 2 PostDocs • 2 Research Programmers • 7 RHD (6 PhD) students • ~5 honours/masters projects • Funding • National and International organizations • Australian Research Council • Many industries (Sun, StorageTek, Microsoft, IBM) • University-wide collaboration: • Faculties of Science, Engineering, and Medicine • Many national and international collaborations. • Academics • Industries • Software: • Our Grid middleware technologies are widely in academic and industrial users. • Publication: • My research team produces 20% of our Dept’s research output. + Community Services

4. Books at Glance: Co-authored/edited

5. Outline • Introduction • Technology Trends in Sensors and Grids • Sensors and Grids Integration Proposal • Grid Computing • What is it?, architecture, and technologies • NOSA (NICTA Open SensorWeb Architecture) • Summary and Conclusion

8. Internet, Web, and Grid Effect 140 ‘Web Services and Grid Effect’ 120 100 The 'Network Effect’ kicks in, and the web goes critical' Number of hosts (millions) Business $$$ 80 60 40 20 0 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 TCP/IP HTML Mosaic XML 4. with XML PHASE 2. The Internet is Born 3. The World Wide Web 5. The Grid 1. Packet Switching Networks HTML hypertext system created 1969: 4 US Universities linked to form ARPANET TCP/IP becomes core protocol CERN launch World Wide Web 1972: First e-mail program created Domain Name System created IETF created (1986) NCSA launch Mosaic interface 1976: Robert Metcalfe develops Ethernet

9. database Grid Resource Broker SensorWeb Service R2 R3 R4 R5 RN R6 R1 Grid Information Service The customer can have access to the Grid Resource Broker, and then through Web Service, requests can be sent to obtain real sensor network data. Sensor Network

10. Traditional Devices, Sensors, and their Networks at Glance • Traditional Devices (Computers and High End Resources) are: • Powerful • Connected to Power Grid – so we don’t worry too much about it power consumption • Large Storage Space • Good for archival and large-scale analysis • Connected by High Bandwidth/Speed Network • Sensors: • Less powerful • Scarcity of power (battery operated, or even self-power generated) • Less Storage • No good for archival • Connected by Low Bandwidth/Speed Network • But they can sense/smell a phenomena in the physical world.

11. It will be nice to marry them • Both of them benefit: • Grids: • Get Eye to see the world (so that it can sense and assist the us) • Sensors: • Off load their processing, storage, archival, analysis, etc. requirements to the Grid. • Sensors Grids = SensorGrid

12. Outline • Introduction • Technology Trends in Sensors and Grids • Sensors and Grids Integration Proposal • Grid Computing • What is it?, architecture, and technologies • NOSA (NICTA Open SensorWeb Architecture) • Summary and Conclusion

13. Grid (computing) Paradigm:Cyberinfrastructure for sharing resources • Inspired by Power Grid! • * A service-oriented/utility computing paradigm that enables seamless sharing of geographically distributed, autonomous resources. • * This was the original aim of building Internet although it ended up in giving birth to email!

14. Grid Information Service Grid Resource Broker Application R2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service database Grids have Emerged as Scalable Cyberinfrastructure for e-Science Applications

15. Type of Services Modern Grids Offer • Computational Services – CPU cycles • SETI@Home, NASA IPG, TeraGrid, I-Grid,… • Data Services • Data replication, management, secure access--LHC Grid/Napster • Application Services • Access to remote software/libraries and license management—NetSolve • Information Services • Extraction and presentation of data with meaning • Knowledge Services • The way knowledge is acquired and managed—data mining. • Utility Computing Services • Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Utility Grid Knowledge Grid Information Grid ASP Grid Data Grid Computional Grid

16. Computational Economy Security Data locality Resource Allocation & Scheduling Uniform Access System Management Resource Discovery Application Construction Network Management Grid Capabilities

17. Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India I-Grid Japan NAGERI Korea... N*Grid Singapore NGP USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture and many more... Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Infosys – Business Grid StorageTek –Grid.. and many more Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science Some Grid Initiatives Worldwide 27 million 1.3 billion – 3 yrs 2? billion 120million – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 1 billion – 5 yrs http://www.gridcomputing.com

18. The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand Distributed Data WWG Gridbus World Wide Grid! On Demand Utility Computing www.gridbus.org

20. Gridbus Architecture Layer

21. Application Code Explore data 1 Data Visual Application Composer 10 Results+Cost Info 2 GridResource Broker Data Catalogue 5 4 Grid Info Service Data Replicator (GDMP) 12 6 3 ASP Catalogue Grid Market Directory 9 7 Job Results 8 Grid Service (GS) (Globus) Bill Alchemi GS CPU orPE PE GTS 11 GridbusGridBank Cluster Scheduler PE GSP (Accounting Service) GSP (e.g., IBM) GSP (e.g., VPAC) GSP (e.g., UofM) On Demand Assembly of Services: Putting Them All Together Data Source (Instruments/distributed sources) Cluster Scheduler PE Grid Service Provider (GSP)(e.g., CERN)

22. Alchemi: .NET-based Enterprise Grid Platform & Web Services Alchemi Manager Web Services Internet Alchemi Users Internet • SETI@Home like Model • General Purpose • Dedicated/Non-dedicate workers • Role-based Security • .NET and Web Services • C# Implementation • GridThread and Job Model Programming • Easy to setup and use • Widely in use! Alchemi Worker Agents

23. Some Users of Alchemi Tier Technologies, USA Large scale document processing using Alchemi framework Satyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework CSIRO, Australia Natural Resource Modeling The University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbH, Germany Asynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework. The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Many users in Universities: See next for an example.

25. The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999-2001] And Extends its notion for Data and Service Grids

26. Gridbus Broker Architecture Gridbus Client Gribus Client Gridbus Client (Bag of Tasks Applications) App, T, $, Opt (Data Grid Scheduler) Gridbus Farming Engine Schedule Advisor Trading Manager RecordKeeper Grid Dispatcher Grid Explorer Grid Middleware TM TS $ GE GIS, NWS Grid Info Server RM & TS G $ Data Catalog Data Node C $ U G Unicore enabled node. Globus enabled node. L A RM: Local Resource Manager, TS: Trade Server Alchemi enabled node.

27. Gridbus Services for eScience applications • Application Development Environment: • XML-based language for composition of task farming (legacy) applications as parameter sweep applications. • Task Farming APIs for new applications. • Web APIs (e.g., Portlets) for Grid portal development. • Threads-based Programming Interface • Workflow interface and Gridbus-enabled workflow engine. • Resource Allocation and Scheduling • Dynamic discovery of optional computational and data nodes that meet user QoS requirements. • Hide Low-Level Grid Middleware interfaces • Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

28. Click Here for Demo Figure 3 : Logging into the portal. Drug Design Made Easy!

29. Sample Applications of Gridbus Broker • Molecular Docking - WEHI • Drug Discovery • Brain Activity Analysis – Osaka University • Neuroscience studies • Natural Language Engineering – Melbourne NLP • Indexing of newswire data • High Energy Physics – School of Physics/Melbourne • Belle experiment data analysis • Finance - Portfolio Analysis – U. Comp. Madrid/Spain • Investment risk analysis • Astronomy – School of Physics@UoM and QUT (Queensland University of Technology) • Australian Virtual Observatory • Spreadsheet Processing • Microsoft Excel

30. Outline • Introduction • Technology Trends in Sensors and Grids • Sensors and Grids Integration Proposal • Grid Computing • What is it?, architecture, and technologies • NOSA (NICTA Open SensorWeb Architecture) • Summary and Conclusion

31. Sun Proposal: For Integration of Sensors and High End Computers using Network Source: Anil Velluri, Sun

32. Our Proposal: SensorGrid - Integrated sensor network and grid architecture sensor and actuator networks query response grid command sense actions actuate

33. Sensor Web and NOSA • Goal: To create Web-based sensor networks by exploiting web-connected sensors (flood gauges, air pollution monitors, mobile heart monitors, satellite-borne earth imaging monitors). • This would make all sensors and repositories of sensor data discoverable, accessible, process-able and where applicable controllable via the WWW. • OGC (Open Geospatial Consortium) describes five important encoding and service standards for a Sensor Web Implementation (see next slide).

34. What is NOSA? • The NICTA Open SensorWeb Architecture (NOSA) project is developing a complete standards compliant platform and middleware for integration of sensor networks with emerging distributed computing platforms such as Grids. • It confirms to Web Services standard defined by W3C (World-Wide Web) and SensorML (Sensor Model Language) standard defined by OpenGeospatial Consortium.

35. What are NOSA Benefits? • This integration of sensor networks with Grid computing brings out dual benefits: (i) sensor networks can off-load heavy processing activities to the Grid and (ii) Grid-based sensor applications can provide advance services for smart-sensing by deploying scenario-specific operators at runtime. • Fundamental services are provided by lower-level layer components whereas components at the higher-level layer provide tools for creation of applications and management of life-cycle of data captured through sensor networks.

36. SensorWeb Applications SensorWeb User-Level Middleware SensorWeb Core Middleware SensorWeb Fabric or Emulation NICTA Open SensorWeb Architecture (NOSA) ApplicationsLayer ApplicationDevelopmentLayer ApplicationServicesLayer Sensor FabricSimulationEnvironment

37. OGC-Sensor Web Enablement Standard Specifications • SensorML: XML encoding language for sensors. Used to discover, query and control Web-resident sensors. • Observations & Measurements: The general models and an XML encoding for what a sensor observes or measures (The value returned by or derived from a sensor observation -e.g. quantity, count, boolean, category, ordered category, position-). • Sensor Collection Service: A service to fetch observations from a sensor or constellation of sensors. Provides real time or archived observed values. Clients can also obtain information that describes the associated sensors and platforms. • Sensor Planning Service: A service by which a client can determine collection feasibility for a desired set of collection requests for one or more mobile sensors/platforms, or the client may submit collection requests directly to these sensors/platforms. SPS enables sensor tasking, acquisition requests, processing and simulation requests, and registration for alert notification. • Web Notification Service: A service by which a client may conduct a dialog with one or more other services. Provides a means for Sensor Planning Services to alert people, software, or other sensor systems of SPS results or alerts regarding phenomena of interest.

38. A sensor node includes a sensor and radio board. Then the sensor node could send data message through radio to Station which is the gateway connected with the PC. Here is the architecture of the interface connecting sensor network and the real PC.

39. Crossbow Wireless Sensor Network Kit and Testbed Setup in GRIDS Lab Sensor MTS 300 Base station MIB510CA

40. SensorWeb Implementation

41. A SensorWeb Collection Client

42. Performance of collecting auto-sending and query data

43. Performance of collecting auto-sending and query data

44. Outline • Introduction • Technology Trends in Sensors and Grids • Sensors and Grids Integration Proposal • Grid Computing • What is it?, architecture, and technologies • NOSA (NICTA Open SensorWeb Architecture) • Summary and Conclusion

45. Summary and Conclusion • Sensors and Grids two major elements of emerging Cyberinfrastructure that support e*Applications (e-Science, e-Business, e-Health, e-Life). • They support creation of smart office, house, and business environments. • Current developments in Sensors and Grids is heavily driven by applications and both compliment and need each other. • SensorGrid is just emerging and there are many opportunities available for creating many interesting applications in various domains.

46. References • Rajkumar Buyya and Srikumar Venugopal, A Gentle Introduction to Grid Computing and Technologies, CSI Communications, pages 9-19, Computer Society of India, Vol.29, No.1, July 2005. • Chen-Khong Tham and Rajkumar Buyya, SensorGrid: Integrating Sensor Networks and Grid Computing, CSI Communications, pages 24-29, Computer Society of India, Vol.29, No.1, July 2005.

47. Any Questions ? Web - http://www.gridbus.org http://www.buyya.com