DFuse and MediaBroker: System support for sensor-based distributed computing

1. DFuse and MediaBroker:System support for sensor-based distributed computing Kishore Ramachandran (http://www.cc.gatech.edu/~rama) College of Computing Georgia Tech Colleagues: Rajnish Kumar, Bikash Agarwalla,Junsuk Shin, David Hilley, Dave Lillethun, Jin Nakazawa, Bin Liu, Xiang Song, Nova Ahmed, Seth Horrigan, Matt Wolenetz, Arnab Paul, Sameer Adhikari, Ilya Bagrak, Martin Modahl, Phil Hutto

2. High connectivity Low connectivity / Wireless Cameras, sensor nodes High Performance Computing (HPC) resources Computing/Communication Continuum Sensor Network HPC resources Ambient Computing Infrastructure

3. What does this enable? • Application context • distributed sensors with varying capabilities • control loop involving sensors, actuators • rapid response time at computational perception speeds • Sample applications • Video based surveillance • Transportation • Emergency Response • Collaborative search and rescue • Evacuation management • “Aware” Environments

4. Application Characteristics • Physically distributed heterogeneous devices • Interfacing and integrating with the physical environment • Diverse stream types (low to high BW) • Diverse computation, communication and power capabilities (from embedded sensors to clusters) • Stream fusion/transformation, with loadable code • Resource scarcities • Dynamicjoin/leave of application components

5. Stampede MediaBroker Key Requirements Middleware • Programming infrastructure • Distributed data fusion • Stream data management Network Level • Protocol stack • Energy efficient routing Ambient HPC resources • Grid DFuse SensorStack Streamline

6. Stampede (TPDS 2003) • Distributed programming system covering the continuum • Temporal stream data transport • Multilingual (C, C++, Java) program components sharing data abstractions • Multiple platforms (x86-Linux, ARM-Linux, x86-Windows, x86-Solaris, Alpha-Tru64) thread • put(ts, item) • get(ts, item) • consume(ts) • many to many connections • time sequenced data • correlation of streams • automatic GC Channel

7. MediaBroker Key Requirements Middleware • Programming infrastructure • Distributed data fusion • Stream data management Network Level • Protocol stack • Energy efficient routing Ambient HPC resources • Grid Stampede DFuse SensorStack Streamline

8. Collage Sink Filter Fusion Channel (a ‘Virtual Sensor’) Cameras Producers (sensors or other fusion channels) Consumers (actuators or other fusion channels) f() . . . . . . DFuse (ACM SenSys 2003) • Future Sensor Networks • Today’s handhelds, gateways • Ubiquitous high-bw sensors • Challenges • Overlaying the application onto the physical network • Programming abstraction for data fusion

9. Fusion Module Fusion Module Collage Placement Module Sink Filter Cameras Resource Monitor, Routing Layer Interface Resource Monitor, Routing Layer Interface Operating System Operating System Hardware Hardware Fusion( Arg[ ]) { .. } Cost Function Placement Module DFuse functions: Placement of fusion and relay points Plumbing as required Dynamic migration of fusion points

10. Status of DFuse • Fusion and placement modules implemented on top of Stampede • A prototype iPAQ farm (simulating future sensor network) runs DFuse • Stampede and DFuse available for downloads • MSSN, a simulator for sensor networks middleware design guidance (BaseNets’04, IJNM’05, Wolenetz’s thesis) • Available for downloads

11. MediaBroker Key Requirements Middleware • Programming infrastructure • Distributed data fusion • Stream data management Network Level • Protocol stack • Energy efficient routing Ambient HPC resources • Grid Stampede DFuse SensorStack Streamline

12. MediaBroker • An architecture for stream management • A clearing house for sensors and actuators in a given space • Stream registry, discovery, plumbing, sharing, • Dynamic connection of sources (producers) and sinks (consumers) • Dynamic injection, and safe execution of transformation code • Feature extraction, fusion • Dynamic sharing of transformations and streams

13. Producer Consumer Producer Consumer Type Server • Elements • Type server: stores data types, relationships, and transformation code • Transformation engine: allow safe execution of injected code on cluster nodes • Scheduler: manages workload, and allows prioritizing transformation requests • Data brokers: manages connections between producers and consumers Transformation Engine Data Broker Data Broker Data Items Transformation Requests Transformation Code Transformation Engine Scheduler Transformation Engine

14. What it enables • Dynamic instantiations and sharing of transformations

15. MediaBroker Status • MediaBroker V.1 • A subset of the functionalities • Application example: Family intercom • IEEE PerCom 2004, PMC 2005 • MediaBroker++ • Currently under development • EventWeb built on top

16. Key Requirements Middleware • Programming infrastructure • Distributed data fusion • Stream data management Network Level • Protocol stack • Energy efficient routing Ambient HPC resources • Grid Stampede DFuse MediaBroker SensorStack Streamline

17. SensorStack • Adaptability Vs. Stackability of protocol layers • Adaptability deals with cross-layer data • A must for wireless sensor networks • Stackability deals with cross-layer functionalities • A must for modular design • Principle behind SensorStack • Decouple data from functionalities

18. SensorStack without Cross-layering Support Fusion requirement Fusion layer Application Neighborhood Information, Topology Data periodicity, Size, delay tolerance AODV routing Flood routing Data transmission requirement Fusion data transmission requirement  Link quality information, Neighborhood status change,  topology Time Sync Service MAC Time synchronization accuracy, accuracy requirement

19. Information Exchange Service (IES) Design Goals: • Efficient use of limited memory • Simplifying information sharing interface • Extensibility • Asynchronous delivery of the information • Complex event notification

20. IES Design • Data management module  Stackability by using standard data interface • Publish/subscribe based shared memory system • Fully-associative cache for performance • Event management module  Adaptability by notifying when to adapt • Complex event notification • Reactive memory access

21. DRE: Data request event RSE: Rule satisfied event DAE: Data available event EMM: Event management module

22. SensorStack with Cross-layering support Implemented in TinyOS and iPAQ Linux Initial results (increasing application lifetime) very promising

23. Application Lifetime Improvementwith Cross-layer Information DFuse performance without Cross-layer information DFuse performance with Cross-layer information for role-migration

24. Key Requirements Middleware • Programming infrastructure • Distributed data fusion • Stream data management Network Level • Protocol stack • Energy efficient routing Ambient HPC resources • Grid Stampede DFuse MediaBroker SensorStack Streamline

25. Grid Infrastructure MediaBroker DFuse Stampede SensorStack

26. Streamline (MMNC 2006) Scheduling problem • Input: • Computation and communication requirements of various stages of a coarse-grain dataflow graph • Application-specified constraints • Current resource (processing and bandwidth) availability • Resource specific constraints • Output: • Placement of the stages of the pipeline on available HPC resources • Performance criteria: • latency and throughput of the application

27. S0 Stage Prioritization S2 S3 {S2 S0 S1 S3} S1 Streamline Scheduler • Expects to maximize throughput by assigning best resource to most needy stage • Additional policies concerning resources, applications, and local schedulers can be incorporated in the cost of a particular assignment R0 R3 R1 Resource Filtering R2 {S2 {R0 R2 R3}} Resource Selection {S2 R0}

28. Streamline System Architecture • Results • Outperforms condorby an order of magnitude for both compute and communication bound kernels,particularlyunder non-uniform load conditions • Performs close to Simulated Annealing but at considerably low scheduling time (by a factor of 1000)

29. Streaming Grid • Streamline scheduler integrated into Globus toolkit • Example of a mock traffic monitoring app as a service composition using Web Services • Blue boxes are the Streaming Grid services

30. Demo

31. Demo Output (live if stars are aligned!) http://www.cc.gatech.edu/~bikash/sgrid/trafficapp/ http://www.cc.gatech.edu/~bikash/sgrid/trafficapp/trafficapp.html • What is the takeaway? • Several technologies working together • Service composition, Streamline scheduling, Web Services, Globus toolkit to process the video stream and show the output in the browser • Streaming grid instantiates, connect, manages the streaming app

32. High connectivity Low connectivity / Wireless Cameras, sensor nodes High Performance Computing (HPC) resources Computing/Communication Continuum Sensor Network HPC resources Ambient Computing Infrastructure

33. Conclusions • MediaBroker • stream transformation and typed transport engine • DFuse • data fusion architecture • Stampede • distributed programming environment • SensorStack • Information Exchange Service for cross-layer support • Streamline • Scheduling support for streaming apps on grid MediaBroker DFuse Stampede SensorStack Streamline

34. Ongoing Work • Programming tools • Adaptive resource management • Marrying grid computing and ubiquitous computing • Wireless networking considerations • sensorstack • energy efficient protocols • mobility considerations

35. Web Links • http://www.cc.gatech.edu/~rama • http://www.cc.gatech.edu/~rama/stampede • http://www.cc.gatech.edu/~rama/up • http://www.cercs.gatech.edu/

36. Applause!!! 

38. Sensor Networks • Current Sensor Network Nodes • Limited capabilities (Mica-2) • habitat monitoring, vineyard monitoring… • Recent trends • iMotes 8x radio, 4x CPU, power++ • Telos motes >3x radio, similar CPU & power • Future Sensor Network Nodes • Today’s handhelds, gateways • Ubiquitous high-bw sensors Source: CACM #47-6 “The platforms enabling wireless sensor networks”, Hill, Horton, Kling, Krishnamurthy, 2004.

39. HPC resources Unix / Linux / XP cluster

40. uncompress collage filter Overlaying the fusion graph Fusion Applications : need hierarchical fusion support

41. Family Intercom • Clients connected via MediaBroker • Type attributes include audio rate and buffer specs • A client tracking system used for I-combo selection • ‘Colocated’ clients can perform mixing for N-way conferencing

43. Consumers Producers . . . . . . Fusion Module Structure Management  Plumbing Issues

44. f2() f() . . . . . . f1() Fusion Module Computation Management  Dynamic embedding of user-specified fusion function Correlation and aggregation of input streams Fusion channel migration

45. f2() f() . . . . . . f1() Fusion Module Computation Management  Dynamic embedding of user-specified fusion function Correlation and aggregation of input streams Fusion channel migration Memory Management

46. f2() f() . . . . . . f1() Fusion Module Error and Status management  Failure / Latency hiding

47. S1 S2 Collage Sink (Display) Filter S3 Sources Placement Module User inputs the task graph And, a cost function.

48. Simple Solution ? • Why not push the fusion points towards its sources ? • Data sources may be lying all around • Fusion points may cause data expansion • Network is dynamic

49. DFuse: Placement Module’s Algorithm • Three phases: • Naïve role assignment • Deploy task graph into the network • Start app at a designated root node and delegate task graph subtree roles to richest neighbors recursively • Optimization • Given anticipated application behavior (attributes in the task graph), perform rapid local decisions to adjust which node performs which role. • Decisions guided by application-provided cost-function • Maintenance • Monitor actual application behavior and perform less frequent optimizations given application-provided cost-function

50. Source n1 n2 2 Kbps 2 Kbps f() Sink 1 Kbps 2 Kbps Source 2 Kbps Example Cost Function • MT1 (Minimize Transmission Cost-1) CMT1 ( n2, f ) = 9 kbps CMT1 ( n1, f ) = 6 kbps