Prasanta Bose Advanced Technology Center Lockheed Martin Space Systems Company Palo Alto, CA

1. DyMND:Robust Adaptive Coordinationin Dynamic Meshes of Networked DevicesNEST PI MeetingJune 29, 2004 Prasanta Bose Advanced Technology Center Lockheed Martin Space Systems Company Palo Alto, CA

2. Outline • Robust DyMND Engineering • DyMNDSim and DyMNDES: The first generation • Lessons learned • DyMND-EE: The next generation • Architecture • Implementation • Experiments • Adaptive Coordination in DyMND • Explicit local coordination for optimal control • Dynamic programming (reinforcement learning) • Design and validation strategy within DyMND-EE • Summary and Future Plan

3. Client Proxy Simulation Director Motes Radio Effectors Sensors Target models Active message handler SvcCoord TaskQ DyMNDSim server DyMNDSim and DyMNDES – The First Generation • DyMNDSim • Discrete Event Simulation • Full-featured tool for implementing sensing/radio models • Integration with Motes and Tossim • DyMNDES • Real-time injection and data retrieval • Scenario driven interaction • Script, save, and load emulation event sequences • Generation of abstract mote and target capabilities via virtual objects • 3D visualization of aggregate state

4. Lessons Learned • Translation of services from DyMNDSim to motes required manual recoding • DyMNDSim emulated TinyOS active messages and event/task distinction, but executed Java classes • Hardware nodes led to noisy, non-repeatable experiments • Debugging and monitoring required source-code modifications • Principled integration of simulation and emulation components required for • Accurate validation of NEST systems

5. DyMND-EE – The Next Generation • Goal: End-to-end validation of sensor net software • Efficient execution of embedded source code • Realistic models for radio, sensing, actuation • Mixed models of execution for staged development • Goal: Accurate validation environment • Model of execution (concurrency) faithful to TinyOS • Goal: Scale to NEST challenges (e.g. EXSCAL) • Exploit tiered structuring via aggregate simulation and interaction across aggregates

6. Tossim Emsim / Emstar Processor fidelity DyMNDEE DyMNDSim Ptolemy II / VisualSense Generic modeling Simulink Evolving DyMNDSim To Realize DyMND-EE Goals Based on principled composition of existing simulators • Target-level simulators • Emsim (Emstar): cross-compiled nesC in Linux user space • High-level DE/ continuous-time simulators • Ptolemy II/VisualSense: Multiple models of computation, actor-oriented programming • Simulink: Mostly continuous time, rich control/DSP libraries

7. Ptolemy II Features in DyMND-EE • Actor-oriented development environment • Actors implement operations on data • Directors specify model of computation (timing and order of communication between actors) • Multiple models of computation in a designed system • Example: discrete-event sensor model to track evader with continuous control law • Extensions to WirelessDirector (implicit connectivity) • ComposableChannel allows modification of each token • Channel modifiers for propagation delay, deterministic / probabilistic message loss, bit errors • Extension to create distributed director

8. EmStar Features in DyMND-EE • Emstar is a flexible software framework for heterogenous NesC Code simulation using the FUSD kernel module. (http://cvs.cens.ucla.edu/emstar) • Features • Executes TinyOS/NesC code via EmTOS • HIL and simulated execution • Heterogeneous node simulations • Can use real or simulated RF channel • FUSD user space devices • Non-TinyOS dependent as any device can be registered under FUSD • Provides a clean interface to abstract hardware.

9. DyMND-EE Concept of Operations Graphical representation of a configuration of the DyMND-EE simulation environment Data Logging to Files for Analysis DyMNDES generates simulation configuration and generates Ptolemy Models. Ptolemy Executes aggregate sensing and comm. models EmStar runs NesC code

10. DyMND-EE Architecture Ptolemy II • Decomposes simulation into sensing/actuation “universes” • Implemented as composite actors with directors • Radio and sensors as actors • Services as Emstar processes with proxy actors • Software services execute in Emstar (native code) or Ptolemy (prototype code) • Communicates with external processes via XML/object streams • Configurations generated from DES script using XSLT and predefined Ptolemy II actors Acoustic Universe Processors Radio Universe Node 1 Microphone Node 1 EmstarProxy Node 1 Radio Magnetometer Universe Node 2 EmstarProxy Node 2 Radio Node n Microphone Node n EmstarProxy Node n Radio Emstar Gateway Emlink Radio Channel Abstraction Radio Channel Abstraction Sensing Channel Abstractions (EmSensorServer) FUSD Emstar (Linux user space) Radio Channel Abstraction Radio Channel Abstraction Radio Channel Abstraction Node 1 Processes Node 2 Processes Node n Processes

11. DyMND-EE Component Collaboration Ptolemy II Acoustic Universe Processors Radio Universe • Target emits chirp • WirelessDirector transmits chirp to node k mic • Mic processes chirp • Ptolemy forwards ActiveMessage (AM) to node k EmstarProxy • Proxy routes AM to Emstar gateway • Gateway routes AM to EmSensorServer • EmSensorServer forwards AM to node k processes via FUSD • Node k process generates detection message • Node k process forwards radio AM to radio abstraction via FUSD • Radio abstraction forwards AM to Emstar gateway • Node k EmstarProxy forwards AM to node k radio • Node k radio broadcasts alert Node 1 Mic Target Chirp Node 1 EmstarProxy Node 1 Radio 1 12 2 Node k Mic Node k EmstarProxy Node k Radio 3 4 Node n Mic Node n EmstarProxy Node n Radio 5 11 Emstar Gateway 10 6 Emlink Radio Channel Abstraction Radio Channel Abstraction Sensing Channel Abstractions (EmSensorServer) 9 7 Emstar (Linux user space) Radio Channel Abstraction Radio Channel Abstraction Radio Channel Abstraction Node 1 Processes Node k Processes Node n Processes 8

12. Outline • Robust DyMND Engineering • DyMND-Sim and DyMNDES: The first generation • Lessons learned • DyMND-EE: The next generation • Architecture • Implementation • Staged Development Configuration • Experiments • Adaptive Coordination in DyMND • Explicit local coordination for optimal control • Dynamic programming (reinforcement learning) • Design and validation strategy within DyMND-EE • Summary and Future Plan

13. Implementing DyMND-EE: Extensions To Emstar • Primary Modifications • Interfaces into TinyOS scheduler for fine grained control of timing and interrupt execution • New sim-component EmLink implements radio and sensing link layer driver registration for interface into Ptolemy • Sensing components in Emstar component library are rewritten to use new ADC.nc components that register link users. No changes are required to get NesC code working as expected in DyMND-EE. • Additional components, including hypothetical components, are emulated to examine effects of sensor networks with actuation capabilities via Ptolemy environment • Extensions for non-TinyOS-based node synchronization with external simulations

14. Implementing DyMND-EE: Emstar-Ptolemy Link • EmLink • Encapsulates TinyOS ActiveMessages as Ptolemy tokens • EmSensorServer implements Linux user-space proxy for Ptolemy sensor/radio models • P-Star protocol defines send/receive, device read/write, synchronization, sense/actuate, and configuration messages • Communication between Ptolemy-space and user space transparent to services and actors • P-Star Protocol • Protocol between Ptolemy and EmStar • Header format includes simulation time, device ids, group ids (for spatial or logical partitioning of EmStar simulations), and various simulation options. • EmStar Java Gateway • Consumes P-Star messages and dispatches them to EmstarProxy actors • Actors can represent entire nodes, components of a node, node tasks and services.

15. DyMND-EE Configurations Configuration Options • Modeling the physical environment • Radio Channel models • Sensor channel models • Modeling component interactions • Service coordination • Timing & interrupts • Sensing & actuation

16. Matlab Minimal Staged Development in DyMND-EE • Matlab – processing • Ptolemy – distributed radio and sensing model

17. Staged Development in DyMND-EE (Contd.) • Introducing NesC code into the simulations

18. Staged Development in DyMND-EE (Contd.) • Replacing Ptolemy models with actual environment • Testing performance in real world

19. Staged Development in DyMND-EE (Contd.) • Transfer code to actual hardware and test like you fly

20. Outline • Robust DyMND Engineering • DyMND-Sim and DyMNDES: The first generation • Lessons learned • DyMND-EE: The next generation • Architecture • Implementation • Staged Development Configuration • Experiments • Adaptive Coordination in DyMND • Explicit local coordination for optimal control • Dynamic programming (reinforcement learning) • Design and validation strategy within DyMND-EE • Summary and Future Plan

21. DyMND-EE Experiment 1: Target Detection • Simple translation of wireless sound detection Ptolemy demo (UCB) to nesC • Simulation setup models radio and sensing channels • NesC code performs triangulation

22. Target Detection Scenarios

23. Experiment Configurations • Experiment parameters are easily changed in Ptolemy. • Experiment parameters can be set either by the DyMNDES or by the Vergil GUI in Ptolemy • Experiment runs can be saved in Ptolemy or as a script by the DyMNDES.

24. Source Localization Results Limited radio range More aggregators; modified density of detectors Increased target speed; limited radio ranges Increased target speed

25. Issues • Insight from these simple experiments demonstrate that its easy to find the optimal simulation configuration to give data that “fits” your algorithm. Results from the perfect simulation configuration

26. DyMND-EE Experiment 2: Localization • Kestrel localization code • No changes necessary in NesC code • Simulation setup models radio and sensing channels. • NesC code performs localization

27. Localization Scenarios

28. Comparing Fidelity of Simulation • DyMND-EE results match Kestrel’s results (TOSSIM w/custom sensor models) • Confirmed that localization produced standard deviation ~0.7-1.0% of total distance for accurate ranging

29. Advantages of the DyMND-EE Approach • Software reuse and rapid prototyping • Reuse/test embedded code • Implement software prototypes as Ptolemy actors or Linux user-space processes (managed memory, out-of-band communications) • Implement new sensing concepts as Ptolemy actors • Extensibility • Simulation container not restricted to single hardware model • Scalability • Functional decomposition allows tradeoffs between spatial locality (neighboring nodes) and functional locality (sensors vs. radios vs. processors) for distributed implementation • Synchronization • Ptolemy messaging bus synchronizes events and communication between services

30. Limitations • Limitations • Task-level simulator executes in a single thread per node. Timing and interrupts may not be handled as expected if using interrupts directly • NesC component libraries only partially ported – UART, ADC, and actuation not yet completed. • Radio and sensor models still under development and require compilation for new models • Each node runs C or NesC code

31. Outline • Robust DyMND Engineering • DyMND-Sim and DyMNDES: The first generation • Lessons learned • DyMND-EE: The next generation • Architecture • Implementation • Staged Development Configuration • Experiments • Adaptive Coordination in DyMND • Explicit local coordination for optimal control • Dynamic programming (reinforcement learning) • Design and validation strategy within DyMND-EE • Summary and Future Plan

32. Optimal control: The DyMND Approach • Management of sensor nets can be framed as an optimal control problem • Optimal policy maximizes value function: Expected (discounted) sum of future rewards • Engineer specifies rewards/penalties for good/bad events • Actions rewarded/penalized per node (no global oracle) • Objective function depends on … • Quality of sensor data (higher is better) • Power consumption (lower is better) • Detection performance (precision & recall) … over the life of the network • Large sensor nets require global optimization over (large numbers of) local state variables • Low communications overhead (packet size, power management) • Low memory overhead (!)

33. Optimal control: An example Intruder detection with wireless sensors • State variables (per mote) • Internal state: e.g., mote battery power, current detection assignment • Observed state: e.g., amplitude and frequency of acoustic sample • Actions (per mote) • Changing sensor sampling rates • Sending a detection message • Switching between sensing “policies” (e.g. “vigilant” / “slack”) • Rewards/penalties (per mote) • Penalty proportional to power consumption: Encourages long network life • Reward proportional to “quality” of sensor coverage Variance of observations over mote and neighbors

34. Sampling rate Lo Hi Junction Tree RL Coordination Service Initializing a junction tree • Identify neighbors and shared state • Build routing tree and propagate shared state Making decisions (policy improvement) • Pass messages to ascertain neighbors’ policies • Determine the optimal policy conditioned on • Neighbors’ policies • Current local value function estimate • Pass state observations in toward root, then out toward leaves Updating value estimates (policy evaluation) • New estimate is convex combination of: • Old estimate • New observation of (rewards + successor-state expected value)

35. Loopy value propagation • Junction-tree control imposes routing tree on communications mesh • Worst-case path length  radius of network • Costly to maintain running intersection property (storage exponential in path length) for large networks • Expensive recovery from link/node loss to maintain tree • Analogy to probabilistic inference • Junction-tree RL relies on optimal-control form of Generalized Distributive Law (GDL) • Statistical-mechanics approximation to GDL assumes interaction graphs have cycles (atoms in a crystal lattice) • Junction graph relaxes global tree structure • Tree-structured coordination graph per variable • Natural coordination structure for sensor nets (immediate neighborhoods) • Introduces circular dependencies to decision-making process • Decisions are globally suboptimal, but locally optimal (and approach optimality after many iterations) • Standard JTRL: • 5 hops • 2 extra nodes • 3 hops • 1 extra node • Loopy JTRL: • 2 hops • 2 hops • 0 extra nodes! Junction-tree RL makes MDPs feasible for small sensor nets; Loopy junction-tree RL makes approximate MDPs feasible for large sensor nets!

36. RL in TinyOS: Design Constraints • RL implementation requires three components • Exploration strategy to balance “exploration” (visiting new states) and “exploitation” (returning to old states) • Exploration can occur in simulation/lab • Deployed system will favor exploitation • Value function approximator to identify best actions for states • Initial values biased to incorporate domain knowledge • Exact (tabular) value functions are storage-intensive • Nonlinear approximators (neural nets) are compute-intensive • Linear combinations of features are efficient • Update rule to apply changes to value function/policy Minimal computational overhead • RL service requires access to service state • Messages between nodes should be small and infrequent • Loopy value propagation avoids long message paths • Coarse-grained policy learning reduces complexity of interaction

37. RL in TinyOS: Implementation Options • Runtime configuration: RL-aware services register callback with RL service • Registered services provide accessors for relevant state/action or • Registered services fire events for state changes, and RL service fires events for policy changes • Similar to ServiceC/ServiceM approach • Requires retrofit of existing services • Compile-time configuration: RL service code generated with static references to ordinary nesC services • Exploits static allocation: RL service directly reads state variables, writes configuration parameters • Reuses services without modification • Violates isolation of nesC services • “Shared memory” hack risks race conditions

38. Summary • Developed DyMND-EE for • End-to-end validation of NEST systems • Staged development of NEST systems • Robust analysis via controlled experiments within an accurate validation environment • Experimented with DyMND-EE • Replicated experiments to validate claims (localization) • Design of adaptive mechanisms for optimal performance in the context of change • Developed algorithms • API design for implementing within DyMND-EE

39. Future Plan • Extensions to DyMND-EE • Include actuation processes • Interruptable tasks • Higher fidelity sensor, radio models • Experimental approach to design/evaluation of robust sensing/actuator networks • Modeling (sensor, radio, actuation) in DyMND-EE • Experiment driven design of services (parameters) for specific application contexts and dynamics • Adaptive coordination implementation