EE206A - Spring 2005 Lecture #4: Emstar

1. EE206A - Spring 2005Lecture #4: Emstar Martin Lukac Andrew Parker Nithya Ramanathan EE206A

2. What is EmStar? • Software environment for sensor networks built from Linux-class devices (microservers) • Such as Intel StarGate • Still embedded and distributed, but much less constrained than motes with more complex processing EE206A

3. What does EmStar Provide? CollaborativeSensor ProcessingApplication Client Server State Sync 3d Multi- Lateration /dev/servicename /dev/fusd Topology Discovery Acoustic Ranging Neighbor Discovery Leader Election Reliable Unicast Time Sync kfusd.o Hardware Radio Audio Sensors • Modular, but not strictly layered, architecture • graph of software modules with well-defined interfaces • basic modules are Linux processes • FUSD for inter-module communication via Linux’s device (/dev/…) system • event-driven structuree • asynchronous notification among modules • library of standard reusable services • e.g. routing, neighbour list, timesync… • Library of “device patterns” • Encapsulate common patterns of inter-module interactions e.g. “status”, “packet queue” etc. • Robust, autonomous operation • fault tolerance within node and between nodes • start, stop, monitor, and respawn services • Support for tiered sensor networks • Microservers (Emstart) + motes (TinyOS) • High visibility into system • logging and visualization for debugging • Tools • EmRun, EmView,EmSim, EmTOS etc. EE206A

4. Key Capability: Simulation, Emulation, and Deployment with Identical Code! EE206A

5. Reading List for this Lecture • L. Girod, T. Stathopoulos, N. Ramanathan, J. Elson, D. Estrin, E. Osterweil, and T. Schoellhammer, "A System for Simulation, Emulation, and Deployment of Heterogeneous Sensor Networks", in the Proceedings of the ACM SenSys, November 2004. • http://nesl.ee.ucla.edu/courses/ee206a/2005s/papers/L04/Girod04_SenSys.pdf • L. Girod, J. Elson, A. Cerpa, T. Stathopoulos, N. Ramanathan, D. Estrin, "EmStar: a Software Environment for Developing and Deploying Wireless Sensor Networks", in the Proceedings of USENIX General Track, 2004. • http://nesl.ee.ucla.edu/courses/ee206a/2005s/papers/L04/Girod04_USENIX.pdf EE206A

6. Rest of this Lecture… How to write sensor network applications using Emstar? EE206A

7. Audio Server Reliable State Sync “bus” types/orient types/ranges types/orient types/orient types/ranges mlatd ar/mlat orient_bcast Audio Data ar/request_chirp ar_recv orient/pub_seq orient/status ar_send audiod Acoustic Signal orientd audio/[02]/sync_bin Chirp Notification syncd Conversion Chirp Notification Conversion Flooding with hop by hop time conversion Software Module Output Device Message/Device File Distributed Module EE206A

8. Emstar basics • Each node is a collection of processes • Processes are server or clients or both • The processes communicate through device files • Each process maintains its own soft state • Soft state provides basic form of reliability and robustness • All processes are event based: no polling or blocking • Processes start and wait for an event OR • Processes start and set a timer to trigger an event • Same code runs in simulation and deployment! EE206A

9. An EmStar Node /dev/link/ls0/linkstats linkstats /dev/link/udp0/data udp neighbor-app /dev/link/ls0/neighbors neighbord EE206A

10. An EmStar Simulation node004/link/ls0/linkstats node002/link/ls0/linkstats node003/link/ls0/linkstats node001/link/ls0/linkstats linkstats linkstats linkstats linkstats Node 1 Node 4 Node 2 Node 3 neighbor-app neighbor-app neighbor-app neighbor-app node004/link/ls0/neighbors node002/link/ls0/neighbors node003/link/ls0/neighbors node001/link/ls0/neighbors neighbord neighbord neighbord neighbord udp udp udp udp node003/link/udp0/data node004/link/udp0/data node002/link/udp0/data node001/link/udp0/data sim_radio EE206A

11. EmStar is Event Based • Your processes sit and wait for something to happen: receive packet, timer fire, etc… Event Loop dispatches Callback1 (Data, Context_info) Callback2 (Data, Context_info) Callback3 (Data, Context_info) Callback4 (Data, Context_info) Read Write Timer Packet EE206A

12. Event Driven Interface Server Process query_process_cb(input_buf, output_buf) { … memcpy(output_buf, result, y) … return …; } Client Process funct_A(…) { … // this is non-blocking query_client_issue(input_buf) … } query_response_cb(output_buf) { … } triggers triggers EE206A

13. Devices for IPC Client Server /dev/servicename /dev/fusd kfusd.o • Event driven • Blocking functionality provided by synchronous function calls ending in “_s” • Server-Client model / Services • Server exports a device(s)/service • Client writes/reads data/uses service • Communication enabled by FUSD • Devices accessed through • Ascii: command-line (either using cat or bin/echocat) • Ascii/Binary: Client libraries (all devices provide a client library for program access) EE206A

14. Status Device • No message queueing • Provides current status (binary / printable) • Getting status • Status-Device process exports a device file • Status-Clients process opens this device file • Status-Client reads from fd • Status-Device process is informed of read and sends data (ascii / binary) to Status-Client through FUSD • Status-Device “notifies” clients when new updates are available (i.e. fd is readable) … => goto Step 3 • Demo (command device, status device) – obj.i686-linux/libdev/examples/simple_status EE206A

15. Server Fill opts struct & register device Open cb called if specified, used to track client-related state Write cb called if specified, command is processed Notify() client when data is ready Provide data in binary/printable Client Fill opts struct & call open() (Optional) Write string to device Read device when notified Status Devices EE206A

16. Packet Device • Provides message queuing so clients do not lose information – useful for high event rates. • Useful for exchanging packets between modules • Getting packets • Packet-Device process exports a device file • Packet-Clients process opens this device file • Clients can add filters to specify packets they are interested in 3. Packet-Device process is informed of read and sends data (ascii / binary) to Packet-Client through FUSD …. • Printable => un-parse • Demo (Status => Debug, Usage) – obj.i686-linux/libdev/examples/simple_pd EE206A

17. Server Fill opts struct & call New() Open cb called if specified, used to track client-related state Enqueue cb called if specified, command is processed Send cb called (MUST BE IMPLEMENTED) -> calls pd_receive() Pd_receive sends data to clients Provide data in binary/printable (unparse cb if provided) Client Fill opts struct & call open() Write packet to device Read device when notified Packet Devices EE206A

18. Link Devices • Built on top of Packet-Device • Link servers (or providers) fill out lp_opts_t struct and call lp_register() • Link-Clients • Use the link-device to send packets to specified destinations (including broadcast) • Specify filter to only receive certain packet types • Skeletons/samples/read_packets.c • Devices automatically created by lp_register(): • /dev/node<id>/link/<link-name>/command • /dev/node<id>/link/<link-name>/data • /dev/node<id>/link/<link-name>/status EE206A

19. Server Fill opts struct & call New() Open cb called if specified, used to track client-related state Filter cb (called by pd_enqueue) called if specified, packet is accepted/not accepted Send cb called (MUST BE IMPLEMENTED), sends data to clients Provide data in binary/printable (unparse cb if provided) Client Fill opts struct, set filter to specify types of packets to receive, & call open() Call lu_send() to send a packet Receive cb (MUST BE SPECIFIED) is called with packets that match filter Link Devices EE206A

20. Common components /dev/link/ls0/linkstats linkstats /dev/link/mote0/data /dev/mote/0/data /dev/link/udp0/data motenic hostmote udp neighbor-app • Example: • neighbor-app prints total neighbors • neighbord translates linkstats output into neighbor list • linkstats talks with linkstats on other nodes to evaluate links • udp provides link device to ethernet/wireless network • What device patterns are used here? • Code is in: neighbor-app: link/examples/neighbors neighbord: link/neighbors linkstats: link/linkstats udp: link/udp /dev/link/ls0/neighbors neighbord EE206A

21. Demo of common components and interfaces EE206A

22. Anatomy of an EmStar Program • Based on emstar/skeletons/samples/query_example.c #includeFOO #defineBAR struct state { /* device context pointers */ }; int init_funct(…){…}; int helper_funct(…){…}; int callback1(…){…}; int callback2(…){…}; void usage(…){…}; void shutdown(…){…} int main(int argc, char * argv[]) {…}; Wow! EE206A

23. Anatomy of main intmain(intargc, char * argv[]) { init_state(); /* set-up emrun options */ … misc_init(…) /* process command-line arguments */ … /* create devices, clients, timers */ … emrun_init(…); g_main(); // Enter event loop! /* Should never get here! Log error! */ … return 1; // return error } EE206A

24. Global State Structure • Global state should go into a single structure • EmStar callbacks usually take a void* as one of the arguments, or a device_context *,either of which will let you gain access to the global state structure 86 typedef structsample_state { 87 query_context_t *query_ref; /* a reference to our query device */ 88 g_event_t *timer_ref; /* a timer for delayed responses */ 89 char *reply; /* will report this text */ 90 intdelay; /* will delay before returning */ 91 } sample_state_t; 92 EE206A

25. Query Device Callback: Process • We can return either ASCII or binary data • Don’t forget to return QUERY_DONE! • 134 intsample_process(query_context_t *q, // All kinds of useful context • char *command, // Data from caller • size_t buf_size, // Size of command • buf_t *print, // Our ASCII response • buf_t *bin) // Our binary response • 136 { • // Example of how you can get at the global state struct • sample_state_t *state = (sample_state_t *) qdev_data(q); • … // Parse command and do what you need • … // Note that we are writing our response into the variable “print” • 206 bufprintf(print, "Immediate response: '%s'\nUptime is %d seconds\n", • state->reply ? state->reply : "No reply requested", • (int)(misc_time_elapsed()/MILLION_I)); • … • 213 return QUERY_DONE; // Indicates that the transaction is complete • } EE206A

26. Query Device Creation in main() • emstar/skeletons/samples/query_example.c line 304 • Note the syntax used to initialize the structures… int main(…) { … { query_dev_opts_t q_opts = { device: { devname: SAMPLE_QUERY_DEV, /* This is a char*: “/dev/samples/query” */ device_info: &state /* Pointer to our global state */ }, process: sample_process, /* Pointer to function */ usage: sample_usage /* Pointer to function */ }; // This actually creates the query device and returns a pointer to it, if (query_dev_new(&q_opts, &(state.query_ref)) < 0) { elog(LOG_CRIT, "Unable to create query device: %m"); exit(1); } } … } EE206A

27. buf_t • emstar/libmisc/include/misc_buf.h line 88 • A "buf_t" is a structure that makes it easy to construct a growing buffer of data, without worrying about sizing it correctly. • Initializing and freeing: • Heap: buf_new(…) and buf_free(…) • Stack: buf_init(…) • Writing: • Similar to sprintf : bufprintf(buf * b, …); • bufprintf(my_buf, "Immediate response: '%s'\nUptime is %d seconds\n", …); EE206A

28. elog(…) • emstar/libmisc/include/elog.h line 50 • Similar to printf: • elog(LOG_ALERT, “File name is %s. Something went wrong: %m”, file_name); • Anyone know what %m is? • Prints out time_stamp, node id, process, function, message • Fri Oct 29 10:02:21.517 [5] trig: helper: Current state is 10 • Many different log levels: • LOG_ALERT, LOG_CRIT, LOG_WARNING, LOG_NOTICE, etc. EE206A

29. Compiling Using EasyBuild • EasyBuild docs http://cvs.cens.ucla.edu/emstar/ref/make.html • You type “make” only in one place, at the EmStar root. • If you add a new top-level directory, like emstar/Assignment1, then you’ll want to edit emstar/BUILD and add “Assignment1” to the include list: • include { fusd, libmisc, … , Assignment1 } • Then emstar/Assignment1/BUILD might look something like: build bins { local_libs { link/link, emview/emview, emrun/emrun, libmisc/misc, libdev/dev, fusd/fusd } cflags := ‘pkg-config -–cflags gtk+-2.0 gdk-2.0 atk` target cam_xy { cam_xy.c } } EE206A

30. Great. Now where is my binary? • Normal builds leave a huge mess inside your source tree, and “make clean” never seems to quite work. • EasyBuild implements an “out of tree build” • The results of the compilation process (binaries, object files, libraries, etc.) are all placed in a parallel tree structure • This leaves your source tree clean • Simplifies building for multiple architectures simultaneously • From the previous slide, if we compiled for i686-linux (the default), then the target “cam_xy” would be found here: • emstar/obj.i686-linux/Assignment0/cam_xy EE206A

31. emrun • Emrun: process manager for a node • Starts, stops, and manages processes • Collects log messages from processes • Presented to interactive client as in memory log ring • Various runtime configurable log levels available • (you should use elog() in your code, not printf()!!!) • Sets up control channel to each processes to monitor health • emrun_init() opens control channel back to emrun and starts heartbeat • Respawns dead or stuck processes • Initiates graceful shutdown • Receives notification when starting up that initialization is complete EE206A

32. emrun devices • /dev/emrun/status • Reports the current state of all processes managed by emrun • /dev/emrun/command • Command interface for controlling emrun • /dev/emrun/last_msg • Lists the last time each process was terminated and why, as well as the last log message before death • /dev/emlog/<process>/... • Per-process in-memory log rings • The -f versions can be used to stream the continuous log to a file with cat (or just use tail -f on non -f version) • The non -f versions dump recent messages. • /dev/emrun/emview_config • Reports configuration information used to automatically configure the EmView visualizer EE206A

33. emrun example /dev/link/ls0/linkstats linkstats /dev/link/udp0/data udp neighbor-app • A cool example: • neighbor-app prints total neighbors • neighbord translates linkstats output into neighbor list • linkstats talks with linkstats on other nodes to evaluate links • udp provides link device to ethernet/wireless network • What device patterns are used here? • Code is in: neighbor-app: link/examples/neighbors neighbord: link/neighbors linkstats: link/linkstats udp: link/udp /dev/link/ls0/neighbors neighbord http://cvs.cens.ucla.edu/emstar/ref/run.html EE206A

34. Process block: .run files • .run config files • Specify how the EmStar services are "wired" together • Specifies dependency order for processes • Allows emrun to maximize parallelism for startup Key value pairs: • Required: type: once, daemon, boot cmd: the command to run • Optional: waitfor: specifies a dependency nice: set the “nice” level for the process loglevel: set the default loglevel no-core: don't create a core file on crash no-sim: only run when not in sim mode sim-only: only run in sim mode process <tag> { <key> = <value>; ... } EE206A

35. Let’s build a .run file! /dev/link/ls0/linkstats linkstats /dev/link/udp0/data udp • WARNING! .run files typically assume you are running from the obj directory! Build a run file for ---> neighbor-app /dev/link/ls0/neighbors neighbord EE206A

36. emrun macros save the day include link/link.run &link_udp(udp0); &link_linkstats(udp0,ls0); &link_neighbors(ls0); &link_blacklist(ls0,bl0,ls0); &link_frag(bl0,frag0); &link_pingd(frag0); • Macros exist for most of the common components and services • Easiest to understand by looking at examples • Lets look at examples: • link/include/link.run • mote/include/mote.run include link/link.run include routing/routing.run &link_udp(udp0,class="raw"); &link_linkstats(udp0,ls0); &link_neighbors(ls0); &routing_flood(ls0); &link_pingd(flood); EE206A

37. emsim • emsim enables multiple nodes on one machine • Start, stops, and manages emrun's and sim-components • Requires SIM_GROUP environment var to be set • How do the processes know which group and node? • emsim sets environment variables, misc_init() reads • accessible in code as my_node_id after misc_init() • In reality, emsim and emrun are almost the same code • emsim has its own configuration file • Note: Always call misc_init() and emrun_init() in all your emstar code!!!! EE206A

38. emsim devices • /dev/sim/group#/config • Reports position, config file, and on/off status of each node • /dev/sim/group#/emrun/status • Reports the current state of all processes managed by emsim • /dev/sim/group#/emrun/command • Command interface for controlling emsim • /dev/sim/group#/emrun/last_msg • Lists the last time each process was terminated and why, as well as the last log message before death • /dev/sim/group#/emlog/<process>/... • Per-process in-memory log rings • The -f versions can be used to stream the continuous log to a file with cat (or just use tail -f on non -f version) • The non -f versions dump recent messages • For emsim, look at the sim-component-# EE206A

39. sim-components • sim-radio • simulated radio interface • simulated radio channel • nodes use to communicate to each other • Radios • HW: mica1, mica2, 802.11 • MACs: BMAC, SMAC, 802.11 • Channel Models • some simple models, and models based of off empirical data • see sim/libchannel/ • Can feed in topo file with fixed model • Run sim_radio -h to see all the options • Also see status devices in /dev/sim/group#/radio/ • udp discovers it is in sim mode and register with sim-radio EE206A

40. emsim example node004/link/ls0/linkstats node002/link/ls0/linkstats node003/link/ls0/linkstats node001/link/ls0/linkstats linkstats linkstats linkstats linkstats Node 1 Node 4 Node 2 Node 3 neighbor-app neighbor-app neighbor-app neighbor-app node004/link/ls0/neighbors node002/link/ls0/neighbors node003/link/ls0/neighbors node001/link/ls0/neighbors neighbord neighbord neighbord neighbord udp udp udp udp node003/link/udp0/data node004/link/udp0/data node002/link/udp0/data node001/link/udp0/data sim_radio EE206A

41. .sim files Node Block: • .sim files specify • sim components • number of nodes and field size • .run files for each nodes • position of nodes • default on/off node <id> { <key> = <value>; ... } <id> values: default: default for all nodes #: for a specific node #-#: for a range of nodes Required: num-nodes: usually passed in through command line field-size: (max x, max y) sim-component: sim-radio command Key value pairs: position: (x, y) emruntab: specify the emrun tab power: on or off http://cvs.cens.ucla.edu/emstar/tut/emsim-advanced.html EE206A

42. Lets build a .sim file! • WARNING! Most .run scripts assume you run from the obj directory Build a .sim file for multiple neighbor_apps. Most of your sim files will look very much like this. EE206A

43. emproxy and emview • emproxy exposes status devices to network • Used by emview for visualization • emview uses exposed status devices to visualize network • Sometimes command line is not expressive enough • emview typically 'just works' for existing services • emview sends udp broadcasts to locate hosts running emproxy • emrun/emview_config contains information for emview • emview can load modules based on information from emview_config emview demo! EE206A

44. The Debugging Tutorial • There’s debugging tutorial online (link on next slide) • Covers basics of log devices • Tracking down problems in callbacks • Using GDB with emstar EE206A

45. References • Docs & Tutorials – http://cvs.cens.ucla.edu/emstar • CVS & LXR – http://cvs.cens.ucla.edu • Mailing list archives - http://www.cens.ucla.edu/mailman/listinfo/emstar-users EE206A