A Dynamically Reconfigurable Data Stream Processing System

1. A Dynamically Reconfigurable Data Stream Processing System Jerzy M. Nogiec Fermilab

2. Outline Dynamism in Software Systems …..… EMS Framework ………………………. Reconfiguration in EMS ………………. Dynamism using Scripting ………..….. Wrap-Up ……………..………….……… A Dynamically Reconfigurable Data Stream Processing System

3. dynamism Dynamism in Software Systems • Open dynamic system Changes at runtime that were not anticipated at build time. • Closed dynamic system Configuration changes performed at runtime, but planned at build time. • Tailorable system Some aspects of the system can be modified at runtime. • Configurable system Many different configurations can be set up. • Static system System behavior fully described at build time. A Dynamically Reconfigurable Data Stream Processing System

4. Dynamic Reconfiguration Dynamic reconfiguration is a process of modifying/evolving a running application, in which a composition of interacting components changes at runtime. It could be used to: • Modify behavior of the system why preserving continuity of its execution (mission or safety critical systems) • Apply patches to a running system • Extend base application functionality with additional nonfunctional services (end user customization and extension) • Adapt to changing requirements or environment that are difficult to anticipate (dynamically optimizing system functionality or recovering from errors and failures). A Dynamically Reconfigurable Data Stream Processing System

5. EMS Framework EMS is a component-based framework for building measurement, data acquisition, and data stream processing systems. The goal of the EMS project was to design, implement, and deploy a system that is extensible, flexible and dynamic. EMS can be used to develop configurable and dynamically reconfigurable systems. *Framework – the skeleton of an application that can be customized and reused **Component – an independently released software module suitable for composition (together with other components) into multiple applications A Dynamically Reconfigurable Data Stream Processing System

6. Visual Components A Dynamically Reconfigurable Data Stream Processing System

7. Component Interfaces Components have properties and state. Typical components input, process, and output data. Their behavior depends on their state and property values. Components can be forced to perform certain actions by sending control events to them. Components also output debug and exception information. control • properties • state • error status data data exception debug A Dynamically Reconfigurable Data Stream Processing System

8. filter splitter source merger sink Component Connectivity • There are several categories of data processing components: • Data sinks • Data sources • Data filters/processors • Data splitters • Data mergers A Dynamically Reconfigurable Data Stream Processing System

9. Connecting Components • EMS provides for independent “wiring” of components for property, data, exception, control, and debug events. • EMS supports unicast, multicast, and broadcast patterns. • Source routing (dynamic, run-time) & routing tables (static, configuration-time). A Dynamically Reconfigurable Data Stream Processing System

10. EMS XML Dialect <configuration version="0.1" title="Display Test XML"> <!-- Component definitions --> <component id="Producer” class="ems.core.components.SimpleDataGenerator"> <property name="title" value="Data Generator Component"/> </component> <component id="Chart" class="ems.measurement.chart.ChartDataDisplay"> <property name="XPosition" value="0"/> <property name="YPosition" value="200"/> <property name="title" value="Plot Display"/> </component> <!-- Routing information --> <route type="Data" origin="Producer" destination="Chart" /> <!-- Control signals --> <control signal="init" destination="!"/> <control signal="start" destination="!"/> </configuration> Producer Chart A Dynamically Reconfigurable Data Stream Processing System

11. Reconfiguration in EMS There is several techniques that can be employed to alter the behavior of the system: • Use of component properties that are modifiable also at runtime • Definition and modification of event paths • Deactivation/activation of components • Deactivation/activation of data paths using gate components • “Coloring” data in data driven processing • Addition/removal of components • Use of externally controllable source routing A Dynamically Reconfigurable Data Stream Processing System

12. Reconfiguration in EMS: Modification of Component Properties Each component has a set of properties that both control its behavior and exhibit its state. These properties can be introspected and modified externally at runtime. A Dynamically Reconfigurable Data Stream Processing System

13. Reconfiguration in EMS: Modification of Data Paths Data paths between components can be manipulated (added, modified or removed) at runtime. A Dynamically Reconfigurable Data Stream Processing System

14. Reconfiguration in EMS: Deactivation/Activation of Components A component can be made inactive, so it passes the data but does not perform any processing. A Dynamically Reconfigurable Data Stream Processing System

15. Reconfiguration in EMS: Gate Components A gate component passes or stops all data traffic going through it. Therefore it effectively activates or deactivates the data path that originates at it. A Dynamically Reconfigurable Data Stream Processing System

16. Reconfiguration in EMS: Coloring Data A “coloring” component (Marker) is used to mark data. Other components act only on the data marked (“colored”) to be processed by them. A Dynamically Reconfigurable Data Stream Processing System

17. Reconfiguration in EMS: Adding/Removing Components A component can be added to or removed from the system. A Dynamically Reconfigurable Data Stream Processing System

18. Reconfiguration in EMS: Source Routing A component using source routing can send data to a recipient defined in one of its properties that can be manipulated by other (controlling) components. A Dynamically Reconfigurable Data Stream Processing System

19. Reconfiguration Components Separation of concerns between functional code and reconfiguration code can be achieved by introducing a dedicated reconfiguration component (-s). A Dynamically Reconfigurable Data Stream Processing System

20. Reconfiguration with Scripting • Scripting allows for automation of processing and for quick construction of new tests or data processing logic. • Scripting allows for easy implementation of reconfiguration in closed dynamic systems. The script control panel controlling execution of a Jython script. A Dynamically Reconfigurable Data Stream Processing System

21. WrapUp • EMS is a flexible component-based framework for building dynamic systems. • The system has been applied to building measurement and data acquisition systems where continuous data streams are processed in real-time. • EMS allows for high level of reuse. • The architecture supports separation of concerns between reconfiguration and functional aspects. • EMS features allow for agile development of new applications and easy adaptation of existing applications to changing requirements. A Dynamically Reconfigurable Data Stream Processing System

22. Resources EMS web site http://sdsg.fnal.gov/emsweb EMS on-line documentation http://wwwtsmtf.fnal.gov/ems_javadoc2.X A Dynamically Reconfigurable Data Stream Processing System