Self-Organization of Distributed Embedded Real-Time Systems – Conceptions and Architectures

1. Self-Organization of Distributed Embedded Real-Time Systems – Conceptions and Architectures Uwe Brinkschulte Dagstuhl Seminar Organic Computing – Controlled Emergence

2. Organic Computing Problem: Increasing complexity makes the development, maintenance and operation of distributed real-time systems more and more difficult Idea: Make those systems more „life-like“ => Organic Computing Realization of properties like: • Self-organization • Self-configuration • Self-optimization • Self-healing • Self-protection • Self-explanation • Self-awareness Self-X

3. Own Research Efforts on Organic Computing Microcontroller • Self-organization (control) of the real-time behavior of a multithreaded microcontroller Middleware • Self-organizing middleware for distributed embedded real-time systems • Mobile robots (SIMON) • Reconfigurable hardware (DoDOrg) • Sensor-actor-networks (SELINA, graduate school) • Systems On Chip (CARSoC)

4. Self-Organizing Middleware for Distributed Embedded Real-Time Systems • What does „middleware“ mean here? • Mediating software layer to simplify the development of distributed systems • Example „service oriented middleware “ Service F ServiceE Service D Service A Service B Service C … … … Middleware Middleware Middleware Middleware Computation Resource1 Computation Resource2 Computation Resource3 . . .

5. Self-Organizing Middleware for Distributed Embedded Real-Time Systems • How can middleware help with Organic Computing? • It can give self-x properties to distributed systems Optimize priorities, bandwidth, clock frequencies, ... Move services in case of resource failures Assign priorities, bandwidth, clock frequencies, ... Detect foreign services and jobs, e.g. by computer immunology • Self-Configuration • Self-Optimization • Self-Healing • Self-Protection Optimize service and job assignment Assign jobs to services Assign services to resources Service A Service C Service B Service D … … Job Middleware Computation Resource1 Computation Resource2 . . .

6. Appli-cation Service Appli-cation service … Middleware-Mikrokernel Extension-Services Basic-Services Self-Organizing Middleware for Distributed Embedded Real-Time Systems • Basic-middleware: OSA+ • Efficient real-time middleware for embedded systems with low resources (e.g. microcontrollers) • Service oriented middleware • Microkernel concept • small memory overhead • small runtime overhead • well-defined time behavior • scalability • dynamic real-time reconfiguration • (OSA+: Open System Architecture – Platform for Universal Services) (kernel size 28 kBytes, with all basic services 44 kBytes) (typ. < 10% compared to a native solution) (stdev < 3 on TimeSys RTOS, stdev = 1 on Komodo)

7. Self-Organizing Middleware for Distributed Embedded Real-Time Systems Dynamic real-time reconfiguration Movement or replacement of services • Important times: • Reconfiguration timetreconf Time from start to end of reconfiguration • Blackout timetblackout Time where the system does not react due to reconfiguration Service A Service C Service B Service Dneu Service D … … Middleware treconf Reconfiguration-Manager t Computation Resource1 Computation Resource2 . . . tblackout

8. Self-Organizing Middleware for Distributed Embedded Real-Time Systems • Realization: • Reconfiguration is triggered by a job to the reconfiguration manager concurring by the QoS-parameters (e.g. priority) with all other jobs in the system • More important jobs are done before, less important jobs after the reconfiguration  Real-time behavior • The reconfiguration manager moves or replaces services while the old service is still running  minimization of tblackout • tblackout is dominated by the time needed for state transfer • treconf calculates from tblackout + time for service transfer

9. Self-Organizing Middleware for Distributed Embedded Real-Time Systems Different self-organization techniques for services, jobs and operation parameters for the basic middleware OSA+ in different projects: SIMON: Evaluation of cost/benefit functions, decision trees DoDOrg: Exchange of messengers, hormones SELINA/GK: Forming of clusters, election of coordinators CAR-SoC: Decentralized control loops, multithreading

10. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project • Development environment for safe, self-organizing software with mobile components for factory automation Motivation: Mobile robots, flexible production, overall networks, increasing complexity produce demands:: • Improve software-flexibility  Self-Organization

11. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project Research goals: • Impact of self-organization on software engineering • Impact of self-organization on requirements like safety, security, real-time, fault tolerance, efficiency, user friendliness • Potentials and limits of self-organization for complex distributed applications with mobile components in factory automation Challenges: • Combining component-based software engineering with self-organizing software opens new territories • There are loads of open questions regarding emergence, confidentially, protection, time behavior, etc.

12. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project System architecture Development environment (Wörn) Application Component libraries (Wörn) Organic Management Monitoring (Brinkschulte) (Zitterbart) Self-organizing middleware (Brinkschulte) Basic-Middleware OSA+ Safe Communication (Zitterbart) Hardware: PDAs, Notebooks, Workstations….

13. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project Tasks of the organic management Job (e.g. driving job, grabbing job, visualization job, ...) determines, which job will be executed by which service Service (e.g. driving services, grabbing services, visualization services, ...) determines, which service will be executed on which resource Resource (e.g. robot, PDA, PC, ...)

14. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project The application determines, which services or service classes are suitable for a job The organic management selects one of these services to execute the job by evaluating cost/benefit functions

15. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project The application determines, which resources or resource classes are suitable for a service The organic management selects one of these resources to execute the service by evaluating cost/benefit functions

16. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project Additionally, the application can define dependencies: • Service dependencies: job X and Y must be executed by the same service • Resource dependencies: job X and Y must be executed by the same resource • Temporal dependencies: job X must be executed before job Y Execution path: a set of jobs, which must be executed in consecutive order by the same resource

17. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project The middleware receives from the application the execution paths of a mission Ex.:loading of palettes by robots, visualization of the loading process Execution path 1: Drive to stock, grab product 1, drive to palette, unload product 1 Execution path 2: Drive to stock, grab product 2, drive to palette, unload product 2 Execution path 3: Visualize robot positions Execution path 4: Visualize state of the stock .... The execution paths and elementary jobs are assigned to services and resources by the organic management by evaluating the cost/benefit functions Ex.:Execution path 1 Robot A, Drive to stock Service “Energy efficient driving”, ... Execution path 2 Robot B, Drive to stock Service “Fast driving”, ... Execution path 3 PDA, Visualize position Service “Process visualization” Execution path 4 PDA, Visualize state Service “Process visualization”

18. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project The application gives the margin for the organic management Furthermore, the application can influence the cost/benefit functions  the application gives the marginal conditions, the organic management handles the details (Self-X)

19. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project • Assignment of the execution paths to resources by a decision tree EP1 EP1 EP1 R1 R2 cb11 cb12 cb13 R3 EP2 EP2 EP2 R1 R2 R1 cb23 cb21 cb22 cbij: cost/benefit parameters • Self-configuration • Dynamic process, reassignment caused by events reported from monitoring is possible (e.g. low energy level, resource failure, ...)  Self-optimization, self-healing • During normal operation: ttree calculation << texecution, furthermore the tree calculation can be interrupted any time resulting in a suboptimal solution  Calculation time is masked by execution time, real-time behavior

20. Rob 1 Palette 1 Palette 3 Palette 2 Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project Demonstrator A B C Rob 4 Rob 2 Rob 3

21. Environment / Task ← Learning/ Adaptation ↓ Algo-rithm ↓ Brain Level Mod. 1 Mod. 2 Environment/ Task Organ Level ControlLoop OrganicManager ControlLoop PC PC PC PC Cell Level PC PC PC Processing Cells Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The DoDOrg Project From Biology towards an Organic Computing System Body / Emotions ParasympatheticNervous System SympatheticNervousSystem Sinus Node AtrioventricularNode Low Power – Fail Safe – Real Time MyocardialCell

22. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The DoDOrg Project Project organization Environment / Task ← Learning/ Adaptation ↓ Algorithm ↓ Application Monitoring Organic RobotControl System (Brändle) Brain Level (Wörn) Mod. 1 Mod. 2 Environment/ Task (Karl) Application API Organic MiddlewareDecentralized Control Loops MiddlewareMonitoring, Feedback Organ Level Monitoring Neurobiological Concepts (Brinkschulte) ControlLoop OrganicManager ControlLoop Ultra Low Power Processing Task Distribution, Configuration, Optimization, Healing (Henkel) PC PC PC PC Organic Processing Elements Hardware Monitoring Cell Level (Becker) PC PC PC Processing Cells

23. Processing Cell Processing Cell Processing Cell Processing Cell Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The DoDOrg Project • Interaction between single processing cells (reconfigurable hardware) is coordinated by organic middleware • The organic middleware formsvirtual organs this way • Continuous modifications in the system (from the environment or the brain level) require the middleware to interfere at each point in time (configuration, optimization) • The organic middleware implements control loops, which fulfill these functionalities • The control loops are decentralized to deal with component failures … Application Application Brain Level … Virtual Organ Virtual Organ Organ Level Organic Middleware Cell Level … …

24. Processing Cell Processing Cell Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The DoDOrg Project • The signals of the decentralized control loops are realized as messengers (accelerator/suppressor pairs) Executed periodically Organic Middleware … Task j Task i Task j Task i Virtual organs Suppressor for task i derived from monitoring Suppressor for task i sent from another cell Accelerator for task i derived from monitoring Accelerator for task i sent from a neighbor cell Send Eager Value of task i to all other cells Eager Values for task i from all other cells Send suppressor for task i to all other cells Send accelerator for tasks related to i to neighbor cells + Eager Value i >? - Initial Eager Value Task i Execute

25. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SELINA Project • Self-organizing sensor actor networks • Explore principles and potential of self-organization in this field • Propose and evaluate architectures with respect to • weak resources • limited energy • frequent change of topology • Interdisciplinary research • Middleware concepts for these requirements Example: measurement of river pollution Project partners: Jürgen Becker (Hardware) Uwe Brinkschulte (Middleware) Uwe Hanebeck (Localization) Klaus Müller-Glaser (Hardware) Heinz Wörn (Application)

26. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SELINA Project Who does the organic management? To scale for large sensor networks and to improve time behavior:Self-organization by self-establishing a hierarchical structure Each functionality present on a node is characterized by a value which denotes its organizing capability. • Example:

27. Radio 20 Bluetooth 10 Radio 20 Storage 5 No organizing power No organizing power Self-Organizing Middleware for DistributedEmbedded Real-Time Systems – The SELINA Project Each node computes its organizing power by adding the organizing capabilityvalues of all the functionalities on the node. The coordinators for a group of nodes are those nodes which have the highest organizing power. Their number depends on the requirements (e.g. real-time) for the group. An hierarchy can be established by introducing intervals for the nodes organizing power. E.g. hierarchy with 4 levels: • level 4: (200,infinite) • level 3: (100, 200] (100,200] • level 2: (50,100] (50,100] (50,100] • level 1: [0,50] [0,50] [0,50] [0,50]

28. 100 20 10 0 15 5 0 5 55 Self-Organizing Middleware for DistributedEmbedded Real-Time Systems – The SELINA Project • Coordinators manage groups • When certain tasks are not possible in the actual group, they can forward it to the next level nodes • By a simple mathematical model certain properties of the self- establishing organizing structure can be proved, e.g. acyclic structure

29. Self-Organizing Middleware for DistributedEmbedded Real-Time Systems – The SELINA Project • Election of coordinators and assignment of services and jobs is a periodic process, where P is the period • P variable and bounded: minP ≤ P ≤ maxP • P can be influenced by the frequency of the network changes and the available energy Advantages: • The system can iteratively find the most suitable coordinators, nodes for services and services for jobs in the actual environment • The system can adapt to changes in the environment, e.g. the network structure • Fault tolerant process, i.e. if a key node or a service fails the net will start a reorganizing process

30. Self-Organizing Middleware for DistributedEmbedded Real-Time Systems – The CAR-SoC Project • Connective Autonomic Real-time Systems on Chip (CAR-SoC) • Network of Systems on Chip • Multithreaded pro- cessor core (SMT) • Real-time scheduling • Local and global control loops for organic management • Decentralized control loops Project partners: Uwe Brinkschulte (Middleware) Theo Ungerer (Hardware, OS)

31. Real-time and Self-Organization The current OSA+ middleware is fully real-time (predictable execution times, end-to-end priorities, deadlines, real-time reconfiguration, ...) Problem: real-time and self-organization Approaches: • Initial self-organization is done before operation (start-up and mission phase) • Re-organization/self-optimization is done only when time with appropriate priorities • Interruptible algorithms for re-organization/self-optimization resulting in sub-optimal solutions • Redundancy to backup self-healing times (e.g. multiple coordinators for a group to avoid loosing the knowledge of a failed coordinator during re- election)

32. Conclusions • Organic Computing is a challenge and a chance for the development of complex, distributed and embedded real-time systems • Conceptions and architectures have been proposed based on different projects • Middleware is a promising layer to realize self-X features • Different principles can be used to implement the self-X features • A lot of work lies ahead of us!

33. Thanks for your attention! Questions?

34. Self-Organizing Middleware for Distributed Embedded Real-Time Systems Variant: state transfer while the old service is still running  further minimization of tblackout, but state changes during transfer  increase of treconf Set a maximum blackout timetblackoutmax: tradeoff between tbackout and treconf Realization: periodically monitoring of the remaning amount of state information to transfer Is the necessary transfer time < tblackoutmax  stop the old service and do the remaining transfer This terminates, if: Amount of state information In tblackoutmax transferable amount of state information with s : average amount of changing state information during job execution tp: period of consecutive jobs r: transfer rate of the state transfer ta: job execution time s tp  r and tblackoutmax ta

35. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project • The mission is handed to the organic management by an XML file • The XML file is structured according to the tree diagram • Restrictions: the application can define values for parameters (min, max, exact) • Weightings: the application can define weights for parameters

36. Service Service Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project Development environment Application defines mission/execution paths Hand over execution paths to OM Limit the usable robots Distributed Organic Manager Monitoring gives state information to OM Robot is executing job Robot is busy OM selects robot for job execution Robot starts job execution Robot is free again

37. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The SIMON Project Distributed Organic Manager OM receives information about other possible robots by monitoring Robot failure OM selects new robot

38. Self-Organizing Middleware for Distributed Embedded Real-Time Systems – The DoDOrg Project Architecture of the organic middleware