Principles for Safe and Automated Middleware Specializations for Distributed Real-time Embedded Systems

1. http://www.dre.vanderbilt.edu/~aky/docs/Dissertation.pdf Ph.D. Dissertation Defense April 2, 2012 Akshay Dabholkar aky@dre.vanderbilt.edu www.dre.vanderbilt.edu/~aky Principles for Safe and Automated Middleware Specializations for Distributed Real-time Embedded Systems Department of Electrical Engineering & Computer Science Vanderbilt University, Nashville, TN, USA Research supported by NSF CAREER CNS# 0845789, Vanderbilt Discovery

2. Presentation Road Map Motivation Overview of Solution Approach: Automated Middleware Specialization Process Research Area Focus Safe Middleware Adaptation for Real-Time Fault-Tolerance Research Contributions Concluding Remarks

3. Context: Distributed Real-time Embedded (DRE) Systems • Large-scale, system-of-systems • Operation in resource-constrained environments • Memory-constraints • Low Processor speeds, Power availability • Component/Process/Processor failures • Stringent and simultaneous QoS demands • Efficient Resource Utilization • Timeliness • High Reliability • Examples • Intelligent Transportation Systems (ITS) • Inventory Control Systems • NASA’s Magnetospheric Multi-scale mission (MMS) (Images courtesy: Google)

4. Overcoming Variability in DRE System Domain Concerns Direct Application Generalization Maximize Throughput Resource Constrained Group Failover Semantics • Development via General-purpose Middleware • Feature-rich • Satisfies wide range of DRE systems • Uses extensible frameworks • CORBA, .NET, J2EE, etc. Reconfigurable Conveyor Belt System Intelligent Transportation System Real Time Updates Performance Requirements Reliability Requirements

5. Impediments to Using General-purpose Middleware • General-purpose middleware supports a wide range of DRE applications • However, individual DRE applications have streamlined requirements • Antagonistic Design Forces • Excessive features due to wide applicability • Unnecessary overhead due to high flexibility and configurability • Moreover, focus is on horizontal decomposition into layers • Incurs timeand space overhead due to rigid layered processing • Application concerns are tangled across middleware modularization boundaries

6. Preferred Approach to Overcome Generality in Middleware Direct Application Generalization Maximize Throughput Resource Constrained Real-Time Fault Tolerance Specialization Specific Application Reconfigurable Conveyor Belt System Intelligent Transportation System Doesn’t mean develop middleware from scratch Real Time Updates Performance Requirements Reliability Requirements

7. What is Middleware Specialization? Specialization Customized Middleware Stack Standards-based, General-purpose, Layered Middleware Architecture • Resolves the tension between Generality and Specificity • Creates specialized forms of middleware for each system by • Pruning away unnecessary features based on application requirements • Augmenting application-specificity by embedding their semantics • Optimizing performance by moving away from the rigid layered processing by creating specialized processing paths • Adaptingat runtime to enable safe failure mitigation in real-time

8. Taxonomy of Middleware Specialization Techniques • Taxonomy of specializations developed based on literature survey • Dimensions can be combined to synthesize new specialization techniques • Overlapping dimensions share concepts e.g. MDE/AOP includes both feature pruning & augmentation and can be used for customization as well as tuning • Serves as a guideline for synthesis of tools for design, V&V, analysis of specializations When? How? What? • Three dimensional Taxonomy of Middleware Specializations

9. Assessment of Taxonomy of Middleware Specialization Techniques • DSMLs, Feature Diagrams, IDL used to capture application concerns (e.g., Bypass) MDE (e.g., Modelware), Reflection (e.g., AspectOpenORB) used to deduce specialization context DSML, IDL used to map concerns to code artifacts (e.g., AHEAD, Bypass ) • MDE, AOP used in generating rules for specialization (e.g., Bypass) Augment, Prune middleware sources (e.g., Bypass, CIDE, AHEAD, FOCUS) Pre-postulated, Just-in-time (e.g., Caesar, AFM, AspectOpenORB, JAsCO, PROSE, Abacus) Group of techniques that perform a common function

10. Taxonomy-induced Middleware Specialization Lifecycle SpecializationLifecycle • DSMLs, Feature Diagrams, IDL used to capture application concerns (e.g., Bypass) Specification MDE (e.g., Modelware), Reflection (e.g., AspectOpenORB) used to deduce specialization context Deduction DSML, IDL used to map concerns to code artifacts (e.g., AHEAD, Bypass ) Inference • MDE, AOP used in generating rules for specialization (e.g., Bypass) Generation Augment, Prune middleware sources (e.g., Bypass, CIDE, AHEAD, FOCUS) Transformation Pre-postulated, Just-in-time (e.g., Caesar, AFM, AspectOpenORB, JAsCO, PROSE, Abacus) Adaptation Taxonomy gives rise to a specialization lifecycle

11. Taxonomy-induced Middleware Specialization Lifecycle SpecializationLifecycle • Specify and Reason about the desired application features Specification Detect the context from the application models that drives specialization opportunities Deduction Infer the specializations applicable to the context and actual middleware features desired Inference • Generate the middleware transformation rules that realize the specializations Generation Transformation Transform middleware into specialized forms Adapt middleware safely and predictably to changes in runtime conditions Adaptation • Must deal with variability over the specialization lifecycle • Multiple steps involved

12. Ruling out Manual Middleware Specializations Proprietary, one-off solutions are insufficient and expensive • How to: • Devise an automated reusable, systematic, correct and maintainable middleware specialization process ? FOCUS: A. Krishna et al., “Context-Specific Middleware Specialization Techniques …”, EuroSys 2006 • Cumbersome to implement • Not repeatable and reusable • Lack proper structure or process • Difficult to • maintain as middleware sources evolve over time • guarantee their correctness • extend to other middleware technologies

13. Presentation Road Map Motivation Overview of Solution Approach: Automated Middleware Specialization Process Research Area Focus  Safe Middleware Adaptation for Real-Time Fault-Tolerance Research Contributions Concluding Remarks

14. Research Synopsis • Feature Oriented Reverse Engineering based Middleware Specializations (FORMS) • Coarse-grained Feature Pruning • Feature-oriented deduction of desired features • Prunes middleware sources using a novel reverse-engineering algorithm • Provides Build Specialization • Generative Middleware Specializations (GeMS) • Fine-grained Feature Pruning • Automatically Deduces the specialization context • Identifies of specialization points by extending FORMS • Generates source-to-source transformation algorithms • Generative Aspects for Fault-Tolerance (GrAFT) • Fine-grained Feature Augmentation • Weaves reliability concerns in system artifacts • Provides model-to-text, model-to-code transformations • Safe Middleware Adaptation for Real-Time Fault-Tolerance (SafeMAT) • Fine-grained middleware adaptation to failures while maintaining safety, predictability and improving resource utilizations within the hard real-time constraints SpecializationLifecycle Specification Deduction Inference Generation Transformation Adaptation

15. Research Synopsis • Feature Oriented Reverse Engineering based Middleware Specializations (FORMS) • Coarse-grained Feature Pruning • Feature-oriented deduction of desired features • Prunes middleware sources using a novel reverse-engineering algorithm • Provides Build Specialization • Generative Middleware Specializations (GeMS) • Fine-grained Feature Pruning • Automatically Deduces the specialization context • Identifies of specialization points by extending FORMS • Generates source-to-source transformation algorithms • Generative Aspects for Fault-Tolerance (GrAFT) • Fine-grained Feature Augmentation • Weaves reliability concerns in system artifacts • Provides model-to-text, model-to-code transformations • Safe Middleware Adaptation for Real-Time Fault-Tolerance (SafeMAT) • Fine-grained middleware adaptation to failures while maintaining safety, predictability and improving resource utilizations within the hard real-time constraints SpecializationLifecycle Specification Deduction Inference Generation Transformation Adaptation

16. Challenge 1: Horizontal v/s Vertical Middleware Decomposition • How to: • Reason about the middleware features desired by the application/application family? • Modularize of middleware along domain concerns (i.e., Vertical Decomposition) without refactoring the middleware code? Middleware is traditionally decomposed along the horizontal dimension into layers However, applications only use a subset of features within each middleware layer Application domains expect vertical decomposition along domain concerns Moreover, application domain concerns are tangled across middleware modularization boundaries It becomes hard to specify the desired features for middleware specialization and decomposition

17. Resolution 1: Feature Oriented Reverse Engineering based Middleware Specializations (FORMS) Specialization Lifecycle • FORMS: Feature-Oriented Requirements Reasoning • Utilizes a Feature Oriented Decision Tree Specification Deduction • FORMS: Automated Inference of desired Middleware Features • Utilizes a lookup table that provides PIM-to-PSM mapping Inference • FORMS: Closure Computation • Coarsely Prunes the middleware by finding the feature modules using a novel closure computation algorithm by recursively inspecting source code dependencies Generation • FORMS: Build Specialization • Specialize/Prune the middleware build configurations and generate binaries using MPC perl scripts Transformation Adaptation

18. Resolution 1: Feature Oriented Reverse Engineering based Middleware Specializations (FORMS) • Solution Approach: Facilitate Vertical Decomposition (along domain concerns)of features within a horizontally decomposed middleware without refactoring the middleware code • Original ACE (Adaptive Communication Environment) middleware • 1,388 PSM source files • 436 features • 2,456 KB static footprint • Specialized ACE middleware • ~500 PSM source files  64% reduction • ~ 100-175 features  60-76% reduction • ~ 1,500 KB footprint  41% reduction

19. Challenge 2: Fine-grained Pruning Middleware Specialization SpecializationLifecycle Specification Deduction Specilization Contexts Specilization Contexts Specilization Contexts Specilization Contexts Specilization Contexts Inference Key Insight Generation • How to: • Deduce the specialization context from application invariants? • Infer the set of specializations from the context? • Identify the specialization points within the middleware code? • Generate the specializations to improve developer productivity? • Transform the middleware sources by executing specializations? Transformation Adaptation

20. Resolution 2: Generative Middleware Specializations (GeMS) SpecializationLifecycle Specification • GeMS: Automated Deduction of Specialization Context • Interpreters that parse application models for application invariants that provide the context to drive specializations Deduction • GeMS: Automated Inference of Specializations • Utilizes a lookup table for specializations that apply Inference • GeMS: Specialization Transformation Generator • Finely Prunes the middleware sources by utilizing design pattern optimization algorithms with the aid of a source inspection engine to determine the specialization points Generation • GeMS:FOCUS Source code-level transformations • Execute the generated transformations using FOCUS perl scripts Transformation Adaptation

21. Resolution 2: Generative Middleware Specializations (GeMS) • Specification of desired application features • Deduction of Specialization Context • Inference • Map Application Invariants to Specializations from the catalog • Determine the middleware features from mappings • Generate Transformations through Algorithms • Transform middleware sources and build files into specialized forms • Compile to generate specialized middleware binaries • GeMS’s code generator substantially reduces middleware developer efforts

22. Resolution 2: Performance Metrics Evaluation of FORMS+GeMS SpecializationLifecycle • Cumulative benefits of applying both FORMS and GeMS when applied to The ACE ORB (TAO) specifically the ORB and POA frameworks • Static footprint is the size of compiled shared middleware library • Dynamic footprint is the combined average size of runtime executables of BasicSP application components each of which is running on a specialized middleware (TAO) version • 10-15% savings if applied on individual specialization basis • Substantial footprint reductions are mainly a result of applying the FORMS closure computations • Runtime performance improvements are mainly due to GeMS framework optimizations Specification Deduction Inference Generation Transformation Adaptation

23. Challenge 3: Augmentation of Application-Specific Semantics • Missing application-specific semantics (run-time middleware) • E.g., Group failover is DRE-specific & often not provided as first class support out-of-the-box • However • Application-level solutions lose transparency & reusability • It is costly to modify the middleware manually • Therefore, automatic middleware instrumentation required to augment application-specificity Primary Distributed Processing Unit (DPU) A A’ B B’ C C’ Backup Fail Over Unit (FOU) How to: Augment application-specific additional semantics in general-purpose middleware retroactively? Automate the augmentation to improve productivity & reduce cost?

24. Resolution 3: Generative Aspects for Fault-Tolerance (GrAFT) SpecializationLifecycle Specification • GrAFT: Modeling Environment and Transformations • Fine Grained Feature Augmentation • Provides model-to-text, model-to-code transformations Deduction Inference • GrAFT: Aspect C++ Generator • Generates the application-specific semantics in form of aspect code for fault handling and masking to enable transparent group failover Generation • GrAFT: Source code-level transformations • Finely Augments these application-specific semantics in system artifacts by weaving in the generated aspects with application and client stubs Transformation Adaptation

25. Resolution 3: Generative Aspects for Fault-Tolerance (GrAFT) Reconfigurable Conveyor Belt System • GrAFT’s code generator completely eliminates middleware developer efforts Specify Application-specific semantics of Group Failover Parse application models and determine the components that require group failover semantics Generate the fault detection, masking, and failover code through exception handling mechanisms Weave inAspectC++ code in the generated code in the respective component stubs

26. Relevant Publications FORMS Publications FORMS: Feature-Oriented Reverse Engineering-based Middleware Specialization for Product-Lines, JSW 2011 Middleware Specialization for Product-lines using Feature Oriented Reverse Engineering, ITNG 2010 Developing and Evaluating a Taxonomy of Modularization Techniques for Middleware Specialization, ACoM2008 Towards a Holistic Approach for Integrating Middleware with Software Product Lines Research, McGPLE 2008 GeMS Publications GeMS: An Automated Middleware Specialization Process for Distributed Real-time and Embedded Systems, Elsevier-JSA 2012 (in submission) A Generative Middleware Specialization Process for Distributed Real-time and Embedded Systems, ISORC 2011 Architecture-Driven Context-Specific Middleware Specializations for Distributed Real-time and Embedded Systems, LCTES-WIP 2010 An Approach to Middleware Specialization for Cyber Physical Systems, WCPS 2009 GrAFT Publications Fault-tolerance for Component-based Systems – An Automated Middleware Specialization Approach, ISORC 2009 CQML: Aspect-oriented Modeling for Modularizing & Weaving QoS Concerns in Component-based Systems, ECBS 2009 Towards A QoS Modeling & Modularization Framework for Component-based Systems, AQuSerM2008 MoPED: A Model-based Provisioning Engine for Dependability in Component-based Distributed Real-time Embedded Systems, ECBS 2011 First Author Second Author 26

27. Presentation Road Map Motivation Overview of Solution Approach: Automated Middleware Specialization Process Research Area Focus  Safe Middleware Adaptation for Real-Time Fault-Tolerance Research Contributions Concluding Remarks

28. Motivation: Addressing Failures in System-of-Systems • Safety-critical applications such as in avionics, automotive, industrial automation domains • Composed of system-of-systems • Must handle variety of failures stemming from the composition • Certified to be schedulable and guaranteed to meet stringent QoS • However, individual subsystems have • Closed nature • Static execution schedules • Resource-constrained • Often over-provisioned in terms of allocated time and required capacity of resources to guarantee predictability in worst-case scenarios • How to handle failures in system-of-systems in the context of • Closed and over-provisioned, individual subsystems • Stringent real-time QoS assurance

29. Challenge 4: How to Safely and Predictably Adapt to Failures? • Fault tolerance solutions need additional resources, but • No additional resources are available due to over-provisioning • Rigid execution schedules severely constrain the extent of runtime failure adaptability in real-time • Cannot compromise on system safety • Redesigning and reimplementing the individual subsystems is not an option due to economic forces • Since system-of-systems are being formed, we need to think holistically about how to handle faults within this concept. Primary Distributed Processing Unit (DPU) ? A A2 A1 B B1 B2 C1 C C2 Backup Fail Over Unit (FOU) Backup Fail Over Unit (FOU) X Over-Provisioned Resource-Constrained

30. Challenge 4: How to Safely and Predictably Adapt to Failures? • Side Effects of Over-provisioning • Most of the time resources remain under-utilized • Large amount of processor utilization and time slack within each allocated task quantum that can be better leveraged for adaptive fault management • Therefore it is important to, • Identify availability of unused resources at runtime • Do No HarmProvision fast and resource-aware failure adaptation to ensure safety and predictability while obeying real-time constraints • Key Insight Existence of significant slack in over-provisioned individual subsystems Primary Distributed Processing Unit (DPU) ? A A1 A2 B B1 B2 C C1 C2 Backup Fail Over Unit (FOU) Backup Fail Over Unit (FOU) How to: Identifythe opportunities for slack in the DRE execution schedule Design safe and predictable dynamic failure adaptation Validate system safety in the context of DRE system fault tolerance

31. Related Research: Middleware Adaptation Techniques Runtime adaptations to reduce failure recovery times Change of replication styles, reduced degree of active replication

32. Related Research: Real-time Fault-Tolerant Middleware & Software Health Management Schedulability analysis to schedule backups in case primary replica fails, faster processing times Detect , Diagnose and Reason only known failures with predefined failover strategies

33. Related Research: Middleware Composition Techniques Middleware building blocks for fault-tolerant systems Focus only on composing one QoS at a time • Software Health Management (SHM) approaches ensure safe and predictable adaptations, however • Apply to only errors in components implementations known a priori • Support only predefined failover strategies • Are resource agnostic • Adaptive Fault Tolerance (AFT) approaches improve overall resource utilizations, however • Mostly applied to soft real-time applications • Require additional resources consuming precious time from the real-time schedule • Excessively dynamic

34. Resolution 4: Safe Middleware Adaptation for Real-Time Fault Tolerance (SafeMAT) How do we safely adapt middleware to runtime failures while maintaining predictability in real-time? SpecializationLifecycle Specification Deduction Inference Generation Transformation • Safe Middleware Adaptation for Real-Time Fault-Tolerance (SafeMAT) • Fine-grained middleware adaptation to failures while maintaining safety, predictability and improving resource utilizations within the hard real-time constraints Adaptation

35. SafeMAT: Platform Assumptions • Build upon and leverage ACM (ARINC Component Model) Middleware - an emulation of the avionics ARINC-653 specification for time and space partitioning in safety-critical real-time operating systems • Hierarchical fixed priority preemptive task model • Specifies the platform in terms of modules (processors) that are composed of one or more partitions (processes) allocated as tasks • Each partitionhas one or more componentsallocated as sub-tasks • Each partitionhas dedicated execution time and memory space allocated and executes at highest priority and can only be preempted when it is allocated time quantum expires.

36. SafeMAT: System Model and Fault Handling • Two primary sources of failure for each component port • Logical Failure – component failure due to internal software, concurrency & environmental faults, latent bugs in the developer code • Critical Failure – process/processor failures, undetected component failures • Failover Strategies • Logical Failure – failover to alternate backup replica only • Critical Failure – failover to identical or alternate backup replicas • Replica Placement • Identical Replica – Always deploy to different partition than primary • Alternate Replica – Can be deployed within same partition • Component Execution States • Active – all ports are operational • Semi-Active – required ports operational, provided ports disabled • Inactive – none of the ports are operational • Fail-Stop failures • Semi-Active replication due to hard real-time constraints and to avoid state synchronization overhead • One primary replica – active state – handles all client requests • Multiple backup replicas – semi-active state – only process client’s requests; not produce any output. • Failure Granularity • Component • Component Group • Partition (process) • Module (processor)

37. SafeMAT: Key Requirements Primary Distributed Processing Unit (DPU) ? A A1 A2 B B1 B2 C C1 C2 Backup Fail Over Unit (FOU) Backup Fail Over Unit (FOU) • Fine-grained resource monitoring capability required that provides real-time utilization while not imposing significant overhead on the system to enable failure adaptation in real-time while utilizing available slack • Dynamic failure adaptation should - • Tolerate different failure types and granularities • Achieve better resource utilization • Safely and Predictably achieve failure recovery • To reduce the extent of recovery required, dynamic failure adaptation should be • Be Fast & Lightweight • Obey hard real-time constraints (predictability) • Flexible (account for failure type, granularity, replica placements)

38. ACM Middleware: Architecture A. Dubey, G. Karsai, and N. Mahadevan, “A component model for hard real-time systems: CCM with ARINC-653,” SPE 2011 • System Level • Module Scheduler • Alarm Aggregator • Diagnoser • Deliberative Reasoner (DR) • Module Level • Partition Creator • Partition Scheduler • Module Initializer • Partition Level • CLHM • Partition Initializer • Components

39. SafeMAT: Architecture • System Level • Module Scheduler • System Resource Manager (sRM) • Failure Handler • Alarm Aggregator • Diagnoser • Resource Aware Deliberative Reasoner (RADaR) • Module Level • Partition Scheduler • Module Resource Monitor (mRM) • Failure Handler • Partition Manager Level • Partition Launcher • Partition Resource Monitor in computemode (pRMc) • Failure Handler • Partition Level • CLHM • Partition Resource Monitor in notify mode (pRMn) • Components

40. SafeMAT: Safe and Fast Failure Detection and Isolation • Prevents partition resynchronization upon restart • Instructs the dependent partitions to reread the object references if facet side is restarted • Enables monitoring of partition resource utilizations through pRMc • Partition Manager • Handles execution and failure management of each partition • Detects and Isolates the impact of failed partitions • Relieves partition recovery responsibility from the Module Manager • Enhances safety by preventing failed partitions from impacting the real-time execution • Enables quick recovery by coordinating with RADaR

41. SafeMAT: Distributed Resource Monitor (DRM) • Monitors Utilization and Liveness of distributed resources • Configuration • Ability to monitor CPU utilizations at various granularities of processor, process, component group, component and thread • Can operate in reactive (on-demand)or periodic (collect history) modes • Can report utilizations of only specific entities RADaR is interested in • Discovering Resource Allocations • Dynamically discover the exact runtime allocations of threads to components, components to partitions, and partitions to modules to enable fast monitoring • Framework Components • Single System Resource Monitor (sRM) • Multiple Module Resource Monitors (mRM) • Multiple Partition Resource Monitors in either compute (pRMc) or notify (pRMn) modes • Resource LivenessMonitoring • Auxiliary to failure handlers that monitor exit statuses of partitions and their managers - dual monitoring capability • Periodically collects liveness statuses from each of its own monitors to determine partition, partition manager, module failures

42. SafeMAT: Enabling Hierarchical Failure Adaptation (HFA) • Adapt failover targets based upon failure type, granularity and backup replica placement • Capable of handling simultaneous module, partition, logical and critical component failures • Component Failure Type • Logical • Critical • Various Failure Granularities due to Hierarchically Scheduled Real-time System • Component • Component Group (e.g. Subsystem) • Partition • Module • Primary-Backup Deployment Topology • Only alternate backup replicas on same partition as primary • Identical backup replicas always on different partition • Both identical & alternate backup replicas on different partition as primary • Same module • Different module

43. SafeMAT: The Hierarchical Failover Adaptation (HFA) Algorithm • Invoked whenever any of the DRM and/or the SLHM frameworks detect a failure • To provide quick and efficient failover • sRMproactively pre-computes the sorted list of least utilized backups • Sends the sorted list to the RADaR piggybacked with the failed primaries • Hands over control to the SLHM which decides when to initiate failover that depends upon • # failures system can withstand • Time for system to stabilize (usually at least a Hyperperiod long) • Ability to • Intelligently mitigate simultaneous failures in an hierarchical fashion • Choose failover target not just based on current utilization but also based on historical averages

44. SafeMAT: Empirical Evaluation A. Dubey, N. Mahadevan, and G. Karsai, “The inertial measurement unit example: A software health management case study,”ISIS Tech. Rep., Vanderbilt University, 02/2012 • Inertial Measuring Unit (IMU) • 4 subsystem types • 7 primaries, 4 backups • 55 components • All secondary subsystems are semi-actively replicated • ADIRU subsystem can withstand 2 accelerometer component failures • GPS and ADIRU subsystem run at 0.1 Hz and 1 Hz respecitvely • PFC fetches GPS data at 0.1 Hz • Display fetches PFC data at 1 Hz

45. SafeMAT: Performance Metric Evaluation (PME) – Runtime Utilization Overhead We executed the IMU system for 100 iterations for faulty scenarios for both ACM-SHM and SafeMAT We artificially introduced failures at 15, 20, 30, 35 iterations in the GPS Processor, Accelerometers 6, 5 and 4 respectively such that the values outputted by them are exceedingly high SafeMATadded only 2-6% utilization overhead on the top of ACM-SHM Do No HarmSafeMATadded negligible runtime utilization overhead thereby not overloading the system while performing better failure recovery within the available utilization slack

46. SafeMAT: Performance Metric Evaluation (PME) – Runtime Failover Overhead • Impact of Replica Placement • We used the Boeing’s BasicSP scenario to demonstrate the impact of replica placement • We altered the deployments of the backup replicas in three ways - • Same Partition • Different Partition, Same Module • Different Partition, Different Module 46

47. SafeMAT: Performance Metric Evaluation (PME) – Runtime Failover Overhead work over ACM-SHM when recovering only one component at a time • Impact of Replica Placement • We used the Boeing’s BasicSP scenario to demonstrate the impact of replica placement • We altered the deployments of the backup replicas in three ways - • Same Partition • Different Partition, Same Module • Different Partition, Different Module • SafeMATroughly added 63-70% 47

48. SafeMAT: Performance Metric Evaluation (PME) – Runtime Failover Overhead • Impact of Recovery Group Size • We tested for different subsystems within the IMU and BasicSP scenarios • SafeMATadded only 9-15% runtime failover overhead over groups of components • Recovery Times are dependent upon • Size of component group • Component deployments within the failover group • Amount of network communication within the DRM • Costs are amortized for large group sizes than individual components • No missed deadlines, application jitter was unaffected • Do No HarmSafeMATadded negligible runtime failover overhead thereby maintaining the predictability of the overall system

49. Presentation Road Map Motivation Overview of Solution Approach: Automated Middleware Specialization Process Research Area Focus  Safe Middleware Adaptation for Real-Time Fault-Tolerance Research Contributions Concluding Remarks

50. Doctoral Dissertation Contributions Principles for Safe and Automated Middleware Specializations for Distributed Real-time Embedded Systems 50