FPGA Self-Repair using an Organic Embedded System Architecture

1. FPGA Self-Repair using anOrganic Embedded System Architecture Kening Zhang, Jaafar Alghazo and Ronald F. DeMara University of Central Florida 06 December 2007

2. Reliability Availability Sustainability Organic Computing (OC)biologically-inspired computing with “self-x” properties Technical Objective: support long lifetime missions with multiple failure occurrences Example Relevance: How to achieve sustainable presence in NASA’s Moon, Mars & Beyond objective??? Research Focus: OC Approach:addresses system controllability with increasing complexity Communication networks among autonomous systems Composed of large collection of autonomous systems Autonomous system owned sensor and actuators System Property • Self-organization • Self-configuration • Self-optimization • Self-healing • Self-protection • Self-explaining • Context-awareness • Self-synchronization Self-x Characteristics Reconfigurable Hardware with Self-Healing based on SRAM FPGA platform Sponsors: NASA:FPGA platform and Genetic Algorithm research DARPA:OC approach and SOAR Longevity Platform

3. Goal:Autonomous FPGA Refurbishment increase availability without carrying pre-configured spares … Redundancy increases with amount of spare capacity restricted at design-time based on time required to select spare resource determined by adequacy of spares available (?) yes Refurbishment weakly-related to number recovery capacity variable at recovery-time based on time required to find suitable recovery affected by multiple characteristics (+ or -) yes  Overhead from Unutilized Spares weight, size, power Granularity of Fault Coverage resolution where fault handled Fault-Resolution Latency availability via downtime required to handle fault Quality of Repair likelihood and completeness Autonomous Operation fix without outside intervention     

4. Fault-Handling Techniques for SRAM-based FPGAs Device Failure Characteristics Duration: Transient: SEU Permanent: SEL, Oxide Breakdown, Electron Migration, LPD Device Configuration Processing Datapath Device Configuration Processing Datapath Target: BIST Evolutionary Approach: TMR Scrubbing STARS CED Vigander OC Methods Supplementary Testbench Duplex Output Comparison Duplex/Triplex Output Comparison Detection: (not addressed) Cartesian Intersection Isolation: (not addressed) Bitwise Comparison Majority Vote Autonomous Element (AE) Fast Run-time Location Worst-case Clock Period Dilation Diagnosis: Autonomous Supervisor (AS) unnecessary Population-based GA using Extrinsic Fitness Evaluation Evolutionary Algorithm using Intrinsic Fitness Evaluation Recovery: Replicate in Spare Resource Select Spare Resource Reload Bitstream / Invert Bit Value Ignore Discrepancy

5. Autonomous System-on-a-Chip (ASoC) Architecture • Dual-layerASoC proposed by Lipsa et al [Lipsa 05] • Functional Layer • Functional Elements (FEs) e.g. CPU, RAM, Network interface • Autonomic Layer • Autonomic Elements (AEs) • Monitor • Actuator • Communication interface • Autonomic Supervisor (AS) • UCF Approach for fault coverage • Functional Layer & Autonomic Layer • achieved by assessing consensus • among elements • first to realize failure detection • consensus provides an organic method • for fitness evaluation of competing alternatives during • evolution providing a self-regulating approach to fault resolution

6. EHW Environments • Evolvable Hardware (EHW)Environmentsenable experimentalmethods to researchsoft computingintelligent search techniques • EHW operates by repetitive reprogramming of real-world physical devices using aniterative refinementprocess: Extrinsic Evolution Intrinsic Evolution Application Two modes of Evolvable Hardware or Genetic Algorithm Genetic Algorithm Deep Space Satellite: • >100 FPGAs onboard • hostile environment: radiation, thermal stress • How to achieve reliability to avoid mission failure??? Simulation in the loop Hardware in the loop Done? Build it software model new approach to Autonomous Repair of failed devices device “design-time” refinement device “run-time” refinement

7. Genetic Algorithms (GAs) Mechanism coarsely modeled after neo-Darwinism (natural selection + genetics) start replacement offspring population of candidate solutions evaluate fitness of individuals Fitness function mutation crossover selection of parents parents Goal reached

8. Genetic Mechanisms • Guided trial-and-error search techniques using principles of Darwinian evolution • iterative selection, “survival of the fittest” • genetic operators -- mutation, crossover, … • implementor must define fitness function • GAs frequently use strings of 1s and 0s to represent candidate solutions Genotype chromosomes of GA operation: if 100101 is better than 010001 it will have more chance to breed and influence future population Genotype changes during evolution must adhere to the Xilinx-defined format of bitstream To prevent undesirable conditions that may damage the FPGA such as a mutation which has two logic outputs tied together, a logical genotype is used for evolution and mapped to physical phenotype Logic # = functional logic index number for LUT Row/Column= physical location of LUT in FPGA • Can invokeElitism Operator(E=1, E=2 …) • guarantees monotonically increasing fitness of best individual over all generations

9. Loosely Coupled Solution on Xilinx Virtex II Pro & Virtex 4 The Virtex 2Pro/4 is mounted on a development board which can then be interfaced with a WorkStation running Xilinx EDK and ISE. The entire system operates on a 32-bit basis

10. Organic Embedded System (OES) Architecture One Dimensional Column-oriented OES based on Xilinx Virtex II Pro FPGA platform • FEs and AEs reside on two distinct layers with interconnection structure between them • AEs and FEs can either be realized in hardware, software, or co-design • AE layer supervises functionality of FE elements while requiring no application-specific algorithms on the AE layer • Observer/Controller architecture includes an AS element which had no counterpart to evaluate if the AS fault-free, so address by minimizing its complexity in proposed approach • utilize Xilinx partial reconfiguration technology to manipulate relocatable bitstreams

11. OES AE Component Design AEs decentralize Observer/Controller functionality: • Concurrent Error Detection (CED) unit collects 2 FE Outputs for discrepancy identification • A Checksum for AE fault detection which are checked against Stored Checksum values • Evaluator of outputs from 2 FEs against checksum and Actuator which initiates recovery phase • An important architectural property is that all AE components are identical in structure despite the fact that they monitor different types of FEs. • Homogeneous characteristics deliver a uniform-behavior property leveraged for consensus-based evaluation fault-handling methodology • OC Concept: although AE components add an additional complexity to the design, they will ease integration of fault-handling difficulties inherent with current commercial IP cores

12. Consensus-Based Evaluation (CBE) • Uses a Relative Fitness Measure • Pairwise discrepancy checking yields relative fitness measure • Broad temporal consensus in the population used to determine fitness metric • Transition between Fitness States occurs in the population • Provides graceful degradation in presence of changing environments, applications and inputs, since this is a moving measure • Test Inputs = Normal Inputs for Data Throughput • CBE does not utilizes additional functional nor resource test vectors • Potential for higher availability as regeneration is integrated with normal operation

13. Genetic Operators: Mutation Typical Approach: bit inversion of LUT functionality Selected Approach: input interconnection of LUTs mutated Rearrange input interconnection to search unused LUT resources which occlude faulty resource Mutation: Phenotype chromosomes Mutation: Genotype chromosomes • original functionality is • F = F1·(F3＋F4) w/ input F2 unassigned by synthesis tool • mutation operator will change input F4 to unused as F = F1·(F3＋F2) • shadow shows changed input and LUT contents • some opportunity for input stuck-at fault or LUT content stuck-at fault. • functionalities of LUTs remain undistorted while search space explored

14. Genetic Operators: Cell Swapping Cell-Swap operation on Phenotype chromosomes Cell-Swap operation on Genotype chromosomes interchanges two distinct LUT blocks while maintaining correct logic order and functionalities in genotype • exchange all LUT input interconnections, LUT content and physical 2-tuple (Col#, Row#) as well as the logic sequence

15. Genetic Operators: PMX Operator Partial Match Crossover (PMX) maintains crossover information as well as order information • two genotype configuration streams are aligned at LUT boundary • crossover site selected at random along LUT boundary • this crossover point defines a left/right partition used to affect crossover through LUT-by-LUT exchange • suppose crossover point at position 4 of the LUT vector: • first step is to map configuration B to configuration A by exchanging the following aligned LUTs {(4,7),(5,2),(6,1),(7,5)}. • Applying PMX results in two new configurations A’ and B’

16. Illustrative Example:Gate Level Design of OES • Experiment circuit: 1-bit Full-adder • Fault-free model: Duplex • Fault-impact model: TMR • Fault-detect model: CBE • Fault recovery strategy: GA operation • Experimental setup: • Hardware prototype implemented in Xilinx Virtex-II Pro FPGA • VHDL implementation • Using the GNAT library along with the MRRA framework and JTAG reconfiguration interface.

17. MCNC-91 Benchmark Case Studies System Availability under Multiple Faults Fc = number of correct behaviors of FE observed during evolutionary recovery phase Fe = number of errant or discrepant behaviors 1 = exactly one output required to detect the fault during the original CED configuration. 2 = number of the reconfigurations required, i.e. one from CED to TMR, and one back from TMR to CED Fc1 & Fe1 = correct and faulty output number of the FE during the AE repair period Fc2 & Fe2 = correct and faulty output number during the FE repair period n = number of reconfigurations of the FE βrepresents reconfiguration to computation time ratio

18. Experimental Results • Fault Free arrangement: CED FEs with cold standby FE • Inject a stuck-at-zero or stuck-at-one fault at one of the FE’s LUT input pins • CED -> TMR to identify faulty FE or AE • CBE used to resolve faulty AE Redundancy for both FE (RFE) and AE (RAE) = ratio of unused LUT inputs to total number of LUTs inputs Fc = number of correct behaviors of FE observed during evolutionary recovery phase Fe = number of errant or discrepant behaviors n = number of reconfigurations of the FE β represents reconfiguration to computation time ratio

19. Experimental Results • Fault Free arrangement: CED FEs with cold standby FE • Inject a stuck-at-zero or stuck-at-one fault at one of the FE’s LUT input pins • CED -> TMR to identify faulty FE or AE • CBE used to resolve faulty AE Redundancy for both FE (RFE) and AE (RAE) = ratio of unused LUT inputs to total number of LUTs inputs Fc = number of correct behaviors of FE observed during evolutionary recovery phase Fe = number of errant or discrepant behaviors n = number of reconfigurations of the FE β represents reconfiguration to computation time ratio

20. Experimental Results • Fault Free arrangement: CED FEs with cold standby FE • Inject a stuck-at-zero or stuck-at-one fault at one of the FE’s LUT input pins • CED -> TMR to identify faulty FE or AE • CBE used to resolve faulty AE Redundancy for both FE (RFE) and AE (RAE) = ratio of unused LUT inputs to total number of LUTs inputs Fc = number of correct behaviors of FE observed during evolutionary recovery phase Fe = number of errant or discrepant behaviors n = number of reconfigurations of the FE β represents reconfiguration to computation time ratio

21. Conclusion • A self-adaptation and self-healing OES architecture developed for autonomic operation without human intervention. • The OES architecture is capable of handling many single fault scenarios and several multiple fault scenarios for small digital logic design. • Experimental result support our design objectives during the repair phase averaged 75.05%, 82.21%, and 65.21% for the z4ml, cm85a, and cm138a circuits respectively under stated conditions. • Reconfiguration time ratio (β) ratio is key factor limiting availability during AE repair • Future work: evaluate extensions of the OES architecture addressing scalability of in terms of pipelined stages

22. Backup Slides • On following pages …

23. Isolation of a single faulty individual with 1-out-of-64 impact • Outliers are identified after EW iterations have elapsed • Expected D.V. = (1/64)*600 = 9.375 from individual impacted by fault • Isolated faulty individual’s DV differs from the average DV by 3after 1 or more observation intervals of length EW instantaneous DV (point values) for a sample individual in population and population oracles (solid lines) Sliding Window

24. Future Work:Development Board to Self-Contained FPGA Qualitative Analysis of CRR model • Number of iterations and completeness of regeneration repair • Percentage of time the device remains online despite physical resource fault (availability) Hardware Resource Management • Optimization of hardware profile for Xilinx Virtex II Pro Field Testing on SRAM-based FPGA in a Cubesat mission

25. OES Integrated FE and AE Failure Detection Procedure • System Initialization • FE Initialization step • Compute Checksum step • FE Fault Detection/Recovery • AE-CED fault detection • FE fault-recovery • AE fault detection Phase • A fault may exist in the CED, Actuator, or Evaluator, • A fault may exist in Check Sum component, or • A fault may exist in the Stored CheckSum-LUT. Runtime inputs to FE applied to both active instance under a CED strategy. After allowing for FE inputs propagation time through the AE, the expected output will be supplied to AE-CED for the fault detection. The output of the FE is then compared in the AE-CED module and any discrepancy between the two values will indicate that a fault has occurred either of one the FE or the AE-CED itself. Further detection will be required to distinguish which of the two is faulty. If the AE component is identified as innocent and then the fault must of occurred in this output will be discarded and control will branch to a fault identification phase which will wakeup the cold standby FE and construct a temporary TMR system which can articulate the faulty FE under the new supplied external input. Furthermore, as descrived in Section 3.3, the actuator will initiate a repair cycle which may require automatic evolutionary repair of the identified faulty FE which will be set as standby-under-repair and the AE-CED will return to receive the remaining two active FEs’ inputs. The decision-making procedure causes at least one throughput-delay penalty

26.    Previous Work Detection Characteristics of FPGA Fault-Handling Schemes … Strategy #1) Evolve redundancyinto designbeforetheanticipatedfailure or …

27. Previous Work Fault Recovery Characteristics of Selected Approaches … Strategy #2) Evolve recovery from specificfailure after(and if) it occurs or …

28.  = RS:  = (Hamming Distance) CRR Arrangement in SRAM FPGA • Configurations in Population • C = CL CR • CL = subset of left-half configurations • CR = subset of right-half configurations • |CL|=|CR |= |C|/2 • Discrepancy Operator • Baseline Discrepancy Operator is dyadic operator with binary output: • Z(Ci) is FPGA data throughput output of configuration Ci • Each half-configuration evaluates  using embedded checker (XNOR gate) within each individual • Any fault in checker lowers that individual’s fitness so that individual is no longer preferred and eventually undergoes repair WTA: (Equivalence)

29. Terminology and Characteristics Pristine Pool: CP. For anyCiC, is member of CP at generation G if and only if Suspect Pool:CS. For anyCiC, is member of CS at generation G if and only if at least one of Under Repair Pool:CU: For anyCiC, is member of CU at generation G if and only if Refurbished Pool:CR: after Genetic Operator applied, the new generated individual is member of CR at generation G if and only if EDis Discrepancy Count of Ciand EC is Correctness Count of Ci Length of Evaluation Fitness Window:W = ED+EC Fitness Metric:f(Ci) =EC/ EW

30. Sketch of CRR ApproachPremise: Recovery Complexity << Design Complexity • Initialization • Population P of functionally-identical yet physically-distinct configurations • Partition P into sub-populations that use supersets of physically-distinct resources, e.g. size |P|/2 to designate physical FPGA left-half or right-half resource utilization • Fitness Assessment • Discrepancy Operator is some function of bitwise agreement between each half’s output • Four Fitness States defined for Configurations as {CP,CS,CU,CR} with transitions, respectively: Pristine Suspect Under Repair Refurbished • Fitness Evaluation WindowWdetermines comparison interval • Regeneration • Genetic Operators used to recover from fault based on Reintroduction Rate  • Operators only applied once then offspring returned to “service” without for concern about increasing fitness fitness assessment via pairwise discrepancy(temporal voting vs. spatial voting)

31. States Transitions during lifetime of ith Half-Configuration Configuration Health States

32. Procedural Flow underCompetitive Runtime Reconfiguration Integrates all fault handling stages using EC strategy • Detects faults by the occurrence of discrepancy • Isolates faults by accumulation of discrepancies • Failure-specific refurbishment using Genetic Operators: • Intra-Module-Crossover, Inter-Module-Crossover, Intra-Module-Mutation Realize online device refurbishment • Refurbished online without additional function or resource test vectors • Repair during the normal data throughput process

33. W i=1 Fitness Evaluation Window • Fitness Evaluation Window: W • denotes number of iterations used to evaluate fitness before the state of an individual is determined • Determination ofWfor 3x3 multiplier • 6 input pins articulating 26=64 possible inputs • W should be selected so that all possible inputs appear • More formally, • Let rand(X) return some xiX at random • Seek W : [ rand(X) ] = X with high probability • xK = distinct orderings of K inputs showing in D trials • if D constant, can calculate Pk>1 successively • probability PK of K inputs showing after D trials is ratio of xK / KD

34. W Determination When K=64:

35. Integer Multiplier Case Study • 3bit x 3bit unsigned multiplier automated design: • Building blocks • Half-Adder: 18 templates created • Full-Adder: 24 templates • Parallel-And : 1 template created • Randomly select templates for instantiation in modules • GA parameters • Population size : 20 individuals Crossover rate : 5% • Mutation rate : up to 80% per bit • GA operators • External-Module-Crossover • Internal-Module-Crossover • Internal-Module-Mutation Experiments Demonstrate … Experimental Evaluation Xilinx Virtex II Pro on Avnet PCI board • Objective fitness function replaced by the Consensus-based Evaluation Approach and Relative Fitness • Elimination of additional test vectors • Temporal Assessment process

36. Template Fault Coverage Half-Adder Template A Half-Adder Template A Half-Adder Template B • Template A • Gate3 is an AND gate • Will lose correctness if a Stuck-At-Zero fault occurs in second input line of the Gate3, an AND gate Template B • Gate3 is a NOT gate and only uses the first input line • Will work correctly even if second input line is stuck at Zero or One

37. Regeneration Performance Parameters: Difference (vs. Hamming Distance) Evaluation Window, Ew = 600 Suspect Threshold: S = 1-6/600=99% Repair Threshold: R = 1-4/600 = 99.3% Re-introduction rate: r = 0.1 Repairs evolved in-situ, in real-time, without additional test vectors, while allowing device to remain partially online.

38. Isolation of a single faulty individual with 1-out-of-64 impact • Outliers are identified after W iterations elapsed • E.V. = (1/64)*600 = 9.375 from minimum impact faulty individual • Isolated individual’s f differs from the average DV by 3after 1 or more observation intervals of length W