Mini AerCam Project

1. Mini AerCam Project Presented By Jeff Cotner Independent Consultant to NASA JSC through LMSO/Seat Aer Cam Project

2. My background • 15 years of VHDL ( The tools have certainly improved) • 7 years of FPGA (The tools have certainly improved) • 17 years of consulting (The rates have certainly not improved) • VHDL instructor • Communications, Avionics, Power Systems Aer Cam Project

3. Introduction • MiniAerCam – What is it? • Very simply an inspection vehicle • Contains Video Imagers, Avionics, Propulsion, GPS, Gyros, Illumination, C&T System, Power System • Intended for use on Shuttle and Station • JSC Robotics in charge of development Aer Cam Project

4. Mini-AERCam APB Architecture PowerPC 750FX 500MHz 32bit / 100MHz Main FPGA (Virtex II 3000) UART 19.2 kbps GPS SDRAM /EDAC Controller Memory Controller 32bit/100MHz SDRAM 256 MB PPS UART 115.2 kbps? Gyros TBD MBps/TBD MBps (R/W) Flash TBD MB UART 57.6 kbps Console Thruster (x12) Iso (x2) & EM I/F Thrusters Iso Valves Electro Mag I2C 200 kbps Instrumentation UART 100 kbps IrDA Ethernet Chip Reset Controller Loading/Heart Beat ? Mbps ? Mbps I2C or SPI Video 2 Color Illuminator ? Mbps Comm MPEG2 for Video, uncompressed for LDRI LDRI ? Video 1 Mono ? Mbps Aer Cam Project

5. Radiation Mitigation • Requirements -SEU will not cause destructive latch-up -SEU or effect will be detected -SEU can cause undesired consequences (This is not so much a requirement as a fact) -Single point failure can cause reboot -Reboot times allow for fast recovery from SEU induced failures or double bit hits NOTE: AerCam is not redundant, loose a part, can’t fly. Too many SEUs reboot. Aer Cam Project

6. Radiation Mitigation • Rad Tolerant “Hard Core” • Actel based Reset Controller • Incorporates watchdog timer in VHDL • Latch-up immune, extremely low SEU rate, TMR in hardware • Battery • DC-DC power converter • Oscillator • Remaining elements on APB can be power cycled to clear errors • Rad Hardened main FPGA • Partial reconfiguration to detect and correct configuration errors due to SEUs Aer Cam Project

7. Radiation Mitigation • Definitions Main FPGA -Interfaces Program FLASH, SDRAM, I/O to Processor -Provides sanity signals to Hard Core -EDAC SDRAM -CRC Program Flash Aer Cam Project

8. Radiation Mitigation • Definitions Reset Controller (Hard Core) • -Configures Main FPGA and also Partial Reconfiguration (No scrubbing just rewrite configuration memory) -Controls Power to Vehicle Systems -Determines Sanity of Processor and Main FPGA (Cycle Power or Reset if Sanity lost) Aer Cam Project

9. Radiation Mitigation • Multiple copies of configuration data in Flash (In Same Device) • Multiple copies of Application code in Flash (In Same Device) • PowerPC 750FX processor • SOI process, no latch-ups • L1 cache parity protected, L2 cache ECC protected • Thrusters • Separate arm and fire registers • Hardware controlled duration • Individual thruster watchdogs • Iso valve can shutdown all thrusters • Main memory ECC protected • Individual subsystems can be power cycled to clear errors Aer Cam Project

10. Raw Battery +6V (TBR) Converted Power +3V (TBR) Converted Power On/Off Control LDRI ? Gyros MonoImager GPS Comm IrDA LDO LDO LDO LDO LDO ColorImager LDO LEDs ? Mini-AERCamPowerDistributionArchitecture APB I2CBus Thruster Drivers LDOs Thruster, IsoValve, EM Power Supply DC-DC DC-DC Battery (12V-16V) 24V DC-DC Aer Cam Project

11. Why we think this is acceptable • AerCam travels at very slow velocities • A stuck thruster will just spin the vehicle -Multiple stuck thrusters are a problem • Thrusters have multiple guards against sticking • Can reboot fairly quickly to regain control • If communications are lost, Shuttle can pull away from AerCam • These justifications are from Robotics at JSC If you happen to be an astronaut and have an issue, please take this up with Robotics. I’m just a lowly Logic Designer. Aer Cam Project

12. SDRAM Controller • Requirements -Standard SDRAM controller -EDAC -Multi device access (Virtex PPC 405) -Autonomous Rewrite cycle for EDAC -As robust as possible given no TMR & possible reactions to SEU -Design reuse a possible by product -125 MHz clock speed Aer Cam Project

13. Icing on the Cake • Extra things we did on this controller -Programmable scrub range Why scrub memory you aren’t using -Scrub cycle complete signal Nice to know how long a scrub pass takes -Track SDRAM hits by sector Conceivably could tell when a portion of a SDRAM has failed. Aer Cam Project

14. Design Approach • VHDL structured design • As bullet proof a state machine as possible • Verification through -Functional Simulation -Post Synthesis Simulation -Post Lay Out Simulation -Use of complex VHDL stimulus models Aer Cam Project

15. Design Organization Aer Cam Project

16. The Bullet Proof State Machine • Ours isn’t perfect either, but not too bad • Our wish list -Want a pretty safe state machine -SEU disruptions to a minimum -Easy checking for illegal states -Fast clock speed Aer Cam Project

17. How we think we got there • Barrel Shifter provides the safety -Initial state and a linear progression through possible states returning to Idle • Barrel Shifter provides an easy method to check for illegal states More than one bit set and you’ve got problems • Overlay desired SDRAM cycle over linear progression of states (Burst R/W, Precharge, Mode Write, Refresh) Aer Cam Project

18. More About The State Machine Barrel Shifter StateError State Machine Error Single One Detector S0 ‘0’ S1 ‘0’ State Machine Outputs S2 Decoded States (Registered) ‘0’ S3 ‘1’ S4 ‘0’ State Machine Inputs This example denotes S3 since a one is in the S3 position Aer Cam Project

19. More About The State Machine • More detail about the previous slide • Initial State is S0 “0001” (Also Idle) • State Error exists when more than one bit is set • Inputs to state decoder determine what sequence of outputs are generated i.e. for this SDRAM controller there are a maximum of 15 states.If you look at the SDRAM data sheet you can overlay each type of operation (Read,Write,Mode,Refresh) over these 15 states. The nice thing about SDRAM cycles, they are started and run to completion with interruption, that’s why this State Machine works Aer Cam Project

20. Maybe Some Code will help A clocked process to do the barrel shifter State_gen : Process (Return_Idle,clk) begin if (Return_Idle=‘1’) then state <= “0001”; elsif (clk’event and clk=‘1’) then if ((gen-c1=‘1’ or gen-c2=‘1’) and state=“0001”) then state <= “0010” else state <= state (1 to 3) & ‘0’; end if; end if; end process State_gen; Assume state : std_logic_vector (0 to 3); four states Aer Cam Project

21. More code please • State Decoder stuff State_decode : process (Reset_L,clk) Begin If (Reset_L=‘0’) then All_Outputs <= ‘0’; Return_Idle<=‘1’; Done<=‘0’; elsif (clk’event and clk=‘1’) then case state is when “0001” => if (gen-c1=‘1’) then modify outputs for gen-c1 event in S0; Aer Cam Project

22. Return_Idle <= ‘0’; elsif (gen-c2=‘1’) then modify outputs for gen-c2 event in S0; Return_Idle <= ‘0’; end if; Done <= ‘0’; when “0010” => if (gen-c1=‘1’) then modify outputs for gen-c1 event in S1; Return_Idle <= ‘0’; Done <= ‘0’; elsif (gen-c2=‘1’) then modify outputs for gen-c2 event in S1; Return_Idle <= ‘1’; (Return to Idle truncated cycle) Done <= ‘1’; end if; (This should illustrate the concept) Aer Cam Project

23. Some additional information -In the code above All Outputs registers are reloaded during every state. -gen-c2 doesn’t need all provided states so it terminates early using Return_Idle signal (i.e. SDRAM modewrite & precharge don’t need as many clocks as Read/Writes) -SDRAM Ras,Cas,WE can be overlaid on the linear progression of states based on the needed cycle (Read,Write,Refresh,etc) Aer Cam Project

24. More additional Information -In states other than idle, Return_Idle must be setto zero in at least one state, otherwise the state register will eventually become all zeros. If you detect all zeros in the state register you can protect against this error also. -Nested if statements gives priority but beware, too much nesting causes excessive timing delays. -Requests to state machine are held high until Done signal is asserted indicating requested cycle is complete. Aer Cam Project

25. Isn’t that what one hot is? • Not quite the same -One hot can jump from any state to any other state. Barrel shifter always forces linear state transitions. -Barrel shifter has basically two options Hold in initial state or progress through states. State decoder can truncate the cycle of states and return to Idle. Aer Cam Project

26. Additional State Machine Safety • Refresh all outputs every state -Extra logic but minimizes undesired output due to SEU (Some designers only code an output change in states during which a given output should change. You generate extra logic when you code logic for every given output in every state) -Keeps SEU induced output transitions to one clock This is true as long as the SEU occurs in an output register. When the inputs get clobbered all bets are off. Aer Cam Project

27. Where is the design currently • Very much a work in progress -Functional simulation done -Design synthesized -Needs Post Synthesis and Post Layout verification and timing checks -Other FPGA peripheral cells need to be integrated into design and simulated • What’s the holdup -Funding issues and time Aer Cam Project

28. Performance and Size • Design much smaller than anticipated -Place and Route tools indicate entire SDRAM controller occupies approx. 1% of a Virtex II 3000 • How fast will it run -Estimates are SDRAM controller will operate at clock speeds up to 115MHz Final requirement was 100MHz Aer Cam Project

29. Design Tweaks • We’ve got some time before flight -Since design turned out so small might consider some TMR -Perform more exhaustive verification exercise -Investigate periodic SDRAM mode register refresh -Test PPC 405 path to SDRAM Aer Cam Project

30. If I Only Could Of……. • Some things that might be done differently -Had better requirements from the onset -Understood SDRAM SEU failures better -Added some additional protection when processor crashes i.e. Maybe some cumulative vector logic so we could null motion in Reset Controller (Hard Logic) when sanity is lost. -Hard Proximity Detector & RF Beacon AerCam starts screaming if it gets too close to vehicle. Aer Cam Project

31. VHDL Pointers for Safe Logic • Avoid too much clever stuff • Stick to cloud of logic with output register • Plug all the holes -This can really help out in verification -The beauty of the barrel shifter -This can be very hard with conventional state machines. What does “When others” really mean at the end of the case statement? Aer Cam Project

32. VHDL Pointers for Safe Logic • When you want to be sure Instantiate (if you can) this issues dogs me with XST (ADD8 is a good example. XST won’t take an instantiation you must infer, Synplify will) -Use this all the time with DLLs and Buffers IOBs, etc. in Virtex -Keep in mind if you instantiate everything you might be better off doing things with a schematic Aer Cam Project

33. VHDL Pointers for Safe Logic • Be wary of synthesis tools -Optimization can be a real pain Warning messages often tell as much as looking at Logic generated by tools. Warnings often tell you something isn’t coded right. -Take care that registers didn’t get inferred as latches. -Use acceptable coding practices Standard inference rules produce uniform results. Good coding practices reduce the need to sift through synthesizer logic outputs. Aer Cam Project

34. VHDL Pointers for Safe Logic • Get to know your synthesis tool User groups can be a great thing too! • Tool vendors go to great lengths to ensure they meet standards • Lots of examples exist to help produce uniform synthesis results • You might not believe it, but someone else has already had most of your problems and it’s probably in a database you can get to. Aer Cam Project

35. VHDL Pointers for Safe Logic • Be wary of the great sales pitch -Still have concerns about C code to logic conversion tools. The demos look great but they seem to be optimized to the tools. For some real world applications I have seen them generate extra clocking stages (design won’t make timing objectives) or synthesis tools optimize out some desired logic. -Haven’t looked at these recently so it could be better. Aer Cam Project

36. VHDL Pointers for Safe Logic • The boss wants one of the software guys to do VHDL for a FPGA -My little tale from VHDL class -I have seen many programmer types generate great looking code, made functional simulation work but wouldn’t synthesize and/or make timing. -Best results generated when a designer understands logic design as well as HDL Aer Cam Project

37. General Logic Issues • Understand what your part is connected to -Not a good idea to generate invalid data to some connected component. -Your part may continue to work but everything may come to a grinding halt and/or smoke! -Man invented data sheets for a reason Aer Cam Project

38. General Logic Issues • Hiccups will happen -SEUs can wreak havoc on a design. -TMR if you can -Otherwise view unprotected logic blocks in system context. -Detect problems somewhere and recover (sometimes processors and/or software can do the job) Aer Cam Project

39. General Logic Issues • Hiccups continued -If you must glitch try to isolate or minimize the effect on the next block. i.e. the notion of a latch refresh every clock This is true even with TMR. The longer a latch sits uncorrected, the greater the chance a redundant latch can get hit and TMR breaks. Aer Cam Project

40. The Holy Grail of Logic Design • Verification -Perhaps the most daunting task facing logic designers today. -My best advice is to break down design into smaller blocks that you can verify independently. -Try to constrain the outputs of a block even when SEU strikes. If you can characterize the output of a block, even in error or SEU conditions it allows a more sane verification of the design as a whole Aer Cam Project

41. Verification Continued • The old days of running every possible test vector into a complex design are over. • The new verification tools are probably very rich (both in features and cost) • I plan to check out Verification tools the next few days. Aer Cam Project

42. HDLs And Critical Systems • Verification is most significant issue (From my Perspective) • How do you verify the design to confidence • If you can’t verify to 100% how do you mitigate potential problems • Not talking SEUs but logic problems. • SEUs need their own examination Aer Cam Project

43. HDLs And Critical Systems • Synthesis tools can be very predictable -Use accepted inference rules -Combinatorial Logic Inside Clocked Process -Think about verification as a design requirement -Understand that logic can fail. If it is critical in nature, how is the failure dealt with. -These issues (logic and SEU) were around before Apollo but modern silicon densities and design complexities compound problems. Aer Cam Project