Miguel Correia Control and Data Acquisition Group IPFN-IST

1. ATCA/xTCA-based hardware for Control and Data Acquisition on Nuclear Fusion fast control plant systems Miguel Correia Control and Data Acquisition Group IPFN-IST

2. CDAQ in Fusion • Applications of CDAQ in modern Fusion plant systems • Equipment integrity and interlock • System state monitoring • Recording system control actions • Tokamak system operations control • Preparing plasma discharge • Ensuring plasma discharge quality • Record system performance • Human safety/security to enable control of the plant system

3. Requirements for fast CDAQ • Example of requirements for tokamak vertical stabilization of plasma • Measurement resolution • ENOB = 16 bit (ideally 18 bit). • Sampling rate: • 20 kSample/s , 200k Sample/s for micro-instabilities detection • 200 kSample/s as normal operation mode • Dynamic 20/200kSample/s with data decimation, Event distribution network required • Loop cycle latency (general rule for all PCS diagnostics) • Maximum of ~10% of the characteristic time of the phenomenon to be controlled. • To implement a prototype of a Fast Controller for Vertical stabilization the system shall have 150 acquisition channels. (with redundancy)

4. Need for MIMO control • Some actuators are driven by controllers using data from several different diagnostics. • Other (secondary control) diagnostics serve as backup in case of failure of the primary. • Many variables are shared between several controllers. • MIMO required – full-mesh support between boards and low-latency deterministic links between systems for global variable sharing.

5. CDAQ in Fusion • Challenges presented by modern Fusion experiments to the CDAQ subsystem • increasing number of interdependent parameters to be controlled • Increasingly faster loop-cycle response • Implications to the CDAQ • Massive processing power (parallel, multi-processing support) • High bandwidth for data-transfer • Advanced, intelligent, flexible timing & syncronization • Real-time multi-input-multi-output (MIMO) control • Steady-state operation for subsystems (including CDAQ) demands • High-availability • Failure prevention • Redundancy

6. CDAQ in Fusion • Engineering point-of-view • Modularity • Expandability • Programmability • Compatibility with high-performing industry standards • Multi-purpose • Integrate further developed hardware • Adopting commercial-off-the-shelf (COTS) products

7. ATCA • ATCA already successfully delivered valid solutions for MIMO CDAQ • Configurable network topologies • Supports various high-performance serial gigabit protocols • Multiple processor support • ATCA also provides important non-functional features, catering for the performance and security demands of such complex subsystems • hardware management • hot-swapping • fail-safe • large area form-factor and front-panel • large power dissipation capabilities • n+1 redundancy • Obstacles (hardware development) • Complexity – lenghty development • ATCA (as yet) still undefined for Physics/instrumentation applications

8. xTCA for Physics • xTCA – specification based on ATCA, specialized for Physics • Standardizes and facilitates hardware development for device operation in a Fusion control plant environment • Extensive, hot-swappable use of the Rear Transition Module (RTM) panel, for input-output (IO) HA • Concise data port assignment • Improved signals for timing and synchronization • (with compatibility between ATCA and AMC timing specs) • Zone 3 (RTM) connectors specification

9. Hardware design overview • builds upon previous ATCA implementations (e.g. For JET and COMPASS) • AMC carrier cards support • Targeting compliance with xTCA specification • Purpose: develop base prototypes for multiple aplications. • Focus on MIMO control – high-bandwidth, low-latency communications networks (e.g. PCIe, SRIO, hardware assisted GbE)

10. Target boards and networks • 3 types of boards 3 communications networks • PCIe-hubs PCIe network (dual-star) • Timing & Synchronization-hubs T&S network (dual-star) • Node blades Node blade network (full-mesh)

11. PCIe network • PCIe network • PCIe-hub (redundant set) • Dual-star (slots 1 and 2) • Handles data from every PCIe endpoint, in every blade of the shelf • 4 lanes (×4 PCIe Gen 1 or Gen 2)

12. PCIe data hub PEX 8696 (PCIe switch) • 96 PCIe Gen 2 (5 GT/s) lanes • up to 24 flexible ports • up to 8 upstream ports • on-chip NT port for dual-host • and fail-over applications • hot-plug support • 13 ×4 PCIe →ATCA fabric ch. • (PCIe network) • 4 ×4 PCIe → AMC • 1 ×16, ×8 PCIe → RTM • 1 ×4 PCIe → FPGA • Virtex-6 LXT FPGA • 128000 logic cells • up to 600 user IO ports • up to 20 6.6 Gb/s GTX transceivers • ATCA/AMC/ARTM clk switch • 5 ×1 SRIO → AMC/RTM • 1 ×4 PCIe → PEX8696

13. T&S network • Timing & Synchronization network • T&S-hub (redundant set) • Dual-star (slots 3 and 4) • distributes a customized group of timing and synchronization signals to all Node-blades. • LVDS clock/gate/trigger signals • Deterministic Gb timing link • 4 lanes (×1 SRIO / ×3 LVDS)

14. Timing & Synchronization hub PEX 8696 • 2×4 PCIe →PCIe network • 4 ×4 PCIe → AMC • 1 ×16, ×8 PCIe → RTM • 1 ×4 PCIe → FPGA • Virtex-6 LXT FPGA • 128000 logic cells • up to 600 user IO ports • up to 20 6.6 Gb/s GTX transceivers • ATCA/AMC/ARTM clock switch • 5 ×1 SRIO → AMC/RTM • 1 ×4 PCIe → PEX8696 • 11 ×3 LVDS → T&S network • 11 ×1 SRIO → T&S network

15. Node network (full-mesh) • Node blade network • P2P between every Node card • Full-mesh (slots 5 to 14) • Each port has ×1 SRIO / ×3 LVDS • allows, in conjunction with the PCIe and T&S networks, to the CDAQ subsystem to achieve real-time MIMO connectivity to every endpoint.

16. Node-blade PEX 8696 • 2×4 PCIe →PCIe network • 4 ×4 PCIe → AMC • 1 ×16, ×8 PCIe → RTM • 1 ×4 PCIe → FPGA • Virtex-6 LXT FPGA • 128000 logic cells • up to 600 user IO ports • up to 20 6.6 Gb/s GTX transceivers • ATCA/AMC/ARTM clock switch • 5 ×1 SRIO → AMC/RTM • 1 ×4 PCIe → PEX8696 • (9+2)×3 LVDS → Node/T&S • (9+2)×1 SRIO → Node/T&S

17. T&S-hub / Node-blade 2 boards-types • one PCB • one BOM • same assembly process • one DIFFERENCE: FIRMWARE • Virtex-6 LXT FPGA • ATCA/AMC/ARTM clock switch • 5 ×1 SRIO → AMC/RTM • 1 ×4 PCIe → PEX8696 • 11 ×3 LVDS → T&S network • 11 ×1 SRIO → T&S network • (T&S-hub) or (Node blade) • (9+2)×3 LVDS → Node/T&S • (9+2)×1 SRIO → Node/T&S

18. Overall ATCA fabric network setup Resulting ATCA fabric interface configuration, denoting the implemented network topologies and respective fabric channel connections.

19. Examples of use IO PCIe Over cable TRG PCIe CPU IO IO storage IO Clock sync PCIe CPU IO storage IO Clock sync PCIe CPU Events FPGA REFCLK Clk GEN IO Clk GEN Events FPGA REFCLK Clk GEN DSP IO Node T&S-hub PCIe-hub PCIe-hub T&S-hub

20. Summary • A hardware architecture solution for fast control plant systems was presented. • It is based upon a development of the ATCA specification, the most promising architecture to substantially enhance the performance and capability of existing standard systems, which has already delivered very encouraging results. • The hardware designed consists of 3 different prototypes. • Quad-AMC carrier format provides flexible, modular, expandable CDAQ. xTCA compliancy provides diverse IO solutions (e.g. IO just on RTM for improved hot-swapping capabilities), clock and trigger signaling and parallel processing, making it suitable for real-time MIMO control. • Being designed according to the currently available xTCA specifications, this hardware aims to offer a set of base prototypes which can perform highly on many future CDAQ applications, in a Fusion control plant environment.