CONTROL SYSTEM FRONT-END HARDWARE UPGRADE

1. Stanford Linear Accelerator Center CONTROL SYSTEMFRONT-END HARDWARE UPGRADE Eric J. Siskind March 7, 2002

2. Project Goals • Replace Multibus-I micro hardware with commercial-off-the-shelf personal computers. • Replace special purpose accelerator networks (SLCnet, KISnet, PNET) with a single commercial-off-the-shelf TCP/IP network such as switched gigabit Ethernet. • Develop personal computer CAMAC interface to replace MBCD/MBCD-II.

3. Project Impetus • Multibus-I difficult to upgrade, maintain, and support. • Not able to take advantage of Moore’s Law increases in CPU speed, memory size, and network bandwidth for ~10 years. • Two orders of magnitude increase in that time: • CPU clock frequency from ~20 MHz to ~2 GHz; • Memory size from ~10 megabytes to ~1 gigabyte; • Network bandwidth from 1-10 to 100-1000 megabits/second.

4. Front-End Real-Time TCP/IP Networking • Network must deliver 120/360Hz real-time micro-micro traffic in preference to slower Alpha-micro (or generic front-end client to micro) packets. There is currently no expertise in meeting this challenge on the SLAC site. • Single network is preferable for reliability and cost of purchase/maintenance, but use of a separate real-time network at an early stage is possible.

5. New CAMAC Features • CAMAC interface differs from MBCD/MBCD-II in four major ways: • Two board design to support remote I/O; • Real-time preemptive hardware with recovery; • Maximizes CAMAC utilization when user software environment in PC is multi-threaded; • Large (i. e. comparable to existing micro) user programming environment within interface.

6. Hardware Architecture • Hardware consists of two circuit boards connected by up to 10 km of single mode fiber optics: • PCIL (“PCI Link”) board plugs into PCI option slot in PC; • FECC (“Front-End CAMAC Controller”) is double-width CAMAC module. • PC moves from micro’s current location to MCC computer room, leaving behind only FECC.

7. PCIL Features • Basically an intelligent special-purpose Network Interface Card. • Processor with DMA PCI block transfer master and DMA fiber optic link transmitter/receiver. • Makes no assumptions about nature of PC’s real-time operating system. • Semi-static assembly language PROM code with downloaded updates. • First generation hardware doubles as Alpha SLCnet interface.

8. PCIL Code Functions • Supports PC FECC message passing. • Supports FECC remote read/write access to PC memory with optional PC interrupt. • Performs function-specific scatter/gather DMA to/from PC memory for all classes of operations, avoiding unnecessary buffer copying by PC CPU. • Forwards trigger pattern from PCI registers to FECC (only in 1st generation; pattern forwarding in hardware in 2nd generation).

9. FECC Features • Logical successor to MBCD/MBCD-II. • Accesses host memory via point-to-point fiber optic link instead of parallel I/O bus. • Gets power from CAMAC backplane and timing triggers from PDU via auxiliary backplane. • 1st generation supports MBCD-II’s four strings of SLAC serial CAMAC crate controllers (5 MHz, bit serial, half duplex). • 2nd generation adds support for four strings of IEEE standard crate controllers (5 MHz, byte serial, full duplex) and Bitbus.

10. All PCs in MCC computer room have only three connections to outside world: Front-end TCP/IP network interface; MPG IDOM connection to hard reset button (replaces PNET reset); Fiber optic connection to FECC. Only FECC connection is specific to a particular function; i. e. only attachment to PR02’s FECC converts an arbitrary PC on front-end network into PR02. Rapid Micro Replacement

11. Micro Replacement II • MCC computer room patch panel has fiber optic links to all PCs’ PCILs and all regions’ FECCs. • Switching to a spare PC requires changing TCP/IP address, updating IDOM bit number, and moving fiber optic patch cord. • First two accomplished via logical  physical micro association table.

12. Real-Time Constraints • All accelerator operations based on 360 Hz timing; 2.78 milliseconds/pulse. • Many CAMAC operations must be performed on each pulse, or on a particular pulse (e.g. F(19) PPYY broadcasts, BPM data acquisition). • Real-time access to CAMAC for such time-critical usages is currently maintained by limiting all MBCD requests for groups (“packages”) of CAMAC cycles from non-time-critical micro jobs to one millisecond. Only one such micro job may have a single package outstanding in the MBCD at any given time.

13. PCIL-FECC Real-Time CAMAC Access Scheme • All time-critical operations either employ hardware that is distinct from that used for non-time-critical operations, or else can interrupt non-time-critical operations on shared hardware. • All software executing in PCIL or FECC runs under a real-time executive with interrupt-driven prioritized preemptive scheduling.

14. Real-Time Latency • Shared PCI hardware released in three PCI clock ticks (90/45 ns on 33/66 MHz bus). • Shared 2nd generation fiber optic link accessed on next 4.9 microsecond cell (plus second cell if pattern forwarding required). • Shared CAMAC hardware released in one CAMAC cycle (~10/~4 microseconds on SLAC/IEEE standard cable) unless locked. • Real-time executive switches processes in 3-5 microseconds (plus similar interrupt latency if context switch already in progress).

15. CAMAC Interrupt Recovery & Control • If a CAMAC block transfer is interrupted, hardware can repeat the previous write operation (address pointer load?) with updated write data before resuming the transfer. • Hardware can prevent an interrupt between CAMAC cycles transferring two 16-bit halves of a single 32-bit word. • Hardware can prevent an interrupt of a particular block transfer entirely (needed because of a current PIOP feature).

16. MBCD-II Parallelism • Multibus-I backplane bandwidth is ~1 megabyte/second (2 microseconds/16-bit word). • SLAC serial CAMAC bandwidth is ~¼ of that (~8 microseconds/16-bit word in block transfer mode). • Solution to limited bandwidth problem was to operate 4 CAMAC cables in parallel.

17. MBCD-II Performance Limits • MBCD-II firmware only looks at one package at a time, so a package must use multiple crates on different cables in order to achieve parallelism. • Non-time-critical tasks often require multiple packages to read one module, so no parallelism is achieved (e.g. analog status can’t read an entire SAM from collider arcs in a single package). • Time-critical operations with small numbers of words/crate don’t realize full bandwidth because of limited Multibus-I package fetch bandwidth.

18. Package Fetch Improvements • FECC CAMAC hardware has much higher ratio of local memory/CAMAC bandwidth to avoid package fetch bottlenecks. • Very large PCI DMA bandwidth between PC memory and PCIL local memory, and link DMA bandwidth between PCIL local memory and FECC local memory: • Straw between FECC memory and CAMAC; • Fire hose between PCIL memory and FECC memory; • River between PC memory and PCIL memory.

19. System Performance • Many PC jobs now can have simultaneously outstanding long CAMAC packages. • All packages are sent to FECC firmware ASAP, and parsed into their individual (block transfer) operations in a single crate (“packets”). • Packets are immediately queued to hardware for the cable accessing the target crate. • Cable operation begins as soon as any outstanding package has a packet requiring access to a crate on that cable. • Parallelism now achieved over multiple packages.

20. MBCD Intelligence • Set of four 2911 4-bit slices to make 16-bit microsequencer (AMD 2900 series of bit-slice microprocessors). • Code from a 32 x 16 bit PROM via a single 82S105 FPLS. • 5 MHz FPLS clock; 2911 clock generated by FPLS at lower frequency as required.

21. MBCD-II Intelligence • Analog Devices ADSP-2101 12.5 MHz 16-bit DSP. • 2k x 24 bit on-chip program memory; 1k x 16 bit on-chip data memory; 8k x 8 bit external program/data memory (unused). • Actual application program uses no more than a few hundred instructions and very little data memory.

22. PCIL/FECC Intelligence • Analog Devices ADSP-21060 “SHARC” 40 MHz 32-bit DSP on each board. • 40k x 48 bit on-chip program memory; 64k x 32 bit on-chip data memory. • 512k/2048k x 48 bit external program/data memory (256k/512k x 48 in 2nd generation PCIL). • External DMA I/O data memory: • 512k/2048k x 32 bit in 1st generation PCIL; • 256k/512k x 64 bit in 2nd generation PCIL; • 1024k/2048 x 32 bit in 2nd generation FECC.

23. User FECC Programming • Real-time executive plus MBCD/Bitbus emulation uses <10% of on-chip program memory and most of on-chip data memory. • Remaining minimum of 7+ megabytes of memory on 40 MHz 32-bit processor similar to existing 386/486 CPU boards. • Move existing 360 Hz interrupt driven code from micro to CAMAC interface to minimize CAMAC access delay. • C run-time environment with dynamic heap allocation; full environment and stack preserved over context switch.

24. 1st Generation Hardware • PCIL uses 32 bit/33 MHz PCI slot. • Separate fiber optic links for real-time/non-real-time PCIL  FECC communication. • Each link runs at 12.5 megabytes/second half duplex; back compatible with 10 megabyte/second BIPI link (SLCnet backbone). • PCIL peeks and pokes at FECC memory. • PCI master and link are FPGA-based DMA peripherals of PCIL SHARC. • CAMAC is programmed I/O peripheral of FECC SHARC, but link uses SHARC DMA channels.

25. 1st Generation Status • Hardware based on SHARC plus FPGA-based peripherals (PCIL-3; FECC-2). • PCIL debugged in 1999; FECC in 2000. • Five PCIL/FECC board sets, plus 3 PCILs for Alpha SLCnet interface. • Now intended primarily for development: • Link protocol requires 3-4 round trips per message; bandwidth & latency degrade with distance. • Link TAXI chips no longer available.

26. 2nd Generation Hardware • PCIL2 uses 64 bit/66 MHz PCI slot. • Single specialized fiber optic link for PCIL2 FECC2 communication (NLC SBIR). • 125 megabyte/second full duplex link, but 9/19 used for hardware overhead. • All peripherals FPGA-based DMA except for Bitbus (programmed I/O). • Hardware based on SHARC plus one large FPGA per board.

27. 2nd Generation Link • All traffic divided into 608 byte cells. • 320 bytes of user data (real-time, non-real-time, pattern, no-op/zero); 32 bytes of hardware header; 256 bytes of error correction code. • Cell requires 4.864 microseconds; can withstand 128 consecutive bytes or > 1 microsecond of lost data in that time. • Klystron modulator pulse is > 5 microseconds wide with < 1 microsecond rise and fall times; can survive loss of all data on both rising and trailing edges of a single modulator pulse.

28. 2nd Generation Link II • If uncorrectable errors, hardware timeout and retransmission of all lost cells; 100 microsecond (20 cell lengths) typical timeout length. • Hardware flow control for 16 real-time receive buffers and 16 non-real-time receive buffers. • No start-of-message sequences to be corrupted by noise after initial turn-on. • Hardware header supports remote manipulation of initialization register; local and remote hardware error counters.

29. Link Header Format • All fields 16 bits unless noted. • Command : • 4 bits - destination buffer number; • 1 bit - first cell in buffer; • 1 bit - last cell in buffer; • 2 bits - cell type (Pattern/R-T/N-R-T/NOP); • 1 bit - remote initialization register write enable; • 7 bits - unused. • Offset in buffer to end of cell, i. e. cumulative buffer length including this cell (64 byte chunks).

30. Link Header Format II • Remote initialization register write data (32 bits). • 16 non-real-time receive buffer available bits (backpressure flow control). • 16 real-time receive buffer available bits (backpressure flow control). • 16 non-real-time receive buffer filled handshake bits. • 16 real-time receive buffer filled handshake bits.

31. Link Header Format III • Outgoing cell sequence number. • Next expected incoming cell sequence number. • Multiple cells acknowledged event count. • Received cell out-of-sequence error count. • Timeout error count. • 8B/10B decoder error count. • Reed-Solomon corrected error count. • Uncorrectable Reed-Solomon error count. • Outgoing and incoming versions of last 8 counters readable from SHARC.

32. 2nd Generation Status • PCIL2 link now debugged; PCI master not yet tested. • Two PCIL2 boards built in 2001; three more in 2002. • FECC2 design in progress; complete by mid-summer; debugged by end of CY02. • Initial build of two FECC2 boards in 2002; also requires I/O concentrator panel (too many connectors for CAMAC front panel).

33. 3rd Generation Design • Replace SHARCs by PowerPC embedded in each board’s single FPGA. • Time scale: FPGAs available in CY02; begin in CY03(???). • Benefits: • Commercial real-time executive with improved software development tools; • Clearly the upcoming embedded hardware/software technology. • Shortcomings: • Need more time; • Need more money.

34. SLAC CAMAC Bandwidth • ~7.4 microseconds per 16-bit word in block transfer mode. • Add round trip propagation delay to target crate controller for each word (up to 2.7 microseconds in SLC arcs). • Add 1.6 microseconds for 24-bit data. • Subtract 3.2 microseconds for control operation (F8=1). • Add 3.6/4.8 microseconds for new CNAF in read/write operations.

35. IEEE Standard CAMAC Bandwidth • 3.6 microseconds per 16/24 bit word for read/write with new CNAF on each cycle. • Add round-trip propagation to furthest crate controller. • Subtract 0.8 microseconds for control operation. • Includes parity on each byte and message check byte. • 2-3 times faster than SLAC serial CAMAC controllers.

36. Pipelined IEEE Standard CAMAC Mode • 1.0 microsecond per additional 16/24 bit read/write cycle (full CAMAC crate bandwidth). • No change to CNAF allowed. • No round-trip propagation delay degradation. • Order of magnitude improvement over SLAC serial CAMAC controllers. • Only supported by Kinetic Systems model 3952 “enhanced” L-2 crate controller.

37. 1st Generation Bandwidth • ~1 megabyte/second bit serial CAMAC bandwidth (four SLAC CAMAC cables combined) (“straw”). • 12.5 megabytes/second for each link; only one of read/write active at any time on each link (“fire hose”). • 133 megabytes/second PCI master in block transfer read/write (“river”).

38. 2nd Generation Bandwidth • 8 megabytes/second enhanced byte serial CAMAC bandwidth (four IEEE CAMAC cables combined). • ~1 megabyte/second bit serial CAMAC bandwidth (four SLAC CAMAC cables combined). • <0.05 megabyte/second Bitbus bandwidth. • 125 megabytes/second link burst rate in each direction; maximum average rate is 10/19 of this or 66 megabytes/second. • 533 megabytes/second PCI master in block transfer read/write.

39. Bitbus Master • Existing master based on MCS-51 2 MHz 8 bit microcontroller with SDLC/HDLC serial link. • 8044BEM is an MCS-51 with mask-programmed firmware (iDCX-51 RTE plus Bitbus application). • FECC2 has programmed I/O hardware (one SHARC interrupt/32 bits) to send a packet to one slave and receive that slave’s response. • Poll list, counters, etc. maintained by new SHARC code in FECC2. • Command queuing no longer an issue.

40. Software Group Projects • Build, boot, and debug iRMX-III in off-the-shelf personal computers. • Make real-time micro  micro data communications streams coexist with slower back-end  front-end traffic on switched TCP/IP network. • Move 360 Hz interrupt processing from micro to FECC.

41. Software Projects II • Change control of 360 Hz interrupt processing from shared memory to network link model. • Modify fast feedback actuator to specify real-time priority for CAMAC operations. • Modify slower micro jobs to send larger packages to CAMAC interface to maximize parallelism. • Modify klystron job to protect PIOP CAMAC operations from interrupts.

42. Software Projects III • Support logical to physical micro mapping. • Support PC access to file server delivering current versions of PCIL and FECC executable images appropriate to different types of micros.

43. Schedule • SWE development effort is currently in progress (TEG/RCS). • All optimizations not necessary at early stage. • Trying to deploy 1-2 PC-micros in production system during accelerator shutdown beginning this July. • Begin AIP with 2nd generation PCIL-FECC in FY03/04.

44. Simplified Hardware Configuration • No PCPC fast feedback (KISnet equivalent) network traffic. • Use 1st generation PCIL-FECC hardware: • No IEEE CAMAC or Bitbus; • Only need 32 bit/33 MHz PCI slot. • MPG  PC (PNET equivalent) pattern broadcast network traffic on separate real-time TCP/IP network. • PC-based MPG (MP10/MP11) gets pattern from MP00/MP01 via reflective memory link.