pciGeneral PCI Device Support of EPICS which is used in TPS Control System

1. pciGeneralPCI Device Support of EPICSwhich is used in TPS Control System Presented by: Jenny Chen TPS Control Group June 14, 2011

2. Outline Goals of the Device Support – pciGeneral Rules of the Kernel Driver for PCI Cards Details of the Kernel Driver Programs Details of the Device Support Programs Performance Tests Access TLS control system via EPICS

3. Goals of the pciGeneral Programs Manipulate as many PCI cards as possible Manipulate the relevant resources Handle interrupts Convert byte order (big to little endian) if needed Keep maintenance programs to a minimum

4. Manipulate as Many PCI Cards as Possible Generalized kernel driver + pciGeneral + asynDriver(EPICS), all PCI cards we have were tested successfully The tested PCI cards: cPCI-7452, 24DSI32, cPCI-EVG-300, cPCI-EVR-300, TCP201 + DAC8402, TCP201 + DAC8415, TCP201 + ADC8417 Properties of PCI cards: address space: IO-PORT or MEMORY DMA byte order: little-endian or big-endian daughter boards One kernel driver may handle several cards of the same model Special kernel driver (without accessing hardware): creates shared memory – may exchange data between processes

5. Manipulate the Relevant ResourcesExtract 3 regions of the address spaces of a PCI card Fully access the address spaces plx -- the chip sets of PCI interface, ex: plx9000 series, the mostly accessed registers are interrupt control and status card – includes the main control/status registers of the PCI cards which are mostly for configuration data – separated from the main registers (card), ex: DMA, transient data or processed data (may be done by the kernel driver or a running process)

6. Handle Interrupts APs call read() to wait for the interrupts instead of using ioctl() or signal(), which is easy to program APs may directly access interrupt registers (plx), ex: reset the registers when interrupts are not handled properly Kernel drivers serve enable or disable interrupts that are requested by APs (ioctl)

7. Keep Maintenance Programs to a Minimum Keep the software simple and small Minimize the maintenance of IOC software One kernel driver program per card + 2 device programs Ex: kernel driver: pci_adl7452.c for cPCI-7452 EPICS device support programs: pciGeneralDrv.c, pciGeneralLinux.c

8. Rules of Kernel Driver Implementation Create char device file in /dev, file name ex: pciCardName-x, x: pci-slot, ex: pciadl7452-1, pciadl7452-2 ioctl() -- get the card information, enable/disable the interrupt, setup/start/abort DMA, special service for the card mmap() -- map the PCI address spaces into AP's read() -- block AP to wait for an interrupt

9. Details of a Kernel Driver Program Interface to a PCI card Create a char device interface file Serve char device access requests -- open/close/ioctl/mmap/read

10. Interface to PCI cardCreate char device interface file Declaration #define MY_VENDOR_ID 0x144A #define MY_DEVICE_ID 0x7452 #define SUB_VENDOR_ID PCI_ANY_ID #define SUB_DEVICE_ID PCI_ANY_ID #define MYCARD_NAME "pci7452" static struct pci_device_id MY_DRIVER_PCI_TABLE[2] __devinitdata = { {MY_VENDOR_ID, MY_DEVICE_ID, SUB_VENDOR_ID, SUB_DEVICE_ID, 0, 0, 0}, {0, 0, 0, 0, 0, 0, 0} }; static struct file_operations mypci_fops = { .owner = THIS_MODULE, .open = mypci_open, .read = mypci_read, .ioctl = mypci_ioctl, .mmap = mypci_mmap, .release = mypci_release, };

11. Interface to PCI cardCreate char device interface file Probe callback Create device interface file: - alloc_chrdev_region allocate char device resource, assign the char device name (should be unique) - cdev_jnit - cdev_add - class_create create a class in directory /sys/class, ex: /sys/class/pci7452 - device_create create a device in /sys/class/device, ex: /sys/class/pci7452/pci7452-1 to insert/remove kernel module dynamic – according to a rule listed in /etc/udev/rules.d, the udev daemon sends uevent to OS to create interface file: /dev/pciadl7452-1 (kernel version: linux2.6.13 or higher) Request to access PCI address spaces - pci_request_regions request pci resources and associate with the device file: pci7452-1 - pci_resource_start get pci address spaces information - pci_resource_end - pci_resource_len - pci_set_master to be the dma master Initialize a waiting queue for interrupt request and variables - init_waitqueue_head, atomic_set - kzalloc

12. Interface to PCI cardCreate char device interface file Remove callback Delete device interface file: - device_destroy destroy created device structure, ex: delete /sys/class/pci7452/pci7452-1 - class_destroy destroy created class structure, ex: delete /sys/class/pci7452 - cdev_del delete cdev structure the char device interface file will be removed then Release PCI resources - pci_release_regions release pci resources - pci_disable_device release pci access Free allocated memory - kfree

13. Serve the char device access requests Provide char device interface: open/close/ioctl/mmap/read - open allow AP to access the char-device, call request_irq - close release to access, call free_irq - ioctl serve command: IOCTL_CARD_INFO, IOCTL_PLX_INFO, IOCTL_DATA_INFO, IOCTL_INT_ENABLE, IOCTL_INT_DISABLE, IOCTL_DMA_SETUP, IOCTL_DMA_START, IOCTL_DMA_ABORT, IOCTL_INITIAL, IOCTL_COMMAND transfer PCI address information to AP, process interrupt requests, process DMA requests, special service request to the kernel driver - mmap map PCI address (memory type only) to user process address space: card, plx and data - read serve interrupt request: block AP, wake up AP and copy interrupt count to AP

14. Details of the Device Support Programs for EPICS They are based on asynDriver Programs: Header: pciGeneral.h pciGeneralDrv.c – the interface to EPICS, create records, register interrupt, parse arguments, ... pciGeneralLinux.c -- the interface to PCI cards, open/close device file, mmap pci address spaces, access the io-port/memory and create interrupt threads (call read() to wait)

15. EPICS Record types of pciGeneral bi/bo/mbbi/mbbo ai/ao/longin/longout Waveform with DTYP: asynInt32ArrayIn/Out, asynFloat32ArrayIn/asynFloat32ArrayOut asynFloat64ArrayIn/asynFloat64ArrayOut

16. INP/OUT Field of EPICS Record appendix DTYP: asynUInt32Digital for bi/bo/mbbi/mbbo . bi/mbbi INP: @asynMask(portName,byteOffset,bitMask,timeout) command location command: input/input16 location: plx/card/data . bo/mbboOUT: @asynMask(portName,byteOffset,bitMask,timeout) command location command: output/output16/write/write16 location: plx/card/data (output/output16 – get readback first) . bo (special) OUT: @asynMask(portName,0,0,timeout) command location actionState command: intEnable/intDisable/initial/dmaSetup/dmaStart/dmaAbortlocation: card actionState: 0/1/-1 (0/1:process when state is 0/1, -1:process every time) record(bi,"diBank0Ch0") { field(SCAN, “I/O Intr”) field(DTYP,"asynUInt32Digital") field(INP,"@asynMask(pci7452-1,0xc,0x1,1.0) input card") field(ZNAM,"low") field(ONAM,"high") } record(bo,"doBank0Ch1") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0x0,0x2,1.0) output card") field(ZNAM,"low") field(ONAM,"high") }

17. Db example st.cmd: pciGeneralDrvInit("tcp201-0","/dev/pcitcp201-0-IP")) pciGeneralDrvInit("adc8417-0","/dev/pcitcp201-0-IPB")) pciGeneralDrvInit("pci7452-1","/dev/pci7452-1")) pciGeneralDrvInit("er230-2","/dev/er230-2")) appendix record(bo,"enableInt7452") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0,0,1.0) intEnable card 1") field(ZNAM,"null") field(ONAM,"enable") field(DOL, "clearIntBank0") field(OMSL, "1") field(HIGH, “0.1”) field(FLNK, "enableIntBank0") } record(mbbi,"erClockControlSts") { field(DTYP,"asynUInt32Digital") field(INP,"@asynMask(er230-2,0x050,0x0000ffff,1.0) input card") field(DESC, "Clock Control Status") } record(mbbo,"setIntMaskBank0") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0x8,0xffffffff,1.0) write card") }

18. INP/OUT Field of EPICS Record appendix DTYP: asynInt32 for ai/ao/longin/longout . ai/longin INP: @asyn(portName,byteOffset,timeout) command location extractMask command: input/input16, location: plx/card/data, extractMask: extract several bits of 32/16 bits . longin (special) INP: @asyn(portName,0,timeout)intCount card . ao/longout OUT: @asyn(portName,byteOffset,timeout) command location extractMask command: output/output16/write/write16/ioctl, location: plx/card/data extractMask: extract/update several bits of 32/16 bits record(longout,"erPrescaler0Set") { field(DTYP,"asynInt32") field(OUT,"@asyn(er230-2,0x100) output card 0x0000ffff") } record(longin,"diBank0") { field(SCAN,"I/O Intr") field(DTYP,"asynInt32") field(INP,"@asyn(pci7452-1,0xc) input card 0xffffffff") } Ex: request kernel driver service: Ioctl card code parameter Code may be the pv value record(longout,"$(IOC)-adc8417-$(S):ioctlCmd") { field(DTYP,"asynInt32") field(OUT,"@asyn(adc8417-$(S),0x0) ioctl card 0 $(HNUM)") field(DOL, "$(IOC)-adc8417-$(S):dataProc") field(OMSL, "1") field(FLNK, "$(IOC)-adc8417-$(S):ch0WfProc") } record(longin,"intCountEg230") { field(SCAN,"I/O Intr") field(DTYP,"asynInt32") field(INP,"@asyn(eg230-3,0x0) intCount card") }

19. INP/OUT Field Format of EPICS Record DTYP: asynInt32ArrayIn/asynInt32ArrayOut for waveform . waveform input INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset command: inArray/inArray16, location: card/data . waveform output INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset command: outArray/outArray16, location: card/data IncreaseAddrOffset depend on the data layout: typeA ch0ch0ch0... one data block for one channel (addrOffset:2/4) ch1ch1ch1... typeB ch0ch1ch2... one data block for N channels (addrOffset:2N/4N) appendix record(waveform, "dac8402ch0WfOut") { field(DTYP, "asynInt32ArrayOut") field(INP,"@asyn(dac8402-1,0x0,1) outArray16 data 0xffff 2") field(NELM, "2048") field(FTVL, "LONG") field(FLNK, "dac8402ch0Wf") } record(waveform, "dac8402ch0Wf") { field(DTYP,"asynInt32ArrayIn") field(INP,"@asyn(dac8402-1,0x0) inArray16 data 0xffff 2") field(NELM, "2048") field(FTVL, "LONG") field(PINI, “YES”) }

20. INP/OUT Field Format of EPICS Record appendix DTYP: asynFloat32ArrayIn/asynFloat32ArrayOut for waveform INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset factor offset input waveform command: inArray/inArray16, location: card/data output waveform command: outArray/outArray16, location: card/data IncreaseAddrOffset: depend on the data layout record(waveform, "dac8402ch0WfE") { field(DTYP,"asynFloat32ArrayIn") field(INP,"@asyn(dac8402-1,0x0) inArray16F data 0xffff 2 3.051e-4 -10") field(NELM, "2048") field(FTVL, "FLOAT") } Refer to: http://www.icg.nsrrc.org.tw/wiki/EPICS_Drivers

21. Performance Tests - Memory access reduces about 25% CPU time compared to ioctl int_value = *((int *)mapped_ptr ; vs. ioctl(fd, IOCTL_READTEST, &int_value) ; ADLINK cPCI-6910: CPU:dual-core Intel Xeon 2GHz, cPCI:64bit/66MHz - Interrupt service test – EPICS client: get the 5ms interrupt counter by LabCA (matlab) Jitters: . OS scheduler . Dmc controller + network

22. Interrupt Service Test - Interrupt test: the response time of a 5ms interrupt triggered by a network process(measured by an AP) EPICS Client LabCA Prepare for insertion devices control: the field compensation by setting correction coils (setting follows gap/phase) needs to be done as fast as possible. Control Network EPICS ioc Wake up (read) open/mmap Shared memory kernel driver char dev file: /dev/dmc4000-0 open/mmap Data receiver (from DMC4000) ioctl Ethernet kernel driver (socket interface UDP) Ethernet as field bus DMC4000 controller send data UDP at every 5ms

23. Interrupt Test with cPCI-7452 DIO Card - Hardware interrupt test: connect a TTL signal to a DI channel, the state change would trigger the PCI interrupt, the figures show the response time of 100Hz and 200Hz signal; the reason of some missing changes was unknown but it was probably not caused by the OS scheduler (maybe the interrupts were not delivered) Test condition: 1. hi/low signal connect to ch0 2. testing loop: 50000 3. test process priority adjusted ADLINK cPCI-6910: CPU:dual-core Intel Xeon 2GHz, cPCI:64bit/66MHz Test Program open/mmap/read Wake up cPCI-7452 kernel driver char dev file: /dev/pci7452-0 Interrupt cPCI-7452 DI Card 200Hz/100Hz

24. EPICS ioc and LabCA Monitor Test - update test: the ioc updates data at every 5ms, the interrupts trigger the ioc to update the dmc4000 and cPCI-7452 DIO channels Figure 1: ioc CPU load: 8%(top/Irix), db record: about 40 ADLINK cPCI-6910: CPU:dual-core Intel Xeon 2GHz, cPCI:64bit/66MHz EPICS Client LabCA Ethernet Note: in figure 2, 128 DI records scaned but not in figure 1 EPICS ioc Wake up Wake up Shared memory kernel driver char dev file: /dev/dmc4000-0 kernel driver char dev file: /dev/pci7452-0 Receive data sent by DMC4000 open Figure 2: ioc CPU load: 20%(top/Irix), db record: about 170 mmap Ethernet kernel driver Interrupt cPCI-7452 DI Card Private Ethernet 100Hz DMC4000 controller send data UDP at every 5ms

25. Access Existing TLS Control System via EPICS Modify pciGeneral to tlsControl, because the data byte order of TLS control system is not standard Start up a softIOC(+tlsControl) on a TLS console as a gateway A simple char device driver (kernel) allocates memory to store DDB which is updated by TLS ILCs and serves file operations: open/read/ioctl/mmap TLS DDB-receiver flushes the DDBs into the memory which is also mapped by the softIOC, then two control systems access the DDB simultaneously With the kernel driver running, the softIOC may be waked up by the interrupt which is triggered by the DDB receiver in order to update data synchronously To control TLS devices from EPICS, streamDevice is used, a server waits for the string commands (name+value) then converts it (access SDB of TLS) to the format that ILC accepts

26. The Conjunction for TPS and TLS Control System Access any device of TLS control system via EPICS Shared memory kernel driver char dev file: /dev/ilc-x TLS SDB/DDB File interface mmap() ioctl() File interface mmap() read() Wake up TLS control software on console Setting server for EPICS EPICS IOC TLS Console TLS setting functions StreamDevice TCP socket interface On-demand Setting TLS Network TPS Network 10Hz DDB TLS ILCs Run EPICS ioc on an ILC of TLS control system TLS Network DDB Upload Setting server TLS ILC control system EPICS IOC TCP socket Setting server (ca-lib) for ILC TLS ILC kernel driver char dev file IO hardware

27. Summary Efficiency Flexibility Maintenance

28. Acknowledgement Marty Kraimer Thank you for your attention!