Aug. 20, 2002 L.B.L.

1. US ATLAS SCT ROD FDR Aug. 20, 2002 L.B.L. DSP Software Status DSP-BIOS Update New DSP Packages • The Task Manager • The Event Manager (slaves) Overview of DSP Primitives & Tasks • Common to Both DSP Types • Slave Primitives & Tasks • Master Primitives & Tasks Histogramming Tasks Sequence Douglas Ferguson UW-Madison & LBL

2. DSP-BIOS • New TI “operating system” for their DSPs. • Upgraded twice within 1 year: 1.1 to 1.2 to 2.0. Status: • The DSPs are fully functional under version 1.2. • Upgrade to 2.0 has been done; preliminary testing performed. • Further testing planned in September. Issues to be resolved: • BIOS is resource intensive. Many functions are inserted in IPRAM, • using ~26% of available memory; many of these do not need to be • there but version 1.2 does not have a mechanism for their placement • elsewhere. The DSP software would benefit if more could be placed • in IPRAM. • A similar situation exists for the IDRAM.

3. Idle Idle yes List inserted? (host sets bit) ready to Run? Host cleared inList bit? (clear buffer occupied bit) no yes no no Acknowledged Abort? Resume? yes Buffer occupied? Buffer read? (readReqest reset) yes yes (set busy, executing bits) no yes Executing: one primitive/ loop iteration no Paused Waiting Frozen Prepare reply data no yes yes read request? Pause list? yes Finished/ Abort list? no State Machines Primitive List Execution Text Buffer Sending Two types: Host List & Inter-DSP List

4. DSP Packages Task Manager: • Coded & Tested. • Maintains a list of functions that execute continuously on DSP, and • provides updates on their status. • Slave DSPs: event-based functions. Master DSP: control functions • Functions are added/removed from list by primitives: startTask, • taskOperation (inspection, pause, resume and stop). Event Manager: • Coded & Tested. Sequence: a) Copying of incoming event data frames into DSP memory (ISR). b) Buffer overflow handling. c) Event building and insertion onto queue for distribution. d) Removal of event from queue after processing.

5. Primitives vs. Tasks Primitives: • Generally are executed once. Primitives can return a value that requests that the • primitive be repeated (say, for polling), but only one primitive executes at a time. • Return data via the reply list. Each primitive in the input list contributes a header • block to the reply, and optionally adds data, which can either be structured or • unstructured. • Generally are simple. A primitive list is a “program”, and primitives are its functions. Tasks: • Execute continuously, and together; all tasks are executed once per iteration of the • main program loop. Several tasks will happily co-exist on a DSP. Tasks can be • paused & aborted, just like primitive lists. • Return data asynchronously. When a task completes, it can send an info message to • the host. Any output data can be transferred via a primitive. sendData will handle • large chunks of data processed by tasks & primitives (events, histograms, fits etc.); • taskOperation:stop or query return the task’s internal variables. Some tasks • (histogramming) also report via registers. • Typically are more complex. Tasks can have several different internal states; the • task manager will report the states of any running or completed/halted tasks in the • running task register (idata).

6. Beginning of Fragment (9 words total, 3 shown) B0F00000 EE1234EE 00000009 ………… ………… ………….. ………… ………… ………… 21008E91 9CF09EB0 ……….. ………… ………….. ………… ………… ………… ………….. ………… ………… ………… ………….. ………… ………… ………… ………….. ………… ………… Link Header Link Data End of Fragment (6 words, last shown) ………… ………….. ………… ………… ………… ………….. ………… ………… ………… ………….. E0F0000 ………… ………… ………….. ………… ………… ………… ………….. ………… ………… ………… ………….. ………… ………… ………… ………….. ………… 40FF0127 Word 27  Inserted by router  Trailer Marker Word Event Management Event based data allows the DMA ISR to complete its work quickly, since the event’s length is always found at the end of a frame. The expense is that small Events take up more space. Half-word 0x40ff should never occur in data; the router adds the event’s length to this to form the Trailer Marker Word. Event Based Data 2 frame event (1 frame= 0x100 words): • ISR fills in a set of “frame registers”, and an auxiliary • set of “frame summary registers.” (discussed next) • These are located in slave IPRAM at an (~) fixed location, and can be monitored to keep track of the DSP’s activity. Event queuing sequence: • Header is detected: new event, begin filling in its data. • Any frames until trailer is found are marked in the • event’s frame mask. • Trailer is detected: event goes onto event queue.

7. Event Management (cont) Events in memory: 0x80008000 – 0x80010000 “burst buffer”, 0x02092000 – 0x020d2000 “external burst buffers” 8 frame sets, + 1 reserve. Frame set 0  IDRAM, all others are in SDRAM • Each frame set contains 32 frames. • Two register sets describe them: Frame summary registers: • One per frame set, used by the DMA ISR to find where to put the next incoming • data frame. Show at a glance the DSP’s burst buffer usage (one bit per frame). Frame set registers: • 32 per frame set, used by the event manager to jump from one frame to the • next when processing multi-frame events, and determine which tasks to give them to. • defined as 0xSSSD DDDD PPPP PETO FFFF FFFF IIII IIII Frame set delta Delta to Next within Frame set Pending tasks Evt frame # Internal ID Overflow bit Trailer bit Error bit

8. Event Management (cont) • The frame register sets are located in the “idata” memory section (0x80000040- 0840), • for monitored program variables). Also in idata: Event Manager Control structure (small set of control & ISR variables). Overflows: If the IDRAM burst buffer fills up (with a giant event (typically), or many events arriving more quickly than they can be processed), a buffer overflow occurs. All IPRAM frames are copied into SDRAM, and re-organization must be done on the frame set registers, and event frame masks.  EXPENSIVE.

9. Common Primitives: Set Err Msg Mask Event Trap Setup Set LED Set Memory Pause List Flash LED Copy Memory Echo Memory Test Send Data Start Task Task Operation Module Mask Set Trigger Write Buffer Slave Primitives & Tasks: Start Event Trapping Stop Event Trapping Histogram Setup Histogramming Event Trapping Occupancy Counting Tasks: Error Counting Link Re-synchronization Master Primitives & Tasks: Start Slave Executing Transmit Serial Data R/W Register Field Start Slave List Configure Slave R/W Module Data Poll Register Field Send Slave List Send Configuration R/W FIFO R/W Slave Memory Slave List Operation Send Stream Build Stream Tasks: Histogram Control Memory Mirroring Trap Request Monitoring

10. Common Primitive Summaries: Primitive: Summary: Status: Echo: A primitive “ping”. Hands back as a reply whatever you give it. Useful for testing DSP response time, DSP buffers. • Coded & Tested. Set Err Msg Mask: Sets the default mask for printing messages in the error buffer. • Coded & Tested. • Coded & Tested. Pause List: Pauses a primitive list when encountered during its execution. The list can be resumed by setting the Resume List bit in the command register. Set Memory, Copy Memory: Utility primitives for organizing DSP memory. They operate on DSP memory and do not return any data. • Coded & Tested. Memory Test: Diagnostic primitive which uses some basic memory tests to check memory integrity. • Coded & Tested.

11. Common Primitive Summaries (cont): Primitive: Summary: Status: Set Led, Flash Led: Diagnostic primitives which are used to visually verify the state of the ROD, can be used for precise timing tests as well. • Coded & Tested. Write Buffer: Diagnostic primitive which writes the input data in the selected buffer. • Coded & Tested. Start Task: Starts a task, using the input data. The task must not be currently running on the DSP. The order in which tasks run is determined by their priority (input). • Coded & Tested. Task Operation: Perform an operation on a currently running task. Currently these are stop, pause, resume, query & reset. Querying a task returns its current data (internal variables). Stopping and resetting it also return data. • Basic functionality • coded & tested. • Data return not • fully tested.

12. Common Primitive Summaries (cont): Primitive: Summary: Status: Send Data: Returns a block of data which has been processed by a primitive or task. The data is well defined, but with a variable length. If the data is large (histograms), returns a pointer to it; the host will then ask the master DSP to shut down commun- ications to the slave DSPs so that it can retrieve it (currently this is manual). Data types include: master DSP- mirror, router & stream data; slave DSPs- event, histogram, occupancy & fit data. • Partially coded • (coded as needed); • text in red is not yet • fully implememted. • Partially coded, • untested. Module Mask: Perform all the steps necessary to set the front-end module masks. Transmit information to slave DSPs. Set Trigger:Primitive version of the histogram control task. Differ- ences between them are that set trigger only does a single bin, and can be set to send triggers at an input rate. Also runs locally on slave DSPs. • Coded, being • tested. • Coded & Tested. Event Trap Setup:Initialize the event manager on the slave DSPs; the master DSP coordinates the slave DSP & the Router.

13. Slave Primitive Summaries: Primitive: Summary: Status: Start Event Trapping: Once Event Trap Setup has been run, this primitive is run to load the DMA engine, and enable the router’s Trap Enable flag (controlled by slave DSPs). • Coded & Tested. Stop Event Trapping: Stops the DMA engine, and disables the router’s Trap Enable flag. • Coded & Tested. Histogram Setup: Sets up histogramming variables (# and size of bins, placement of histograms, etc.), and initializes the histograms. • Coded & Tested. • Some revisions • requested.

14. Master Primitive Summaries: Primitive: Summary: Status: Read & WriteUsed to set any of the of the ROD’s FPGA RRIF Register Field: Rod Register InterFace) registers. The ROD Operations Manual explains these. • Coded & Tested. Poll Register Field: Some RRIF registers are self-clearing. This primitive monitors them to see when they have done so. • Coded & Tested. Read & Write FIFO: Places data in any of the ROD’s on-board FIFOs; this is used in conjunction with R&W Reg. Field to trigger the FIFOs to play their data into the desired ROD FPGA (formatters, EFB, or Router). • Coded & Tested. • Coded & Tested. Start Slave Executing: After loading the slave’s IPRAM, IDRAM and external memories, SSE sets them running. Once running, the board must be reset to stop them! (the standard list does this). Configure Slave: Turn communications with a slave DSP on and off. • Coded & Tested.

15. Master Primitive Summaries (cont): Primitive: Summary: Status: R/W Slave Memory: Reads from or writes to specific locations in the slave DSPs’ memory. The data length must be less than ~150KB, the size of the primitive list buffers. • Coded & Tested. • Coded & Tested. Start Slave List, When slave DSPs execute host lists, the master Send Slave List, DSP serves as the intermediary, doing all the Slave List Operation actual list coordination using these primitives. Transmit Serial Data: Transmit the input data out of serial port 0, 1 or both, and optionally capture the data in the ROD input memories. • Coded & Tested.

16. Master Primitive Summaries (cont): Primitive: Summary: Status: Build Stream: Build up a command stream from input command & data lists, which can contain up to 6 commands at a time; also used to reset a stream buffer. Successive calls to buildStream will concatenate lists. Either of 2 stream buffers can be used, or both if data is to be transmitted from both serial ports. Commands can be any of the SCT delay, fast, slow, data or mask commands. • Coded & Tested. • Request for • additional cmds. • Add in equivalent • Pixel commands. Send Stream: Sends any data contained in the stream buffer(s) out of the requested serial port(s), with optional capture. • Coded & Tested. Read & Write Module Reads from or writes to a C structure, stored on Configuration: the master DSP, which contains all information necessary to configure an SCT module. • Coded & Tested. • Add in equivalent • Pixel structure. Send Configuration: Sends a serial stream constructed using the structure information to the specified module(s). Two modules can be configured simultaneously. • Coded & Tested. • Add code for dual • module config- • uration.

17. Slave Task Summaries: Task: Summary: Status: Histogramming: Create histograms from incoming events, either alone using pre-set binning, or in cooperation with the histogram control task (or Set Trigger). • Coded & Tested. • Modify for Pixels. • Currently the histogramming engine is written in C, and is too slow. • I am re-writing it in assembly language to improve performance. • Coded & Tested. Event Trapping: Trap (copy) incoming events of the desired type into a trapping buffer. The router must be set properly beforehand. • Currently this is a stand-alone slave task; will be modified to • cooperate with the master DSP in setting the router’s registers on the fly. Occupancy counting:Creates occupancy maps for incoming events. • Not Yet Coded. • Not Yet Coded. Error counting:Counts & categorizes events with errors. Link Re-synchronization:Performs whatever corrective actions are necessary to bring links/events back into synchronization when a trigger is dropped. • Not Yet Coded.

18. Master Task Summaries: Task: Summary: Status: Histogram Control: Controls all aspects of a histogramming run. • Coded, being • extensively tested. • Needs some thought on fault tolerance. • Coded, needs • testing. Memory Mirroring:Mirror the slave DSPs’ memory (for example, their communication registers & idata); allowing slave monitoring without disrupting communications. Trap Request Monitoring:Monitor the slave DSPs trap request registers; when a request is made sets up that DSP’s router registers appropriately. Will cooperate with the slave DSP trapping task. • Not Yet Coded.

19. Histogramming: Registers Slave DSPs Master DSP Histogram CMD Histogram CMD (really status) current bin set current rep. bin set # repetitions 16 4 12 bits 16 4 12 bits Histogram STAT 0 Histogram STAT 0: events received 24 Bit 20 bin set evt rec. DSP 3 DSP 2 DSP 1 DSP 0 events received D P E R 8 8 8 8 R: Ready Bit 8 Histogram STAT 1: Processing time & event length E: Expecting P: Processing D: Done

20. Histogramming: Task States Histogram Task Histogram Control Task INIT INIT • Task variable setup. • Task variable setup. • Check HRDY in slaves. • HSTAT 0 RDY bit HI Slave Polling (all states) • Copy slave HSTAT 0,1 to local • variables. Update own HSTAT 0. READY NEWBIN • Zero out events received variable. • Zero HSTAT 0,1. Place current • bin & set in HCMD. • Wait for #repetitions in • HCMD != 0. Then: • Zero out current repetitions • Set bin, histogram set, # rep. • this bin; copy bin & set to • own HSTAT 0. • Place current bin set, # rep in • all slave HCMD. • Create & send configuration • command stream. Prepare trigger • command stream. • Clear HSTAT 0 events received. • HSTAT 0 DONE LO, EXP HI. WAITEXP • state  EXPECTING • Wait for all slave HSTAT EXP HI.

21. Histogramming: Task States (cont) Histogram Task Histogram Control Task PULSING EXPECTING • Wait for the # events received • by all slaves == current rep per • slave. (= curr. Rep./# slaves). • Check for timeout. Then: • Wait for events to arrive on • queue. Then: • HSTAT 0 PROC HI. • Send out one trigger per slave, • update current rep., HCMD. # rep • Histogram the event • HSTAT 0 PROC LO. • If Current repetitions < # rep. • per bin, repeat. • Increment events received, • HSTAT 0 events received. • HSTAT 0 PROC LO. WAITING • If events received < #rep. • this bin, repeat. Else: • Wait for all slave HSTAT 0 • DONE HI • HSTAT 0 EXP LO, DONE HI PREP • Clear HCMD #repetitions. • Increment # bins done. • state  READY • If bins done == # points: • state  DONE. Else: • Increment the data variables of • the configuration & trigger lists • state  NEWBIN