1 / 18

Overview of the APIC Pacer

This overview provides a general understanding of APIC Pacing, including its purpose, controls, data transfers, and global and per-VC pacing parameters.

robertswilson
Download Presentation

Overview of the APIC Pacer

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overviewof theAPIC Pacer

  2. APIC Pacing: General Stuff • Pacing is for Transmit Channels only • Cells are NOT Paced out onto the wire • Not Exactly • Pacing is done on the PCI bus • Pacing is not a Guarantee, it is just a Restriction • Pacing Calculations include the ATM headers • But not the APIC header • More APIC Info: • http://www.arl.wustl.edu/arl/projects/msr/pdf/ApicArch.pdf

  3. APIC Pacing: General Stuff • Two pacer controls: • Global Pacing • APIC Pacing Parameter register (Global, 0x208) • Per VC Pacing • TX Channel Pacing Parameter Register (TX, 0x500XX68) • XX is the Channel ID • Three types of Channels: • Low Delay (Highest Priority) • Paced • Best Effort (Lowest Priority) • All channels are paced by the Global Pacing • Paced Channels also use Per VC Pacing

  4. APIC Data Transfers • APIC pulls data from memory across the PCI bus in Batches of cells. • The number of cells in a Batch is controlled by a register • The Pacer identifies when it is time to transmit data and which connection should transmit • Pacer “wakes up” every 14 PCI Bus clock ticks • checks to see if it is time to transmit • Controlled by the Global APIC Pacing Parameter (APP) • If it is time to transmit, it takes the first connection off the previously sorted list of keys and transmits its data. • A lot of gory details about keys and heap storage of connections is not going to be included here. Read Rex’s documentation and/or read the VHDL if you want that level of detail

  5. Global Pacing Parameter • Pacing parameters are 24 bits • 16 bits of Integer • 8 bits of fractional part • Global Apic Pacing Parameter (APP) (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = -------------------------------------------------------- (14 * ClockEstimate * LinkRateMbps) [Items in formula explained on next slide]

  6. Explanation of Expression (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = -------------------------------------------------------- (14 * ClockEstimate * LinkRateMbps) • 256 : shifts left by 8 bits to set “decimal point” • BatchSz: How many cells per transfer • 53*8: Translate cells/second into bits/second • 8192, InternalClockMhz (85MHz), ClockEstimate • APIC counts how many of its internal 85MHz clock ticks take place during the time it takes for 8192 PCI bus clock ticks. This value is the ClockEstimate. • PCI Bus Clock Rate in MHz = (8192 * 85)/ClockEstimate • 14: # of PCI Bus Ticks in a Pacer Period • LinkRateMbps: Our target rate [Example on next 2 slides]

  7. Example: Units in the APP Formula (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = -------------------------------------------------------- (14 * ClockEstimate * LinkRateMbps) (256 * Cells * Bytes/Cell * Bits/Byte * 8192 * M/sec) APP = -------------------------------------------------------- (14 * 1 * MBits/sec)

  8. Example: APP for 1Gb/s Link Rate (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = -------------------------------------------------------- (14 * ClockEstimate * LinkRateMbps) • BatchSz=8 • 53*8: Translate cells/second into bits/second • InternalClockMhz = 85MHz • ClockEstimate = 20954 (typical value) • LinkRateMbps: 1000 (1000 Mb/s == 1Gb/s) (256 * 8 * 53 * 8 * 8192 * 85) APP = ---------------------------------------- = 2061.15 (14 * 20954 * 1000) APP = 2061 = 0x80D

  9. Example: APP for 1Gb/s Link Rate APP = 2061 = 0x80D This means that every 14*8 = 112 PCI Bus clock ticks the APIC will be able to pull 8 Cells worth of data across the PCI Bus. (8 Cells)/(112 * 30ns) = (3392 bits)/(3360ns) ~= 1Gb/s

  10. 33 MHz PCI Bus Clock Count to 14 Count to APP Count to TX Channel Pacing Parameter This Tx Channel is Ready to Transmit BATCH Cells Per VC Pacing • Per VC Pacing Parameter • What portion of the full link rate can be used • e.g. an integer value of 2 means that this channel can use half the link rate • If you change the Pacing Paramter for a VC: • It takes affect the next time the OLD pacing expires!!! • Conceptually like this:

  11. Per VC Pacing vcPacingParameter ~ 10 One APIC Pacing Period current pacedTime Expired connections X X X X X X X time oldExpirationTim + vcPacingParameter  newExpirationTime

  12. pacedTime • pacedTime is incremented every global pacing cycle in which a non-LowDelay connection wins contention • Example with two connections: • (L) Low Delay at 1/24th of the global rate • (P) Paced at 1/6th of the global rate (.1666667) L L L L P P P P P P P P P P P P P P P 0 6 12 18 24 30 36 42 48 54 60 66 72 78 84

  13. L L L L P P P P P P P P P P P P P P P 0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 pacedTime (continued) • We might expect the Paced channel to miss its exact turn and fire on the next global pacing interval but keeps it next expiration on the (0,6,12,18,…) boundaries. • But…

  14. L L L L 0 0 6 5 12 11 18 17 22 24 28 30 36 34 42 40 48 45 51 54 60 57 63 66 68 72 78 74 80 84 pacedTime (continued) • Actual rate for Paced connection: • (GlobalRate) * (3*(1/6) + 1*(1/7))/4 • (GlobalRate) * (.1607) • For a Global Rate of 24Mb/s (DQ test example) • 24 * .1607 = 3.8568 P P P P P P P P P P P P P P P t+ pacedTime t+ “Real” time

  15. Per VC Pacing • If you change the Pacing Paramter for a VC: • It takes affect the next time the OLD pacing expires!!! • A Resume of the VC will cause an immediate change to the new Pacing parameter • This artifact causes some strange behavior for DQ where there are large changes in the Per-VC pacing parameters. • Going from a low rate to a high rate can be delayed by several DQ periods. • This can cause the results to appear flawed. • See the next slide for an illustration.

  16. Per VC Pacing Right after, Want to Change Pacing so it will Transmit every other Pacing Interval These will Transmit X X X X X X This will not transmit time Just Transmitted Next Transmission by oldPacing Paramter

  17. D D D D D D D Example of a Pacing Oddity • Suppose we have a channel on which we are sending single cell packets at a rate of 2 cells every pacing period for that channel and the BATCH size is 1 cell so that the channel should only send 1 cell during each pacing period. • You would expect the connection to build up a backlog, but it doesn’t……

  18. D T D T D T D T D T D T D T R R R R R R R Example of a Pacing Oddity (con’t) • Turns out the Driver does a RESUME each time it puts data in an empty transmit queue to restart it. • A RESUME causes the ExpireTime to be set to the current PacedTime. • This causes the channel to be expired at the very next Pacer Period. • Thus the channel transmits at twice its expected rate

More Related