CPE 631 Lecture 12: Dynamic Scheduling – Scoreboard and Tomasulo’s Algorithm

1. CPE 631Lecture 12: Dynamic Scheduling –Scoreboard and Tomasulo’s Algorithm Electrical and Computer EngineeringUniversity of Alabama in Huntsville

2. Outline • Review • Dynamic scheduling with a scoreboard • Tomasulo’s Algorithm

3. Review: ILP Concepts and Challenges • ILP (Instruction Level Parallelism) – overlap execution of unrelated instructions • Techniques that increase amount of parallelism exploited among instructions • reduce impact of data and control hazards • increase processor ability to exploit parallelism • Pipeline CPI = Ideal pipeline CPI + Structural stalls + RAW stalls + WAR stalls + WAW stalls + Control stalls • Reducing each of the terms of the right-hand side minimize CPI and thus increase instruction throughput

4. Review: Techniques to exploit parallelism

5. Overcoming Data Hazards with Dynamic Scheduling • Why in HW at run time? • Works when can’t know real dependence at compile time • Simpler compiler • Code for one machine runs well on another • Example • Key idea: Allow instructions behind stall to proceed SUB.D cannot execute because the dependence of ADD.D on DIV.D causes the pipeline to stall; yet SUBD is not data dependent on anything! DIV.D F0,F2,F4 ADD.D F10,F0,F8 SUB.D F12,F8,F12

6. Overcoming Data Hazards with Dynamic Scheduling (cont’d) • Enables out-of-order execution => out-of-order completion • Out-of-order execution divides ID stage: • 1. Issue—decode instructions, check for structural hazards • 2. Read operands—wait until no data hazards, then read operands • Scoreboarding – technique for allowing instructions to execute out of order when there are sufficient resources and no data dependencies (CDC 6600, 1963)

7. Scoreboarding Implications • Out-of-order completion => WAR, WAW hazards? • Solutions for WAR • Queue both the operation and copies of its operands • Read registers only during Read Operands stage • For WAW, must detect hazard: stall until other completes • Need to have multiple instructions in execution phase => multiple execution units or pipelined execution units • Scoreboard keepstrack of dependencies, state or operations • Scoreboard replaces ID, EX, WB with 4 stages DIV.D F0,F2,F4 ADD.D F10,F0,F8 SUB.D F8,F8,F12 DIV.D F0,F2,F4 ADD.D F10,F0,F8 SUB.D F10,F8,F12

8. Four Stages of Scoreboard Control • ID1: Issue — decode instructions & check for structural hazards • ID2: Read operands — wait until no data hazards, then read operands • EX: Execute — operate on operands; when the result is ready, it notifies the scoreboard that it has completed execution • WB: Write results — finish execution; the scoreboard checks for WAR hazards. If none, it writes results. If WAR, then it stalls the instruction Scoreboarding stalls the the SUBD in its write result stage until ADDD reads its operands DIV.D F0,F2,F4 ADD.D F10,F0,F8 SUB.D F8,F8,F12

9. Four Stages of Scoreboard Control • 1. Issue—decode instructions & check for structural hazards (ID1) • If a functional unit for the instruction is free and no other active instruction has the same destination register (WAW), the scoreboard issues the instruction to the functional unit and updates its internal data structure. If a structural or WAW hazard exists, then the instruction issue stalls, and no further instructions will issue until these hazards are cleared. • 2. Read operands—wait until no data hazards, then read operands (ID2) • A source operand is available if no earlier issued active instruction is going to write it, or if the register containing the operand is being written by a currently active functional unit. When the source operands are available, the scoreboard tells the functional unit to proceed to read the operands from the registers and begin execution. The scoreboard resolves RAW hazards dynamically in this step, and instructions may be sent into execution out of order.

10. Four Stages of Scoreboard Control • 3. Execution—operate on operands (EX) • The functional unit begins execution upon receiving operands. When the result is ready, it notifies the scoreboard that it has completed execution. • 4. Write result—finish execution (WB) • Once the scoreboard is aware that the functional unit has completed execution, the scoreboard checks for WAR hazards. If none, it writes results. If WAR, then it stalls the instruction. • Example: • CDC 6600 scoreboard would stall SUBD until ADD.D reads operands DIV.D F0,F2,F4 ADD.D F10,F0,F8 SUB.D F8,F8,F14

11. Three Parts of the Scoreboard • 1. Instruction status—which of 4 steps the instruction is in (Capacity = window size) • 2. Functional unit status—Indicates the state of the functional unit (FU). 9 fields for each functional unit • Busy—Indicates whether the unit is busy or not • Op—Operation to perform in the unit (e.g., + or –) • Fi—Destination register • Fj, Fk—Source-register numbers • Qj, Qk—Functional units producing source registers Fj, Fk • Rj, Rk—Flags indicating when Fj, Fk are ready • 3. Register result status—Indicates which functional unit will write each register, if one exists. Blank when no pending instructions will write that register

12. MIPS with a Scoreboard Scoreboard Registers FP Mult FP Mult FP Div FP Div FP Div Add1 Add2 Add3 Control/Status Control/Status

13. Detailed Scoreboard Pipeline Control Instruction status Wait until Bookkeeping Issue Not busy (FU) and not result (D) Busy(FU) yes; Op(FU) op; Fi(FU) ’D’; Fj(FU) ’S1’; Fk(FU) ’S2’; Qj Result(’S1’); Qk Result(’S2’); Rj not Qj; Rk not Qk; Result(’D’) FU; Read operands Rj and Rk Rj No; Rk No Execution complete Functional unit done Write result f((Fj( f )≠Fi(FU) or Rj( f )=No) & (Fk( f ) ≠Fi(FU) or Rk( f )=No)) f(if Qj(f)=FU then Rj(f) Yes);f(if Qk(f)=FU then Rj(f) Yes); Result(Fi(FU)) 0; Busy(FU) No

14. Scoreboard Example

15. Scoreboard Example: Cycle 1 Issue 1st L.D!

16. Scoreboard Example: Cycle 2 Structural hazard!No further instructions will issue! Issue 2nd L.D?

17. Scoreboard Example: Cycle 3 Issue MUL.D?

18. Scoreboard Example: Cycle 4 Check for WAR hazards! If none, write result!

19. Scoreboard Example: Cycle 5 Issue 2nd L.D!

20. Scoreboard Example: Cycle 6 Issue MUL.D!

21. Scoreboard Example: Cycle 7 Issue SUB.D!

22. Scoreboard Example: Cycle 8 Issue DIV.D!

23. Scoreboard Example: Cycle 9 Read operands for MUL.D and SUB.D!Assume we can feed Mult1 and Add units in the same clock cycle. Issue ADD.D? Structural Hazard (unit is busy)!

24. Scoreboard Example: Cycle 11 Last cycle of SUB.D execution.

25. Scoreboard Example: Cycle 12 Check WAR on F8. Write F8.

26. Scoreboard Example: Cycle 13 Issue ADD.D!

27. Scoreboard Example: Cycle 14 Read operands for ADD.D!

28. Scoreboard Example: Cycle 15 Read operands for ADD.D!

29. Scoreboard Example: Cycle 16

30. Scoreboard Example: Cycle 17 Why cannot write F6?

31. Scoreboard Example: Cycle 19

32. Scoreboard Example: Cycle 20

33. Scoreboard Example: Cycle 21

34. Scoreboard Example: Cycle 22 Write F6?

35. Scoreboard Example: Cycle 61 Write F6?

36. Scoreboard Example: Cycle 62 Write F6?

37. Scoreboard Results • For the CDC 6600 • 70% improvement for Fortran • 150% improvement for hand coded assembly language • cost was similar to one of the functional units • surprisingly low • bulk of cost was in the extra busses • Still this was in ancient time • no caches & no main semiconductor memory • no software pipelining • compilers? • So, why is it coming back • performance via ILP

38. Scoreboard Limitations • Amount of parallelism among instructions • can we find independent instructions to execute • Number of scoreboard entries • how far ahead the pipeline can look for independent instructions (we assume a window does not extend beyond a branch) • Number and types of functional units • avoid structural hazards • Presence of antidependences and output dependences • WAR and WAW stalls become more important

39. A Dynamic Algorithm: Tomasulo’s Algorithm • Used in IBM 360/91 FPU (before caches) • Goal: high FP performance without special compilers • Conditions: • Small number of floating point registers (4 in 360) prevented interesting compiler scheduling of operations • Long memory accesses and long FP delays • This led Tomasulo to try to figure out how to get more effective registers — renaming in hardware! • Why Study 1966 Computer? • The descendants of this have flourished! • Alpha 21264, HP 8000, MIPS 10000, Pentium III, PowerPC 604, …

40. Tomasulo Algorithm • Control & buffers distributed with Function Units (FU) • FU buffers called “reservation stations” => buffer the operands of instructions waiting to issue; • Registers in instructions replaced by values or pointers to reservation stations (RS) => register renaming • avoids WAR, WAW hazards • More reservation stations than registers, so can do optimizations compilers can’t • Results to FU from RS, not through registers, over Common Data Bus that broadcasts results to all FUs • Load and Stores treated as FUs with RSs as well • Integer instructions can go past branches, allowing FP ops beyond basic block in FP queue

41. Tomasulo-based FPU for MIPS From Instruction Unit FP Registers FP Op Queue From Mem Load Buffers Load1 Load2 Load3 Load4 Load5 Load6 Store Buffers Store1 Store2 Store3 Add1 Add2 Add3 Mult1 Mult2 Reservation Stations To Mem FP adders FP multipliers Common Data Bus (CDB)

42. Reservation Station Components • Op: Operation to perform in the unit (e.g., + or –) • Vj, Vk: Value of Source operands • Store buffers has V field, result to be stored • Qj, Qk: Reservation stations producing source registers (value to be written) • Note: Qj/Qk=0 => source operand is already available in Vj /Vk • Store buffers only have Qi for RS producing result • Busy: Indicates reservation station or FU is busy Register result status—Indicates which functional unit will write each register, if one exists. Blank when no pending instructions that will write that register.

43. Three Stages of Tomasulo Algorithm • 1. Issue—get instruction from FP Op Queue • If reservation station free (no structural hazard), control issues instr & sends operands (renames registers) • 2. Execute—operate on operands (EX) • When both operands ready then execute;if not ready, watch Common Data Bus for result • 3. Write result—finish execution (WB) • Write it on Common Data Bus to all awaiting units; mark reservation station available • Normal data bus: data + destination (“go to” bus) • Common data bus: data + source (“come from” bus) • 64 bits of data + 4 bits of Functional Unit source address • Write if matches expected Functional Unit (produces result) • Does the broadcast • Example speed: 2 clocks for Fl .pt. +,-; 10 for * ; 40 clks for /

44. Tomasulo Example Instruction stream 3 Load/Buffers FU count down 3 FP Adder R.S. 2 FP Mult R.S. Clock cycle counter

45. Tomasulo Example Cycle 1

46. Tomasulo Example Cycle 2 Note: Can have multiple loads outstanding

47. Tomasulo Example Cycle 3 • Note: registers names are removed (“renamed”) in Reservation Stations; MULT issued • Load1 completing; what is waiting for Load1?

48. Tomasulo Example Cycle 4 • Load2 completing; what is waiting for Load2?

49. Tomasulo Example Cycle 5 • Timer starts down for Add1, Mult1

50. Tomasulo Example Cycle 6 • Issue ADDD here despite name dependency on F6?