Current and Future Trends in Processor Architecture

1. Current and Future Trends in Processor ArchitectureTheo UngererDept. of Computer Design and Fault ToleranceUniversity of KarlsruheD-76128 Karlsruhe, Germanyungerer@Informatik.Uni-Karlsruhe.dehttp://goethe.ira.uka.de/people/ungerer/

2. Tutorial Background Material • J. Silc, B. Robic, Th. Ungerer: Processor Architecture From Dataflow to Superscalar and Beyond. Springer-Verlag, Berlin, Heidelberg, New York 1999, 406 pages. • Book homepage: http://goethe.ira.uka.de/people/ungerer/proc-arch/ • Slide collection of tutorial slides: http://goethe.ira.uka.de/people/ungerer/ • Slide collection of book contents (in 15 lectures): http://goethe.ira.uka.de/people/ungerer/prozarch/prslides99-00.html

3. Outline of the Tutorial • State-of-the-art multiple-issue processors • Superscalar – Overview • Superscalar in more detail: Instruction Fetch and Branch Prediction, Decode, Rename, Issue, Dispatch, Execution Units, Completion, Retirement • VLIW/EPIC • Solutions for future high-performance processors • Technology prognosis • Speed-up of a single-threaded application • Advanced superscalar, Trace Cache, Superspeculative, Multiscalar processors • Speed-up of multi-threaded applications • Chip multiprocessors (CMPs) and Simultaneous multithreading

4. Multiple-issue Processors • Today's microprocessors utilize instruction-level parallelism by a multi-stage instruction pipeline and by the superscalar or the VLIW/EPIC technique. • Most of today's general-purpose microprocessors are four- or six-issue superscalars. • VLIW (very long instruction word) is the choice for most signal processors. • VLIW is enhanced to EPIC (explicitly parallel instruction computing) by HP/Intel for its IA-64 ISA.

5. Instruction Pipelining

6. EX Retire and Write Back ID and Rename Instruction Window EX IF Issue EX EX Superscalar Pipeline • Instructions in the instruction window are free from control dependencies due to branch prediction, and free from name dependences due to register renaming. • So, only (true) data dependences and structural conflicts remain to be solved.

7. Superscalar vs. VLIW • Superscalar and VLIW: More than a single instruction can be issued to the execution units per cycle. • Superscalar machines are able to dynamically issue multiple instructions each clock cycle from a conventional linear instruction stream. • VLIW processors use a long instruction word that contains a usually fixed number of instructions that are fetched, decoded, issued, and executed synchronously. • Superscalar: dynamic issue, VLIW: static issue

8. Sections of a Superscalar Pipeline • The ability to issue and execute instructions out-of-order partitions a superscalar pipeline in three distinct sections: • in-order section with the instruction fetch, decode and rename stages - the issue is also part of the in-order section in case of an in-order issue, • out-of-order section starting with the issue in case of an out-of-order issue processor, the execution stage, and usually the completion stage, and again an • in-order section that comprises the retirement and write-back stages.

9. I-cache MMU BHT BTAC 32 (64) Data Bus Branch Unit Instruction Fetch Unit Instruction Decode and Register Rename Unit 32 (64) Bus Inter- face Unit Instruction Buffer Instruction Issue Unit Address Bus Reorder Buffer Load/ Store Unit Floating- Point Unit(s) Integer Unit(s) Retire Unit Control Bus Floating- Point Registers General Purpose Registers Rename Registers MMU D-cache Components of a Superscalar Processor

10. Prediction Branch address Target address bits ... ... ... Branch-Target Buffer or Branch-Target Address Cache • The Branch Target Buffer (BTB) or Branch-Target Address Cache (BTAC) stores branch and jump addresses, their target addresses, and optionally prediction information. • The BTB is accessed during the IF stage.

11. Branch Prediction • Branch prediction foretells the outcome of conditional branch instructions. • Excellent branch handling techniques are essential for today's and for future microprocessors. • Requirements of high performance branch handling: • an early determination of the branch outcome (the so-called branch resolution), • buffering of the branch target address in a BTAC, • an excellent branch predictor (i.e. branch prediction technique) and speculative execution mechanism, • often another branch is predicted while a previous branch is still unresolved, so the processor must be able to pursue two or more speculation levels, • and an efficient rerolling mechanism when a branch is mispredicted (minimizing the branch misprediction penalty).

12. Misprediction Penalty • The performance of branch prediction depends on the prediction accuracy and the cost of misprediction. • Misprediction penalty depends on many organizational features: • the pipeline length (favoring shorter pipelines over longer pipelines), • the overall organization of the pipeline, • the fact if misspeculated instructions can be removed from internal buffers, or have to be executed and can only be removed in the retire stage, • the number of speculative instructions in the instruction window or the reorder buffer. Typically only a limited number of instructions can be removed each cycle. • Mispredicted is expensive: • 4 to 9 cycles in the Alpha 21264, • 11 or more cycles in the Pentium II.

13. Static Branch Prediction • Static Branch Prediction predicts always the same direction for the same branch during the whole program execution. • It comprises hardware-fixed prediction and compiler-directed prediction. • Simple hardware-fixed direction mechanisms can be: • Predict always not taken • Predict always taken • Backward branch predict taken, forward branch predict not taken • Sometimes a bit in the branch opcode allows the compiler to decide the prediction direction.

14. Dynamic Branch Prediction • Dynamic Branch Prediction: the hardware influences the prediction while execution proceeds. • Prediction is decided on the computation history of the program. • During the start-up phase of the program execution, where a static branch prediction might be effective, the history information is gathered and dynamic branch prediction gets effective. • In general, dynamic branch prediction gives better results than static branch prediction, but at the cost of increased hardware complexity.

15. T NT Predict Not Predict Taken Taken T NT One-bit Predictor

16. One-bit vs. Two-bit Predictors • A one-bit predictor correctly predicts a branch at the end of a loop iteration, as long as the loop does not exit. • In nested loops, a one-bit prediction scheme will cause two mispredictions for the inner loop: • One at the end of the loop, when the iteration exits the loop instead of looping again, and • one when executing the first loop iteration, when it predicts exit instead of looping. • Such a double misprediction in nested loops is avoided by a two-bit predictor scheme. • Two-bit Prediction: A prediction must miss twice before it is changed when a two-bit prediction scheme is applied.

17. T NT NT NT (11) (10) (01) (00) Predict Strongly Predict Weakly Predict Weakly Predict Strongly Taken Taken Not Taken Not Taken T T T NT Two-bit Predictors(Saturation Counter Scheme)

18. NT T NT NT (11) (10) (01) (00) Predict Strongly Predict Weakly Predict Weakly Predict Strongly Taken Taken Not Taken Not Taken T T NT T Two-bit Predictors(Hysteresis Scheme)

19. Two-bit Predictors • Two-bit predictors can be implemented in the Branch Target Buffer (BTB) assigning two state bits to each entry in the BTB. • Another solution is to use a BTB for target addresses and a separate Branch History Table (BHT) as prediction buffer. • A mispredict in the BHT occurs due to two reasons: • either a wrong guess for that branch, • or the branch history of a wrong branch is used because the table is indexed. • In an indexed table lookup part of the instruction address is used as index to identify a table entry.

20. Two-bit Predictors and Correlation-based Prediction • Two-bit predictors work well for programs which contain many frequently executed loop-control branches (floating-point intensive programs). • Shortcomings arise from dependent (correlated) branches, which are frequent in integer-dominated programs. Example:if (d==0) /* branch b1*/ d=1; if (d==1) /*branch b2 */ ...

21. Predictor Behavior in Example • A one-bit predictor initialized to “ predict taken” for branches b1 and b2  every branch is mispredicted. • A two-bit predictor of of saturation counter scheme starting from the state “predict weakly taken”  every branch is mispredicted. • The two-bit predictor of hysteresis scheme mispredicts every second branch execution of b1 and b2. • A (1,1) correlating predictor takes advantage of the correlation of the two branches; it mispredicts only in the first iteration when d = 2. • See: http://goethe.ira.uka.de/people/ungerer/prozarch/prslides99-00.htlm9th lecture for animated demonstrations

22. Correlation-based Predictor • The two-bit predictor scheme uses only the recent behavior of a single branch to predict the future of that branch. • Correlations between different branch instructions are not taken into account. • Correlation-based predictors or correlating predictors additionally use the behavior of other branches to make a prediction. • While two-bit predictors use self-history only, the correlating predictor uses neighbor history additionally. • Notation: (m,n)-correlation-based predictor or (m,n)-predictor uses the behavior of the last m branches to choose from 2m branch predictors, each of which is a n-bit predictor for a single branch. • Branch history register (BHR): The global history of the most recent m branches can be recorded in a m-bit shift register where each bit records whether the branch was taken or not taken.

23. Pattern History Tables PHTs (2-bit predictors) Branch address ... ... ... ... 10 1 1 ... ... ... ... select Branch History Register BHR 1 0 (2-bit shift register) Correlation-based Prediction(2,2)-predictor

24. Two-level Adaptive Predictors • Developed by Yeh and Patt at the same time (1992) as the correlation-based prediction scheme. • The basic two-level predictor uses a single global branch history register (BHR) of k bits to index in a pattern history table (PHT) of 2-bit counters. • Global history schemes correspond to correlation-based predictor schemes. • Example for the notation: GAg: • a single global BHR (denoted G) and • a single global PHT (denoted g), • A stands for adaptive. • All PHT implementations of Yeh and Patt use 2-bit predictors. • GAg-predictor with a 4-bit BHR length is denoted as GAg(4).

25. shift direction Branch Pattern Branch History Table History 1 1 0 0 (PHT) ... Register (BHR) predict: 1100 1 1 taken Index ... Implementation of a GAg(4)-predictor • In the GAg predictor schemes the PHT lookup depends entirely on the bit pattern in the BHR and is completely independent of the branch address.

26. Variations of Two-level Adaptive Predictors • the full branch address to distinguish multiple PHTs (called per-address PHTs), • a subset of branches (e.g. n bits of the branch address) to distinguish multiple PHTs (called per-set PHTs), • the full branch address to distinguish multiple BHRs (called per-address BHRs), • a subset of branches to distinguish multiple BHRs (called per-set BHRs), • or a combination scheme. Mispredictions can be restrained by additionally using:

27. Two-level Adaptive Predictors single global PHT per-set PHTs per-address PHTs single global BHR GAg GAs GAp per-address BHT PAg PAs PAp per-set BHT SAg SAs SAp

28. gselect and gshare Predictors • gselect predictor: concatenates some lower order bit of the branch address and the global history • gshare predictor: uses the bitwise exclusive OR of part of the branch address and the global history as hash function. • McFarling: gshare slightly better than gselect Branch Address BHR gselect4/4 gshare8/8 00000000 00000001 00000001 00000001 00000000 00000000 00000000 00000000 11111111 00000000 11110000 11111111 11111111 10000000 11110000 01111111

29. Hybrid Predictors • The second strategy of McFarling is to combine multiple separate branch predictors, each tuned to a different class of branches. • Two or more predictors and a predictor selection mechanism are necessary in a combining or hybrid predictor. • McFarling: combination of two-bit predictor and gshare two-level adaptive, • Young and Smith: a compiler-based static branch prediction with a two-level adaptive type, • and many more combinations! • Hybrid predictors often better than single-type predictors.

30. Simulations of Grunwald 1998 SAg, gshare and MCFarling‘s combining predictor for some SPECmarks

31. Results • Simulation of Keeton et al. 1998 using an OLTP (online transaction workload) on a PentiumPro multiprocessor reported a misprediction rate of 14% with an branch instruction frequency of about 21%. • Two different conclusions may be drawn from these simulation results: • Branch predictors should be further improved • and/or branch prediction is only effective if the branch is predictable. • If a branch outcome is dependent on irregular data inputs, the branch often shows an irregular behavior.  Question: Confidence of a branch prediction?

32. Confidence Estimation • Confidence estimation is a technique for assessing the quality of a particular prediction. • Applied to branch prediction, a confidence estimator attempts to assess the prediction made by a branch predictor. • A low confidence branch is a branch which frequently changes its branch direction in an irregular way making its outcome hard to predict or even unpredictable. • Four classes possible: • correctly predicted with high confidence C(HC), • correctly predicted with low confidence C(LC), • incorrectly predicted with high confidence I(HC), and • incorrectly predicted with low confidence I(LC).

33. Predicated Instructions • Method to “remove” branches • Predicated or conditional instructions and one or more predicate registers use a predicate register as additional input operand. • The Boolean result of a condition testing is recorded in a (one-bit) predicate register. • Predicated instructions are fetched, decoded and placed in the instruction window like non predicated instructions. • It is dependent on the processor architecture, how far a predicated instruction proceeds speculatively in the pipeline before its predication is resolved: • A predicated instruction executes only if its predicate is true, otherwise the instruction is discarded. • Alternatively the predicated instruction may be executed, but commits only if the predicate is true, otherwise the result is discarded.

34. Predication Example if (x = = 0) { /*branch b1 */ a = b + c; d = e - f; } g = h * i; /* instruction independent of branch b1 */ (Pred = (x = = 0) ) /* branch b1: Pred is set to true in x equals 0 */ if Predthena = b + c; /* The operations are only performed */ ifPredthene = e - f; /* if Pred is set to true */ g = h * i;

35. Predication • Able to eliminate a branch and therefore the associated branch prediction  increasing the distance between mispredictions. • The the run length of a code block is increased  better compiler scheduling. • Predication affects the instruction set, adds a port to the register file, and complicates instruction execution. • Predicated instructions that are discarded still consume processor resources; especially the fetch bandwidth. • Predication is most effective when control dependences can be completely eliminated, such as in an if-then with a small then body. • The use of predicated instructions is limited when the control flow involves more than a simple alternative sequence.

36. Eager (Multipath) Execution • Execution proceeds down both paths of a branch, and no prediction is made. • When a branch resolves, all operations on the non-taken path are discarded. • With limited resources, the eager execution strategy must be employed carefully. • Mechanism is required that decides when to employ prediction and when eager execution: e.g. a confidence estimator • Rarely implemented (IBM mainframes) but some research projects: • Dansoft processor, Polypath architecture, selective dual path execution, simultaneous speculation scheduling, disjoint eager execution

37. Branch handling techniques and implementations Technique Implementation examples No branch prediction Intel 8086 Static prediction always not taken Intel i486 always taken Sun SuperSPARC backward taken, forward not taken HP PA-7x00 semistatic with profiling early PowerPCs Dynamic prediction: 1-bit DEC Alpha 21064, AMD K5 2-bit PowerPC 604, MIPS R10000, Cyrix 6x86 and M2, NexGen 586 two-level adaptive Intel PentiumPro, Pentium II, AMD K6 Hybrid prediction DEC Alpha 21264 Predication Intel/HP Itanium and most signal processors as e.g. ARM processors, TI TMS320C6201 and many other Eager execution (limited) IBM mainframes: IBM 360/91, IBM 3090

38. High-Bandwidth Branch Prediction • Future microprocessor will require more than one prediction per cycle starting speculation over multiple branches in a single cycle • When multiple branches are predicted per cycle, then instructions must be fetched from multiple target addresses per cycle, complicating I-cache access. • Solution: Trace cache in combination with next trace prediction.

39. EX Retire and Write Back ID and Rename Instruction Window EX IF Issue EX EX Back to the Superscalar Pipeline • In-order delivery of instructions to the out-of-order execution kernel!

40. Decode Stage • Delivery task: Keep instruction window full==> the deeper instruction look-ahead allows to find more instructions to issue to the execution units. • Fetch and decode instructions at a higher bandwidth than execute them. • The processor fetches and decodes today about 1.4 to twice as many instructions than it commits (because of mispredicted branch paths). • Typically the decode bandwidth is the same as the instruction fetch bandwidth. • Multiple instruction fetch and decode is supported by a fixed instruction length.

41. Decoding variable-length instructions • Variable instruction length:often the case for legacy CISC instruction sets as the Intel IA32 ISA.  a multistage decode is necessary. • The first stage determines the instruction limits within the instruction stream. • The second stage decodes the instructions generating one or several micro-ops from each instruction. • Complex CISC instructions are split into micro-ops which resemble ordinary RISC instructions.

42. Two principal techniques to implement renaming: • Separate sets of architectural registers and rename (physical) registers are provided. • The physical registers contain values (of completed but not yet retired instructions), • the architectural registers store the committed values. • After commitment of an instruction, copying its result from the rename register to the architectural register is required. • Only a single set of registers is provided and architectural registers are dynamically mapped to physical registers. • The physical registers contain committed values and temporary results. • After commitment of an instruction, the physical register is made permanent and no copying is necessary. • Alternative to the dynamic renaming is static renaming in combination with a large register file as defined for the Intel Itanium.

43. Issue and Dispatch • The notion of the instruction window comprises all the waiting stations between decode (rename) and execute stages. • The instruction window isolates the decode/rename from the execution stages of the pipeline. • Instruction issue is the process of initiating instruction execution in the processor's functional units. • issue to a FU or a reservation station • dispatch, if a second issue stage exists to denote when an instruction is started to execute in the functional unit. • The instruction-issue policy is the protocol used to issue instructions. • The processor's lookahead capability is the ability to examine instructions beyond the current point of execution in hope of finding independent instructions to execute.

44. Issue • The issue logic examines the waiting instructions in the instruction window and simultaneously assigns (issues) a number of instructions to the FUs up to a maximum issue bandwidth. • The program order of the issued instructions is stored in the reorder buffer. • Instruction issue from the instruction window can be: • in-order (only in program order) or out-of-order • it can be subject to simultaneous data dependences and resource constraints, • or it can be divided in two (or more) stages • checking structural conflict in the first and data dependences in the next stage (or vice versa). • In the case of structural conflicts first, the instructions are issued to reservation stations (buffers) in front of the FUs where the issued instructions await missing operands.

45. Reservation Station(s) • Two definitions in literature: • A reservation station is a buffer for a single instruction with its operands (original Tomasulo paper, Flynn's book, Hennessy/Patterson book). • A reservation station is a buffer (in front of one or more FUs) with one or more entries and each entry can buffer an instruction with its operands(PowerPC literature). • Depending on the specific processor, reservation stations can be central to a number of FUs or each FU has one or more own reservation stations. • Instructions await their operands in the reservation stations, as in the Tomasulo algorithm.

46. Dispatch • An instruction is then said to be dispatched from a reservation station to the FU when all operands are available, and execution starts. • If all its operands are available during issue and the FU is not busy, an instruction is immediately dispatched, starting execution in the next cycle after the issue. • So, the dispatch is usually not a pipeline stage. • An issued instruction may stay in the reservation station for zero to several cycles. • Dispatch and execution is performed out of program order. • Other authors interchange the meaning of issue and dispatch or use different semantic.

47. Issue and Dispatch Decode Functional and Units Rename The following issue schemes are commonly used: • Single-level, central issue: single-level issue out of a central window as in Pentium II processor

48. Issue and Dispatch Functional Units Decode and Rename Functional Units Single-level, two-window issue • Single-level, two-window issue: single-level issue with a instruction window decoupling using two separate windows • most common: separate floating point and integer windows as in HP 8000 processor

49. Dispatch Issue Functional Unit Functional Unit Decode and Rename Functional Unit Functional Unit Reservation Stations Two-level issue with multiple windows • Two-level issue with multiple windows with a centralized window in the first stage and separate windows in the second stage (PowerPC 604 and 620 processors).

50. Execution Stages • Various types of FUs classified as: • single-cycle (latency of one) or • multiple-cycle (latency more than one) units. • Single-cycle units produce a result one cycle after an instruction started execution. Usually they are also able to accept a new instruction each cycle (throughput of one). • Multi-cycle units perform more complex operations that cannot be implemented within a single cycle. • Multi-cycle units • can be pipelined to accept a new operation each cycle or each other cycle • or they are non-pipelined. • Another class of units exists that perform the operations with variable cycle times.