EE (CE) 6304 Computer Architecture Lecture #5 (9/6/17)

1. EE (CE) 6304 Computer ArchitectureLecture #5(9/6/17) Yiorgos Makris Professor Department of Electrical Engineering University of Texas at Dallas Course Web-site: http://www.utdallas.edu/~gxm112130/EE6304FA16

2. Reg Reg Reg Reg Reg Reg Reg Reg Reg Reg ALU ALU ALU ALU ALU Ifetch Ifetch Ifetch Ifetch Ifetch DMem DMem DMem DMem DMem I n s t r. O r d e r add r1,r2,r3 lw r4, 0(r1) sw r4,12(r1) or r8,r6,r9 xor r10,r9,r11 Forwarding to Avoid LW-SW Data Hazard Time (clock cycles)

3. Reg Reg Reg Reg Reg Reg Reg Reg ALU Ifetch Ifetch Ifetch Ifetch DMem DMem DMem DMem ALU ALU ALU lwr1, 0(r2) sub r4,r1,r6 and r6,r1,r7 or r8,r1,r9 Data Hazard Even with Forwarding Time (clock cycles) I n s t r. O r d e r

4. Reg Reg Reg Ifetch Ifetch Ifetch Ifetch DMem ALU Bubble ALU ALU Reg Reg DMem DMem Bubble Reg Reg Data Hazard Even with Forwarding Time (clock cycles) lwr1, 0(r2) I n s t r. O r d e r sub r4,r1,r6 and r6,r1,r7 Bubble ALU DMem or r8,r1,r9

5. Software Scheduling to Avoid Load Hazards Try producing fast code for a = b + c; d = e – f; assuming a, b, c, d ,e, and f in memory. Slow code: LW Rb,b LW Rc,c ADD Ra,Rb,Rc SW a,Ra LW Re,e LW Rf,f SUB Rd,Re,Rf SW d,Rd Fast code: LW Rb,b LW Rc,c LW Re,e ADD Ra,Rb,Rc LW Rf,f SW a,Ra SUB Rd,Re,Rf SW d,Rd

6. Reg Reg Reg Reg Reg Reg Reg Reg Reg Reg ALU ALU ALU ALU ALU Ifetch Ifetch Ifetch Ifetch Ifetch DMem DMem DMem DMem DMem 10: beq r1,r3,36 14: and r2,r3,r5 18: or r6,r1,r7 22: add r8,r1,r9 36: xor r10,r1,r11 Control Hazard on Branches Three Stage Stall What do you do with the 3 instructions in between? How do you do it? Where is the “commit”?

7. Branch Stall Impact • If CPI = 1, 30% branch, Stall 3 cycles => new CPI = 1.9! • Two part solution: • Determine branch taken or not sooner, AND • Compute taken branch address earlier • MIPS branch tests if register = 0 or  0 • MIPS Solution: • Move Zero test to ID/RF stage • Adder to calculate new PC in ID/RF stage • 1 clock cycle penalty for branch versus 3

8. MEM/WB ID/EX EX/MEM IF/ID Adder 4 Address ALU Pipelined MIPS Datapath Instruction Fetch Execute Addr. Calc Memory Access Instr. Decode Reg. Fetch Write Back Next SEQ PC Next PC MUX Adder Zero? RS1 Reg File Memory RS2 Data Memory MUX MUX Sign Extend WB Data Imm RD RD RD • Interplay of instruction set design and cycle time.

9. Four Branch Hazard Alternatives #1: Stall until branch direction is clear #2: Predict Branch Not Taken • Execute successor instructions in sequence • “Squash” instructions in pipeline if branch actually taken • Advantage of late pipeline state update • 47% MIPS branches not taken on average • PC+4 already calculated, so use it to get next instruction #3: Predict Branch Taken • 53% MIPS branches taken on average • But haven’t calculated branch target address in MIPS • MIPS still incurs 1 cycle branch penalty • Other machines: branch target known before outcome

10. Four Branch Hazard Alternatives #4: Delayed Branch • Define branch to take place AFTER a following instruction branch instruction sequential successor1 sequential successor2 ........ sequential successorn branch target if taken • 1 slot delay allows proper decision and branch target address in 5 stage pipeline • MIPS uses this Branch delay of length n

11. becomes becomes becomes if $2=0 then add $1,$2,$3 if $1=0 then add $1,$2,$3 sub $4,$5,$6 add $1,$2,$3 if $1=0 then sub $4,$5,$6 Scheduling Branch Delay Slots A. From before branch B. From branch target C. From fall through add $1,$2,$3 if $1=0 then add $1,$2,$3 if $2=0 then sub $4,$5,$6 • A is the best choice, fills delay slot & reduces instruction count (IC) • In B, the sub instruction may need to be copied, increasing IC • In B and C, must be okay to execute sub when branch fails delay slot delay slot add $1,$2,$3 if $1=0 then sub $4,$5,$6 delay slot

12. Delayed Branch • Compiler effectiveness for single branch delay slot: • Fills about 60% of branch delay slots • About 80% of instructions executed in branch delay slots useful in computation • About 50% (60% x 80%) of slots usefully filled • Delayed Branch downside: As processor go to deeper pipelines and multiple issue, the branch delay grows and need more than one delay slot • Delayed branching has lost popularity compared to more expensive but more flexible dynamic approaches • Growth in available transistors has made dynamic approaches relatively cheaper

13. Memory Hierarchy Review

14. Gap grew 50% per year Since 1980, CPU has outpaced DRAM ... Performance (1/latency) CPU 60% per yr 2X in 1.5 yrs 1000 CPU • How do architects address this gap? • Put small, fast “cache” memories between CPU and DRAM. • Create a “memory hierarchy” 100 DRAM 9% per yr 2X in 10 yrs 10 DRAM 2000 1990 1980 Year

15. Memory Hierarchy • Take advantage of the principle of locality to: • Present as much memory as in the cheapest technology • Provide access at speed offered by the fastest technology Processor Tertiary Storage (Tape/Cloud Storage) Control Secondary Storage (Disk/FLASH/PCM) Main Memory (DRAM/FLASH/ PCM) Second Level Cache (SRAM) On-Chip Cache Datapath Registers 10,000,000s (10s ms) Speed (ns): 1s 10s-100s 100s 10,000,000,000s (10s sec) Size (bytes): 100s Ks-Ms Ms Gs Ts

16. The Principle of Locality • The Principle of Locality: • Program access a relatively small portion of the address space at any instant of time. • Two Different Types of Locality: • Temporal Locality (Locality in Time): If an item is referenced, it will tend to be referenced again soon (e.g., loops, reuse) • Spatial Locality (Locality in Space): If an item is referenced, items whose addresses are close by tend to be referenced soon (e.g., straightline code, array access) • Last 15 years, HW relied on locality for speed

17. Bad locality behavior Temporal Locality Spatial Locality Programs with locality cache well ... Memory Address (one dot per access) Time Donald J. Hatfield, Jeanette Gerald: Program Restructuring for Virtual Memory. IBM Systems Journal 10(3): 168-192 (1971)

18. Managed by compiler Managed by OS, hardware, application Managed by hardware iMac G5 1.6 GHz Memory Hierarchy: Apple iMac G5 Goal: Illusion of large, fast, cheap memory Let programs address a memory space that scales to the disk size, at a speed that is usually as fast as register access

19. 512K L2 L1 (32K Data) iMac’s PowerPC 970: All caches on-chip L1 (64K Instruction) Registers (1K)

20. Lower Level Memory Upper Level Memory To Processor Blk X From Processor Blk Y Memory Hierarchy: Terminology • Hit: data appears in some block in the upper level (example: Block X) • Hit Rate: the fraction of memory access found in the upper level • Hit Time: Time to access the upper level which consists of RAM access time + Time to determine hit/miss • Miss: data needs to be retrieved from a block in the lower level (Block Y) • Miss Rate = 1 - (Hit Rate) • Miss Penalty: Time to replace a block in the upper level + Time to deliver the block to the processor • Hit Time << Miss Penalty (500 instructions on 21264!)

21. 4 Questions for Memory Hierarchy • Q1: Where can a block be placed in the upper level? (Block placement) • Q2: How is a block found if it is in the upper level? (Block identification) • Q3: Which block should be replaced on a miss? (Block replacement) • Q4: What happens on a write? (Write strategy)

22. 01234567 01234567 01234567 1111111111222222222233 01234567890123456789012345678901 Q1: Where can a block be placed in the upper level? • Block 12 placed in 8 block cache: • Fully associative, direct mapped, 2-way set associative • S.A. Mapping = Block Number Modulo Number Sets Direct Mapped (12 mod 8) = 4 2-Way Assoc (12 mod 4) = 0 Full Mapped Cache Memory

23. Sources of Cache Misses • Compulsory(cold start or process migration, first reference): first access to a block • “Cold” fact of life: not a whole lot you can do about it • Note: If you are going to run “billions” of instruction, Compulsory Misses are insignificant • Capacity: • Cache cannot contain all blocks access by the program • Solution: increase cache size • Conflict(collision): • Multiple memory locations mappedto the same cache location • Solution 1: increase cache size • Solution 2: increase associativity • Coherence (Invalidation): other process (e.g., I/O) updates memory

24. Set Select Data Select Block Address Block offset Index Tag Q2: How is a block found if it is in the upper level? • Index Used to Lookup Candidates in Cache • Index identifies the set • Tag used to identify actual copy • If no candidates match, then declare cache miss • Block is minimum quantum of caching • Data select field used to select data within block • Many caching applications don’t have data select field

25. Block Size and Spatial Locality Block is unit of transfer between the cache and memory 4 word block, b=2 Word0 Word1 Word2 Word3 Tag block address offsetb Split CPU address b bits 32-b bits 2b = block size a.k.a line size (in bytes) • Larger block size has distinct hardware advantages • less tag overhead • exploit fast burst transfers from DRAM • exploit fast burst transfers over wide busses • What are the disadvantages of increasing block size? Fewer blocks => more conflicts. Can waste bandwidth.

26. 31 9 4 0 Cache Tag Cache Index Byte Select Ex: 0x01 Ex: 0x00 Valid Bit Cache Tag Cache Data : Byte 31 Byte 1 Byte 0 0 : 0x50 Byte 63 Byte 33 Byte 32 1 2 3 : : : : Byte 1023 Byte 992 31 Review: Direct Mapped Cache • Direct Mapped 2N byte cache: • The uppermost (32 - N) bits are always the Cache Tag • The lowest M bits are the Byte Select (Block Size = 2M) • Example: 1 KB Direct Mapped Cache with 32 B Blocks • Index chooses potential block • Tag checked to verify block • Byte select chooses byte within block Ex: 0x50

27. CS252-S10, Lecture 03 31 8 4 0 Cache Tag Cache Index Byte Select Cache Data Cache Tag Valid Valid Cache Tag Cache Data Cache Block 0 Cache Block 0 : : : : : : 1 0 Mux Sel1 Sel0 Compare Compare OR Hit Cache Block Review: Set Associative Cache • N-way set associative: N entries per Cache Index • N direct mapped caches operates in parallel • Example: Two-way set associative cache • Cache Index selects a “set” from the cache • Two tags in the set are compared to input in parallel • Data is selected based on the tag result

28. 31 4 0 Cache Tag (27 bits long) Byte Select Ex: 0x01 Cache Tag Valid Bit Cache Data : = Byte 31 Byte 1 Byte 0 : = Byte 63 Byte 33 Byte 32 = = : : : = Review: Fully Associative Cache • Fully Associative: Every block can hold any line • Address does not include a cache index • Compare Cache Tags of all Cache Entries in Parallel • Example: Block Size=32B blocks • We need N 27-bit comparators • Still have byte select to choose from within block

29. Q3: Which block should be replaced on a miss? • Easy for Direct Mapped • Set Associative or Fully Associative: • LRU (Least Recently Used): Appealing, but hard to implement for high associativity • Random: Easy, but – how well does it work?

30. Q4: What happens on a write? Additional option -- let writes to an un-cached address allocate a new cache line (“write-allocate”).

31. Lower Level Memory Cache Processor Write Buffer Holds data awaiting write-through to lower level memory Write Buffers for Write-Through Caches Q. Why a write buffer ? A. So CPU doesn’t stall Q. Why a buffer, why not just one register ? A. Bursts of writes are common. Q. Are Read After Write (RAW) hazards an issue for write buffer? A. Yes! Drain buffer before next read, or check write buffers for match on reads

32. 5 Basic Cache Optimizations • Reducing Miss Rate • Larger Block size (compulsory misses) • Larger Cache size (capacity misses) • Higher Associativity (conflict misses) • Reducing Miss Penalty • Multilevel Caches • Reducing hit time • Giving Reads Priority over Writes • E.g., Read complete before earlier writes in write buffer

33. Summary #1/2: The Cache Design Space Cache Size Associativity • Several interacting dimensions • cache size • block size • associativity • replacement policy • write-through vs write-back • write allocation • The optimal choice is a compromise • depends on access characteristics • workload • use (I-cache, D-cache, TLB) • depends on technology / cost • Simplicity often wins Block Size Bad Factor A Factor B Good Less More

34. Summary #2/2: Caches • The Principle of Locality: • Program access a relatively small portion of the address space at any instant of time. • Temporal Locality: Locality in Time • Spatial Locality: Locality in Space • Three Major Categories of Cache Misses: • Compulsory Misses: sad facts of life. Example: cold start misses. • Capacity Misses: increase cache size • Conflict Misses: increase cache size and/or associativity. Nightmare Scenario: ping pong effect! • Write Policy: Write Through vs. Write Back • Today CPU time is a function of (ops, cache misses) vs. just f(ops): affects Compilers, Data structures, and Algorithms