Heterogeneous Multi-Core Processors

1. Heterogeneous Multi-Core Processors Jeremy Sugerman GCafe May 3, 2007

2. Context • Exploring the CPU and GPU future relationship • Joint work, thinking with Kayvon • Much kibbitzing from Pat, Mike, Tim, Daniel • Vision and opinion, not experiments and results • More of a talk than a paper • The value is more conceptual than algorithmic • Wider gcafe audience appeal than our near term elbows-deep plans to dive into GPU guts

3. Outline • Introduction • CPU “Special Feature” Background • Compute-Maximizing Processors • Synthesis, with Extensions • Questions for the Audience…

4. Introduction • Multi-core is status quo for forthcoming CPUs • Variety of emerging (for “general purpose”) architectures try to offer discontinuous performance boost over traditional CPUs • GPU, Cell SPEs, Niagara, Larrabee, … • CPU vendors have a history of co-opting special purpose units for targeted performance wins: • FPU, SSE/Altivec, VT/SVM • CPUs should co-opt entire “compute” cores!

5. Introduction • Industry is already exploring hybrid models • Cell: 1 PowerPC and 8 SPEs • AMD Fusion: Slideware CPU + GPU • Intel Larrabee: Weirder, NDA encumbered • The programming model for communicating deserves to be architecturally defined. • Tighter integration than the current “host + accelerator” model eases porting and efficiency. • Work queues / buffers allow intregrated coordination with decoupled execution.

6. Outline • Introduction • CPU “Special Feature” Background • Compute-Maximizing Processors • Synthesis, with Extensions • Questions for the Audience…

7. CPU “Special Features” • CPUs are built for general purpose flexibility… • … but have always stolen fixed function units in the name of performance. • Old CPUs had schedulers, malloc burned in! • CISC instructions really were faster • Hardware managed TLBs and caches • Arguably, all virtual memory support

8. CPU “Special Features” • More relevantly, dedicated hardware has been adopted for domain-specific workloads. • … when the domain was sufficiently large / lucrative / influential • … and the increase in performance over software implementation / emulation was BIG • … and the cost in “design budget” (transistors, power, area, etc.) was acceptable. • Examples: FPUs, SIMD and Non-Temporal accesses, CPU virtualization

9. Outline • Introduction • CPU “Special Feature” Background • Compute-Maximizing Processors • Synthesis, with Extensions • Questions for the Audience…

10. Compute-Maximizing Processors • “Important” common apps are FLOP hungry • Video processing, Rendering • Physics / Game “Physics” • Even OS compositing managers! • HPC apps are FLOP hungry too • Computational Bio, Finance, Simulations, … • All can soak vastly more compute than current CPUs can deliver. • All can utilize thread or data parallelism. • Increased interest in custom / non-”general” processors

11. Compute-Maximizing Processors • Or “throughput oriented” • Packed with ALUs / FPUs • Application specified parallelism replaces the focus on single-thread ILP • Available in many flavours: • SIMD • Highly threaded cores • High numbers of tiny cores • Stream processors • Real life examples generally mix and match

12. Compute-Maximizing Processors • Offer an order of magnitude potential performance boost… if the workload sustains high processor utilization • Mapping / porting algorithms is a labour intensive and complex effort. • This is intrinsic. Within any design budget, a BIG performance win comes at a cost… • If it didn’t, the CPU designers would steal it.

13. Compute-Maximizing Programming • Generally offered as off-board “accelerators” • Data “tossed over the wall” and back • Only portions of computations achieve a speedup if offloaded • Accelerators mono-task one kernel at a time • Applications are sliced into successive statically defined phases separated by resorting, repacking, or converting entire datasets. • Limited to a single dataset-wide feed forward pipeline. Effectively back to batch processing

14. Outline • Introduction • CPU “Special Feature” Background • Compute-Maximizing Processors • Synthesis, with Extensions • Questions for the Audience…

15. Synthesis • Add at least one compute-max core to CPUs • Workloads that use it get BIG performance • Programmers are struggling to get any performance from having more normal cores • Being “on-chip” architected and ubiquitous is huge for application use of compute-max • Compute core exposed as programmable independent multithreaded execution engine • A lot like adding (only!) fragment shaders • Largely agnostic on hardware “flavour”

16. Extensions • Unified address space • Coherency is nice, but still valuable without it • Multiple kernels “bound” (loaded) at a time • All part of the same application, for now • “Work” delivered to compute cores through work queues • Dequeuing batches / schedules for coherence, not necessarily FIFO • Compute and CPU cores can insert on remote queues

17. Extensions CLAIM: Queues break the “batch processing” straitjacket and still expose enough coherent parallelism to sustain compute-max utilization. • First part is easy: • Obvious per-data element state machine • Dynamic insertion of new “work” • Instead of being idle as the live thread count in a “pass” drops, a core can pull in “work” from other “passes” (queues).

18. Extensions CLAIM: Queues break the “batch processing” straitjacket and still expose enough coherent parallelism to sustain compute-max utilization. • Second part is more controversial: • “Lots” of data quantized into a “few” states should have plentiful, easy coherence. • If the workload as a whole has coherence • Pigeon hole argument, basically • Also mitigates SIMD performance constraints • Coherence can be built / specified dynamically

19. Outline • Introduction • CPU “Special Feature” Background • Compute-Maximizing Processors • Synthesis, with Extensions • Questions for the Audience…

20. Audience Participation • Do you believe my argument conceptually? • For the heterogeneous / hybrid CPU in general? • For queues and multiple kernels? • What persuades you 3 x86 + compute is preferable to quad x86? • What app / class of apps and how much of a win? 10x? 5x? • How skeptical are you that queues can match the performance of multi-pass / batching? • What would you find a compelling flexibility / expressiveness justification for adding queues? • Performance wins regaining coherence in existing branching/looping shaders? • New algorithms if shaders and CPU threads can dynamically insert additional “work”?