Parallel Programming in Undergraduate Education: A View from the Ground

1. Parallel Programming in Undergraduate Education: A View from the Ground Dan Grossman University of Washington Workshop on Directions in Multicore Programming Education March 8, 2009

2. A few hats • Researcher in programming languages • Semantics for transactional memory, etc. • Teacher at University of Washington • Freshmen through advanced graduate courses • Added 10% concurrency/parallelism to some courses • Current co-chair of department core-curriculum revision • Updating a 20-year-old course structure • Largely juggling 40 brilliant people’s perspectives Today’s opinions, all my own, informed by all • But focus on experience from (3) Dan Grossman: Undergraduate Parallel Programming

3. Trade-offs Do I believe every computer-science undergraduate needs to be prepared for the multicore era? Absolutely. But rather than “preaching to the choir”… What about: Web programming MapReduce Software security Software process UI design Machine learning Embedded systems Technical writing Business Co-op experience(s) Foreign language Shakespeare, Freud, Darwin, … And of course everything we already require? Dan Grossman: Undergraduate Parallel Programming

4. First things first So before what/how/where to teach multicore programming… What does an undergraduate curriculum actually look like? • The outdated one we’re changing • Where we’re probably heading • Similar to many of our peers • Where concurrency/parallelism currently appears Dan Grossman: Undergraduate Parallel Programming

5. A few details Each University has particular constraints University of Washington: • Top-ten department at a large public university • Very selective admission to the major • Quarter system: 10-week courses • Community-college transfers 2+2 years (in theory) • Arcane credit-hour rules and such None of this matters… except it does Walk through an undergraduate career… Dan Grossman: Undergraduate Parallel Programming

6. Curriculum overview Procedural programming in Java • 1800+ students a year • 88% won’t become CS majors • What should they see • Massive logistical infrastructure CS1 Dan Grossman: Undergraduate Parallel Programming

7. Curriculum overview CS1 CS2 ADTs, lists, trees, recursion, inheritance • 900+ students a year • 75% won’t become CS majors Dan Grossman: Undergraduate Parallel Programming

8. Curriculum overview CS1 “300-level” 8 required courses • back to this in a minute • what we are trimming / modernizing CS2 Dan Grossman: Undergraduate Parallel Programming

9. Curriculum overview CS1 “300-level” “400-level” CS2  20 choices • students take  5 •  1 per “area” • almost no sequences Dan Grossman: Undergraduate Parallel Programming

10. Curriculum overview CS1 “300-level” “400-level” CS2 synthesis experience • several options • widely loved capstone design Dan Grossman: Undergraduate Parallel Programming

11. 300-level today (8 required courses) Dan Grossman: Undergraduate Parallel Programming

12. 300-level today (8 required courses) discrete structures logic, proofs, sets, combinatorics, probability, … Dan Grossman: Undergraduate Parallel Programming

13. 300-level today (8 required courses) discrete structures formal models finite automata, regexps, context-free languages, Turing machines Dan Grossman: Undergraduate Parallel Programming

14. 300-level today (8 required courses) discrete structures formal models data structures big-O, balanced trees, heaps, hashing, sorting, graph algorithms, NP-completeness Dan Grossman: Undergraduate Parallel Programming

15. 300-level today (8 required courses) programming languages functional programming, static vs. dynamic typing, modularity, ML/Haskell, Scheme, Ruby discrete structures formal models data structures Dan Grossman: Undergraduate Parallel Programming

16. 300-level today (8 required courses) programming languages digital design boolean algebra, gates, binary numbers, finite automata, ALUs discrete structures formal models data structures Dan Grossman: Undergraduate Parallel Programming

17. 300-level today (8 required courses) programming languages digital design “303” discrete structures C, tools, “ethics”, everything else formal models data structures Dan Grossman: Undergraduate Parallel Programming

18. 300-level today (8 required courses) programming languages digital design “303” discrete structures formal models architecture data structures assembly programming, CPU design, caching, pipelining Dan Grossman: Undergraduate Parallel Programming

19. 300-level today (8 required courses) programming languages digital design “303” discrete structures formal models architecture statistics data structures probability distributions, regression, … Dan Grossman: Undergraduate Parallel Programming

20. programming languages digital design “303” discrete structures formal models architecture statistics data structures What’s the problem Too many requirements • Field now too large for everyone to take everything Outdated • In overall focus and details Have innovated within courses, especially parallelism (more later) • Me: 1 week C threads and shared-memory in “303” • Luis Ceze: 1 week in architecture • Note: Digital design inherently parallel too Dan Grossman: Undergraduate Parallel Programming

21. Where we’re going Revision in progress and highly tentative, but broad strokes are: • Fewer requirements • Cannot require the entire “embarrassed list” • Opportunity for earlier specialization • (cf. Berkeley, Cornell, Stanford, Texas, many others) Only touching 300-level for now • Free up room to modernize 400-level later • Little parallelism at 400-level today • Synchronization mechanisms in O/S • New elective course on MapReduce computing Dan Grossman: Undergraduate Parallel Programming

22. Required core probability for CS math foundations data structures + ?? modern software design hw/sw interface Dan Grossman: Undergraduate Parallel Programming

23. Optional core (50-90%) data mgmt C/C++ programming net-centric computing hardware design ?? ?? Dan Grossman: Undergraduate Parallel Programming

24. Where are we • Done: A top-down view of a curriculum • Now: How might we prepare students for multicore • Reorient entire curriculum • unlikely, risky • Add concurrency and parallelism courses • yes, but “we” don’t mean the same thing by that • Integrate concurrency and parallelism where relevant • my preference • some personal experience Dan Grossman: Undergraduate Parallel Programming

25. Being radical I’m not opposed to parallel-from-day-one • After all, I’m a functional programmer at heart • And we can avoid so much unnecessary sequential thinking But I think it’s unrealistic • What can most instructors teach freshmen in 10 weeks? Even though you can do it: “Educational experiments are doomed to succeed” Albert Shanker, former Pres. Amer. Federation of Teachers (Reminds me of “my” programming-languages course.) Dan Grossman: Undergraduate Parallel Programming

26. Senior-level courses • Yesteryear: Threads + synchronization mechanisms in O/S • Historical accident: only course that needed it • But: “a course on multicore stuff” means too many things • Architecture / very-low-level software • Concurrency* • primary challenge: responsiveness to external events • Example: O/S • Parallelism* • primary challenge: use resources to increase throughput • Example: scientific computing * other words we can agree on welcome Dan Grossman: Undergraduate Parallel Programming

27. Some great, very different things … Dan Grossman: Undergraduate Parallel Programming

28. To be clear • These are all great and important topics • Not just saying that because you are here • (But probably a dozen more books from people not here) • But they’re also really different • Topics, paradigms, emphases, levels of abstraction • Overlap < 30% (?) (probably seems even less to students) • Do we mean students should have: • One of these? • All of these? • “My way” • … Dan Grossman: Undergraduate Parallel Programming

29. My preference Add 1-2 weeks of parallelism to an existing core-topic course • Data structures / algorithms • Introductory architecture* • Programming languages • Compilers and run-time systems • C programming* (or Java, doesn’t matter) Briefly consider each of these, with more details on (5) • Something I’ve done 3 times with 2nd-year students *Actually done at Washington Dan Grossman: Undergraduate Parallel Programming

30. Data structures / algorithms • Amdahl’s Law • Span • The notion of dependencies • Thread-local versioning (caching)? Basically what Guy B. suggested Dan Grossman: Undergraduate Parallel Programming

31. Architecture Topics mentioned “at least in lecture” • Amdahl’s Law • Synchronization primitives (CAS, LLSC, etc.) • Mutual exclusion • Need for cache coherence • SIMD Dan Grossman: Undergraduate Parallel Programming

32. Programming languages • Threading and synchronization primitives • An interpreter supporting concurrent threads • Data races vs. application-level races • Data-race freedom Similar to coverage in Michael’s book chapter, but less is okay Note: In graduate courses, I teach Concurrent ML Dan Grossman: Undergraduate Parallel Programming

33. Compilers / run-time systems • “Threads Cannot Be Implemented as a Library” [Boehm 05] • Why nearly every compiler optimization violates sequential consistency • Maybe why concurrent and/or parallel garbage collection is hard Separate/related note: Would like more focus on “how Java is implemented” than on “how is C converted to assembly” Dan Grossman: Undergraduate Parallel Programming

34. C programming For students with 20 weeks of Java + 3 of C, I do this in 2 hours: • Race condition on a file via bash scripts • Thread basics (multiple call stacks, creation in C and Java) • Why use threads? Any one of: • Performance, Failure isolation, Responsiveness, Natural for the algorithm • Shared memory; the need to synchronize • Data races vs. higher-level races • Fork/join parallelism (in C and Java) • Bank-account example • First with atomic blocks (via Eclipse debugger plug-in) • Then with locks (discussing deadlock and false sharing) • One slide warning them against sequential consistency • So I’m not lying Dan Grossman: Undergraduate Parallel Programming

35. Why this methodology Some “director’s commentary” on this material • Using atomic blocks for pedagogy nicely separates: • Determining application’s critical sections (hard!) • Finding a good locking protocol (also hard!) • I heavily emphasize basic methodology: • Immutable data isn’t racy (remember, I’m a functional guy) • Thread-local data isn’t racy • Rarely acquire a lock while holding one • Obviously there is more to do, such as: • Communication beyond shared memory • Actually achieving speed-up or responsiveness About exposure to the issues, not (yet) about broad competence Dan Grossman: Undergraduate Parallel Programming

36. Conclusions • Computing undergoing a multicore revolution • But it’s okay for education to undergo a multicore evolution • Research should be radical and contrarian • Teaching experiments are doomed to succeed • Replace 10% of your course with parallelism and/or concurrency • Go ahead and teach parallelism to freshmen if you want to • Also add focused senior-level courses • Teach good habits even when assumption is sequential • High-level operations without unnecessary dependencies • Mostly functional • Don’t despair: always better to have too much to teach! Dan Grossman: Undergraduate Parallel Programming