The Closures Controversy

2. The Closures Controversy Joshua Bloch Chief Java Architect Google Inc.

3. Disclaimer: Talk Represents My Opinion, Not Google’s! Google believes that the Java platform would likely benefit from some additional support for closures. Like the broader Java community, we are divided on what form this support should take. Some favor a lightweight approach that addresses the pain of anonymous inner classes without affecting the type system, VM, or libraries; others favor a heavyweight approach designed to provide for programmer-defined control structures. We believe it is premature to launch a JSR that forces us down either path. www.javapolis.com

4. Outline I. Setting the Stage II. Why Enhance Support for Closures? III. BGGA Closures IV. A Lightweight Approach V. Where Do We Go From here? www.javapolis.com

5. OOPSLA Invited Talk October 8, 1996 Digitally reconstructed from the Internet Archive (AKA the Wayback Machine) by Joshua Bloch November 24, 2007

6. Oak • Started as a reimplementation of C++ • Always a tool, never an end itself • Took on a life of its own • The Web happened... • serendipitous match! • and it became Java

7. The Java Language Fusion of four kinds of programming • Object Oriented like Simula/C++/ ObjectiveC… • Numeric like FORTRAN • Systems like C • Distributed like nothing else

8. Java - a language for a job • Doing language research: • an anti-goal • Started using C++ • Broke down, needed: • Architecture neutral, portable, reliable, safe, long lived, multithreaded, dynamic, simple, ...

9. Practical, not theoretical • Driven by what people needed • (but hey, I spent too much time going to school!) • Theory provides • rigour • cleanliness • cohesiveness

10. No new ideas here • Shamelessly ripped off ideas that worked in C, C++, Objective C, Cedar/Mesa, Modula, Simula, ... • (well, we slipped once or twice and invented something)

11. Don’t fix it until it chafes • To keep it simple... • A procedural principle: • Require several real instances before including a feature • i.e. nothing goes in because it’s “nice” • (== “Just Say No, until threatened with bodily harm”)

12. Java feels... • Hyped :-( • Playful • Flexible • Deterministic • Non-threatening • Rich • Like I can just write code… Hey! I left out “Object-Oriented”!

13. IEEE Computer, June 1997

14. So How Are We Doing? Not So Well, Unfortunately Enum<E extends Enum<E>> { ... } <T extends Object & Comparable<? super T>> T Collections.max(Collection<? extends T>) public <V extends Wrapper<? extends Comparable<T>>> Comparator<V> comparator() { ... } error: equalTo(Box<capture of ?>) in Box<capture of ?> cannot be applied to (Box<capture of ?>) equal = unknownBox.equalTo(unknownBox) Arrays.asList(String.class, Integer.class) // Warning! See Angelia Langer's 427-page (!) Java Generics FAQ for more: http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.pdf www.javapolis.com

15. What the Man on the Web is Saying “I am completely and totally humbled. Laid low. I realize now that I am simply not smart at all. I made the mistake of thinking that I could understand generics. I simply cannot. I just can't. This is really depressing. It is the first time that I've ever not been able to understand something related to computers, in any domain, anywhere, period.” “I'm the lead architect here, have a PhD in physics, and have been working daily in Java for 10 years and know it pretty well. The other guy is a very senior enterprise developer (wrote an email system that sends 600 million emails/year with almost no maintenance). If we can't get [generics], it's highly unlikely that the ‘average’ developer will ever in our lifetimes be able to figure this stuff out.” www.javapolis.com

16. Where Does The Complexity Come From? Feature Tuples (exponential) Feature Pairs (quadratic) Features (linear) www.javapolis.com

17. If The Feel of Java is to be Preserved... • We simply cannot afford another wildcards • Further language additions must be undertaken with extreme caution • Minimal addition to conceptual surface area • High power-to-weight ratio www.javapolis.com

18. Outline I. Setting the Stage II. Why Enhance Support for Closures? II. BGGA Closures III. A Lightweight Approach IV. Where Do We Go From here? www.javapolis.com

19. What is a Closure? • One definition: a function that is evaluated in an environment containing one or more bound variables [Wikipedia] • In English: a little snippet of code that can be passed around for subsequent execution • Limited support for closures since JDK 1.1, in the form of anonymous classes www.javapolis.com

20. Why Are We Considering Better Support for Closures? • Fine-Grained Concurrecy - Passing snippets of code to fork-join frameworks using anonymous classes is a pain • Resource Managememnt - Using try-finally blocks for resource management is a pain, and causes resource leaks www.javapolis.com

21. 1. Fine-grained (fork-join) concurrency "It has to be easier to send snippets of code to frameworks for parallel execution; otherwise no one will use them .“ –Doug Lea, 2005 www.javapolis.com

22. Here’s How it Looks Today class StudentStatistics { ParallelArray<Student> students = ... // ... public double getMaxSeniorGpa() { return students.withFilter(isSenior). withMapping(gpaField).max(); } // helpers: static final class IsSenior implements Predicate<Student> { public boolean evaluate(Student s) { return s.credits > 90; } } static final IsSenior isSenior = new IsSenior(); static final class GpaField implements MapperToDouble<Student> { public double map(Student s) { return s.gpa; } } static final GpaField gpaField = new GpaField(); } www.javapolis.com

23. 2. Automatic Resource Management “The C++ destructor model is exactly the same as the Dispose pattern, except that it is far easier to use and a direct language feature and correct by default, instead of a coding pattern that is off by default and causing correctness or performance problems when it is forgotten.” –Herb Sutter, OOPSLA 2004 www.javapolis.com

24. How it Looks Today–Manual Resource Management static String readFirstLineFromFile(String path) throws IOException { BufferedReader r = null; String s; try { r = new BufferedReader(new FileReader(path)); s = r.readLine(); } finally { if (r != null) r.close(); } return s; } www.javapolis.com

25. It’s Worse With Multiple Resources (Puzzler 41) static void copy(String src, String dest) throws IOException { InputStream in = null; OutputStream out = null; try { in = new FileInputStream(src); out = new FileOutputStream(dest); byte[] buf = new byte[1024]; int n; while ((n = in.read(buf)) >= 0) out.write(buf, 0, n); } finally { closeIgnoringException(in); closeIgnoringException(out); } } private static void closeIgnoringException(Closeable c) { if (c != null) { try { c.close(); } catch (IOException ex) { // ignore } } } www.javapolis.com

26. Automatic Resource Management C++ Destructor String^ ReadFirstLineFromFile( String^ path ) { StreamReader r(path); return r.ReadLine(); } C# using Block String ReadFirstLineFromFile( String path ) { using ( StreamReader r = new StreamReader(path) ) { return r.ReadLine(); } } www.javapolis.com

27. Outline I. Setting the Stage II. Why Enhance Support for Closures? III. BGGA Closures IV. A Lightweight Approach V. Where Do We Go From here? www.javapolis.com

28. Controversial Features in BGGA • Function types • Non-local return • Non-local break and continue • Unrestricted access to nonfinal local variables • Design goal: library-defined control constructs www.javapolis.com

29. Function Types The BGGA Spec says: “While the subtype rules for function types may at first glance appear arcane, they are defined this way for very good reason: [...]. And while the rules seem complex, function types do not add complexity to Java's type system because function types and their subtype relations can be understood as a straightforward application of generics and wildcards (existing constructs). From the programmer's perspective, function types just ‘do the right thing.’” www.javapolis.com

30. Function Types are Hard to Read static Pair<{ => int },{ => int }> joinedCounters(int initial) { return Pair.<{ => int },{ => int }>of( { => initial++ }, { => initial++ }); } interface BThunk extends {=>boolean} { } static final {BThunk, { => void} => void} wihle = {BThunk cond, { => void } action => while (cond.invoke()) action.invoke(); }; static <throws X> { {=> void throws X} => void throws X }foo() { return { { => void throws X } block => block.invoke(); }; } These examples come from test code that ships with BGGA Prototype www.javapolis.com

31. Function Types Encourage an “Exotic” Programming Style static <A1, A2, R> {A1 => {A2 => R}} curry({A1, A2 => R} fn) { return {A1 a1 => {A2 a2 => fn.invoke(a1, a2)}}; } <A1, A2, A3, R> {A2, A3 => R} partial({A1, A2, A3 => R} fn, A1 a1); static <A1, A2, A3, R> {A2, A3 => R} partial({A1, A2, A3 => R} fn, A1 a1) { return {A2 a2, A3 a3 => fn.invoke(a1, a2, a3)}; } From Mark Mahieu's blog: Currying and Partial Application with Java Closures http://markmahieu.blogspot.com/2007/12/currying-and-partial-application-with.html www.javapolis.com

32. Nominal Types are RichCompared to Function Types Name → Known Implementations → Documentation, including semantic constraints {T, T => T} www.javapolis.com

33. Function Types Have Unexpected Interactions • Arrays don’t work Foo.java:6: generic array creation { => int}[] closures = new { => int}[N]; • Autoboxing doesn’t work LoopBenchC.java:10: <E,X>forEach(java.util.Collection<E>, {E => void throws X}) in LoopBenchC cannot be applied to (java.util.List<java.lang.Integer>,{int => void}) forEach(list, {int i => • Wildcards produce difficult error messages NewtonWithClosures.java:26: invoke(capture#418 of ? super {double => double}) in {capture#418 of ? super {double => double} => capture#928 of ? extends {double => double}} cannot be applied to (double) return fixedPoint(transform.invoke(guess)); ^ www.javapolis.com

34. Function Types Limit Interoperability With SAM Types • Closure conversion only works with interfaces • Unfortunately, existing APIs sometimes use abstract classes for functions • e.g., TimerTask, SwingWorker • These APIs would become 2nd class citizens www.javapolis.com

35. Summary - Pros and Cons of Function Types + Avoid need to define named interfaces + Avoid incompatibility among SAM types with same signatures - Hard to read under moderate-to-heavy use - Encourage “exotic” style of programming - Don't reflect semantic constraints - Don't provide the same level of documentation - Don’t interact well with autocompletion (or grep) - Limited interoperability may balkanize libraries www.javapolis.com

36. BGGA Closures Have Two Kinds of Returns static boolean test(boolean arg) { {boolean => boolean} closure = { boolean arg => if (arg) return true; // Non-local return false // local return }; return !closure.invoke(arg); } return means something completely different in a BGGA closure and an anonymous class www.javapolis.com

37. What Does test() Return? static <E> Boolean contains(Iterable<E> seq, Predicate<E> pred) { for (E e : seq) if (pred.invoke(e)) return true; return false; } static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, new Predicate<Character>() { public Boolean invoke(Character c) { return c > 'j'; } }); } interface Predicate<T> { Boolean invoke(T t); } www.javapolis.com

38. Now What Does test() Return? (BGGA) static <E> Boolean contains(Iterable<E> seq, {E => Boolean} p) { for (E e : seq) if (p.invoke(e)) return true; return false; } static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, { Character c => return c > 'j'; } ); } www.javapolis.com

39. Now What Does test() Return? (BGGA) static <E> Boolean contains(Iterable<E> seq, {E => Boolean} p) { for (E e : seq) if (p.invoke(e)) return true; return false; } static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, { Character c => return c > 'j'; }); } Accidental non-local return due to cut-and-paste from anonymous class can cause insidious bug www.javapolis.com

40. Suppose You Wanted to Translate this Method to BGGA static <E> Predicate<Iterable<E>> contains( final Predicate<E> pred) { return new Predicate<Iterable<E>>() { public Boolean invoke(Iterable<E> seq) { for (E e : seq) if (pred.invoke(e)) return true; return false; } }; } www.javapolis.com

41. It’s Awkward, as Only One Local Return is Permitted static <E> { Iterable<E> => Boolean } contains( { E => Boolean } pred) { return { Iterable<E> seq => Boolean result = false; for (E e : seq) { if (pred.invoke(e)) { result = true; break; } } result }; } www.javapolis.com

42. Summary - Pros and Cons of Non-Local Returns + Permits library-defined control structures - Having two kinds of returns is confusing - Meaning of return has changed - Unintentional non-local returns can cause bugs - Only one local return permitted per closure www.javapolis.com

43. What Does This Program Print? public class Test { private static final int N = 10; public static void main(String[] args) { List<{ => int}> closures = new ArrayList<{ => int}>(); for (int i = 0; i < N; i++) closures.add( { => i } ); int total = 0; for ({ => int} closure : closures) total += closure.invoke(); System.out.println(total); } } www.javapolis.com

44. What does this program print? public class Test { private static final int N = 10; public static void main(String[] args) { List<{ => int}> closures = new ArrayList<{ => int}>(); for (int i = 0; i < N; i++) closures.add( { => i } ); int total = 0; for ({ => int} closure : closures) total += closure.invoke(); System.out.println(total); } } It prints 100, not 45. The same loop variable is captured by all 10 closures and evaluated after the loop is finished! www.javapolis.com

45. Summary - Pros and Cons of Access to Nonfinal Locals + Permits library-defined control structures + Eliminates some uses of final modifier - Semantics can be very confusing - Locals can persist after their scope is finished - Locals can be modified by other threads - Performance model for locals will change www.javapolis.com

46. Library-Defined Control Constructs • What are the compelling use-cases? • Custom loops • Automatic resource management blocks • Timer block www.javapolis.com

47. Custom for-loops (Example from BGGA Spec) <K,V,throws X> void for eachEntry(Map<K,V> map, {K,V=>void throws X} block) throws X { for (Map.Entry<K,V> entry : map.entrySet()) { block.invoke(entry.getKey(), entry.getValue()); } } for eachEntry(String name, Integer value : map) { if ("end".equals(name)) break; if (name.startsWith("com.sun.")) continue; System.out.println(name + ":" + value); } www.javapolis.com

48. What's Wrong With This example? • BGGA loop competes with Java 5 for-each • Don't define a construct; just implement Iterable • Only compelling use is multiple loop variables • Last example doesn't offer power of for-each • Loop variable can’t be primitive (no auto-unboxing) • Would require 81 (!) overloadings to do fix this www.javapolis.com

49. Loop syntax tailored to for; awkward for while public static <throws X> void for myWhile( {=> boolean throws X} cond, {=>void throws X} block) throws X { while (cond.invoke()) { block.invoke(); } } for myWhile( { => i < 7 } ) { System.out.println(i++); } myWhile( { => i < 7 }, { => System.out.println(i++); }); www.javapolis.com

50. Automatic Resource Management Block (BGGA Spec) <R, T extends Closeable, throws X> R with(T t, {T=>R throws E} block) throws X { try { return block.invoke(t); } finally { try { t.close(); } catch (IOException ex) {} } } with (FileReader in : makeReader()) { // Requires nesting with (FileWriter out : makeWriter()) { // code using in and out } } www.javapolis.com