Functional Reactive Programming, Resource Types, and Wormholes

1. Functional Reactive Programming, Resource Types, and Wormholes Paul HudakYale UniversityDepartment of Computer Sciencehaskell.cs.yale.edu(Joint work with Dan Winograd-Cort)New York Haskell Users GroupOctober 30, 2013

2. Background • Time-varying quantities in a PL is due to Conal Elliott (early work in C++ for animation) • Fran: Functional Reactive Animation, by Elliott and Hudak: FRP implemented in Haskell, showing elegance of higher-order functions, type classes, and so on. • Yampa: at Yale; an arrow-based FRP in Haskell; avoided insidious time- and space leaks; improved modularity. • Since then, tons of new papers, both theoretical and applied: FranTk, Frob, FVision, Reactive Banana, FPorter, … • Antony Courtney: Fruit– a GUI based on Yampa • Today will talk about a MUI(musical user interface) implemented in Euterpea (Haskell library for computer music).

3. Euterpea: Computer Music in Haskell Why do this? After all, there are already many languages for computer music… Motivation: • A required two-course computer-music sequence in the Computing and the Arts major at Yale • The use of modern, state-of-the-art PL ideas in a fun area • To combine my research with my hobby • To use music to teach (functional) programming • To give composers powerful tools to enhance creativity Last term, out of 20 students, 3 were Computing and the Arts majors, one was a graduate student, and the rest were undergrads {mostly CS majors) with an interest in computer music, or a desire to learn Haskell.

4. Euterpea “Euterpea” is derived from “Euterpe”, who was one of the nine Greek muses, or goddesses of the arts, specifically the muse of music.

5. Signals and Signal Functions • Signals are time-varying quantities. • Conceptually they can be thought of as functions of time:Signal a = Time → a • For example: • a slider is a time-varying number • a mouse is a time-varying cartesian coordinate • For efficiency and modularity, we use arrowsto abstract away from signals, and instead use signal functions. Conceptually:SigFun a b = Signal a -> Signal b • Haskell’s Arrowtype class make this especially easy.

6. Key Idea • This signal processing diagram: • is equivalent to this Euterpean code, using “arrow” syntax:y <- sigfun -< x • In turn this is part of a larger syntactic unit, called a proc. E.g.: proc x -> do y <- sf1 <- x z <- sf2 <- y+1 return z*2 • By analogy to an anonymous function: \ x -> let y = f1 x z = f2 (y+1) in z*2 sigfun y x

7. Four Useful Functions • To “lift” a function to a signal function:arr :: (a → b) → SF a b • Example: arr (+1) is a signal function that adds one to its input • To “lift” a constant to a constant signal function:constA :: b → SF () b • Example: constA 440 returns a steady value of 440 • To “compose” to signal functions:(>>>) :: SF a b → SF b c → SF a c(<<<) :: SF b c → SF a b → SF a c • Example: constA 440 >>> arr (+1)is the same as constA 441

8. init: sin ωh init: 0 z-1 z-1 d1 d2 - y * c Audio Example: Sine Wave • From basic trigonometry:sin(nωh) = 2 cos(ωh) sin((n-1) ωh) – sin((n-2) ωh) where ω = 2πf • We can derive a recurrence equation [Goertzel]:y(0) = 0 y(1) = sin ωh y(n) = c · y(n-1) - y(n-2) where c = 2 (cos ωh) • Diagrammatically:

9. init: sin ωh init: 0 z-1 z-1 d1 d2 - y * c Rendered in FRP y(0) = 0 y(1) = sin ωh y(n) = c · y(n-1) - y(n-2) where c = 2 (cos ωh) sine :: Double -> SF () Double sine freq = let omh = 2*pi*freq/sr d = sin omh c = 2 * cos omh in proc _ -> do rec let y = c*d1-d2 d2 <- delay 0 -< d1 d1 <- delay d -< y returnA -< y

10. lineSeg envibr sinA 5 rand 1 * 0.1 flow × emb Embouchure delay delayt (1/fqc/2) vibr * breath + sum1 x – x3 lowpass Flute bore delay delayt (1/fqc) + + × x out * amp returnA env2 bore * feedbk2 lineSeg lineSeg * feedbk1 flute env1 Physical Model of a Flute

11. Flute Model in Euterpea flute dur freq amp vfreq= in proc () -> do amp1 <- linseg … -< () amp2 <- linseg … -< () ampv <- linseg … -< () flow <- rand 1 -< amp1 vibr <- oscils vfreq -< 0.1 * ampv rec let feedbk = body * 0.4 body <- delay (1/freq) -< out x <- delay (1/freq/2) -< breath*flow + amp1 + vibr + feedbk out <- tone -< (x - x*x*x + feedbk, 2000) returnA -< out*amp*amp2

12. Flute Demo • f0 and f1 demonstrate the change in the breath parameter.f0 = flute 3 0.35 440 0.93 0.02 f1 = flute 3 0.35 440 0.83 0.05 • f2 has a weak pressure input so only plays the blowing noise.f2 = flute 3 0.35 440 0.53 0.04 • f3 takes in a gradually increasing pressure signal.f3 = flute 4 0.35 440 (lineSeg [0.53, 0.93, 0.93] [2, 2]) 0.03 • Sequence of notes:

13. Events • Signals are not enough… some things happen discretely. • Events can be realized as a kind of signal:data Maybe a = Nothing | Just a -- pre-defined in Haskell type SEventa = Maybe a • For example, MIDI events have type:SEvent [MidiMessage]where MidiMessageencodes standard MIDI messages such as Note-On, Note-Off, etc. • And a signal function over events would have type:SF (SEvent T1) (SEvent T2) This overall paradigm is often called Functional Reactive Programming (FRP)

14. Musical User Interface (MUI) • Design philosophy: • GUI’s are important! • The dataflow metaphor (“wiring together components”) is powerful! • Yet graphical user interface programming is inefficient… • Goal: MUI widgets using FRP • The MUI arrow is called UISF • UISF a b maps time-varying values (or event streams) of type a, to those of type b • Kinds of UISF signal functions: • Widget constructors • Widget transformers • I/O • Mediators

15. Widget Constructors • display :: Show a ⇒ UISF a () • textbox :: String → UISF String String • button :: String → UISF () Bool • checkbox :: String → Bool → UISF () Bool • radio :: [String ] → Int → UISF () Int • hSlider , vSlider :: RealFrac a⇒ (a, a) → a → UISF () a • hiSlider , viSlider :: Integral a⇒ a → (a, a) → a → UISF () a (Run mui0 from Examples/MUI.hs)

16. MUI Transformers • title :: String → UISF a b → UISF a b • withDisplay :: Show b ⇒ UISF a b → UISF a b • pad :: (Int, Int , Int, Int ) → UISF a b → UISF a b • topDown, bottomUp, leftRight , rightLeft :: UISF a b → UISF a b • setLayout :: Layout → UISF a b → UISF a b • makeLayout :: LayoutType → LayoutType → Layout • data LayoutType = Stretchy { minSize :: Int } | Fixed { fixedSize :: Int } (Run mui2 from Examples/MUI.hs)

17. Mediators • unique :: Eq a ⇒ UISF a (Event a) -- generates an event whenever the input changes • edge :: UISF Bool (Event ()) -- generates an event when input changes from False to True • accum :: a → UISF (Event (a → a)) a -- accum x starts with the value x , but then applies the function -- attached to the first event to x to get the next value, and so on. • mergeE :: (a → a → a) → Event a → Event a → Event a -- mergeE f e1 e2 merges two event streams, using f to resolve -- simultaneous events • hold :: b → UISF (Event b) b -- hold x begins as value x , but changes to the subsequent values -- attached to each of its input events • now :: UISF () (Event ()) -- creates a single event “right now” (Run mui3, mui4 from Examples/MUI.hs)

18. Time and Timers • time :: UISF () Time -- Returns current time -- Time is a type synonym for Double • timer :: UISF (Time, Double) (SEvent ()) -- Emits events at given (dynamic) rate (Run mui6 from Examples/MUI.hs)

19. Bifurcate Me, Baby! Gary Lee Nelson 1995

20. Behind the Scene • Consider the recurrence equation:xn+1 = r * xn * (1 - xn)Start with an initial population x0 and iteratively apply the growth function to it, where r is the growth rate. For certain values of r, the population stabilizes, but as r increases, the period doubles, quadruples, and eventually leads to chaos.It is one of the classic examples in chaos theory. • In Euterpea: grow r x = r * x * (1-x) • Let’s wrap it up in a MUI. (Run bifurcatefrom Examples/MUI.hs)

21. Modular? A program is a single signal function; it must thread all IO actions through the same input and output. electric piano MIDI synthesizer gamecontroller run-time system main program(a signal function) mouse sound card console I/O MIDI instrument printer

22. Maybe Not… I/O bottle-neck; notransparency I/O bottle-neck; notransparency electric piano MIDI synthesizer gamecontroller run-time system main program(a signal function) mouse sound card console I/O MIDI instrument printer

23. Solution: Reify Real-World Objects Expand main program to subsumevirtualizedIO devices! electric piano MIDI synthesizer gamecontroller run-time system main program(a signal function) mouse sound card console I/O MIDI instrument printer

24. IO devices now treated just like other signal functions. • As we have seen, can also virtualize virtual objects (widgets). electric piano MIDI synthesizer main program gamecontroller sf1 sf2 sf3 sf4 mouse sound card sf5 sf7 console I/O sf6 MIDI instrument printer

25. A Problem: Resource Duplication • Consider this code fragment:() <- midiOut <- noteList1 () <- midiOut <- noteList2midiSynth is an output device, but there are two occurrences -- what is the result? Interleaving? Non-determinism? • Likewise, here is an example of input:notes1 <- midiIn <- () notes2 <- midiIn <- ()midiIn is an input device. So do notes1 and notes2 return the same result, or are they different?

26. Solution: Resource Types • Tag each virtualized object with a unique resource type to prevent duplication.midiOut :: SF (S MidiOut) MidiMsg () midiIn :: SF (S MidiIn) () MidiMsg • The first argument to SF is a set of resource types;S MidiOut and S MidiIn are singletonsets. • With these types, the previous code fragmentswill not type-check – resource types of composed signal functions must be disjoint.

27. Resource Types • A value of type SF r a bmaps signals of type a into signals of type b, while “consuming” the set of resources r. • If sf1 :: SF r1 a b and sf2 :: SF r2 b c, then in sf1 >>> sf2, r1 and r2 must be disjoint. • In which case sf1 >>> sf2 :: SF (r1 U r2) a c. • A restricted form of resource types can be implemented in Haskell using higher-order types, type families, etc. • The full version requires type features that don’t exist.

28. Effects in FRP • How can we perform effects in the FRP framework? • Outside of FRP, we use monads: • IO is useful for general I/O effects such as on real-world devices, but because we cannot escape it, it is suboptimal for localized effects. • ST is useful for mutable data structures, as it is escapable after its localized effects:runST:: (forall s. ST s a) -> aThe “phantom” type s provides the safety.

29. Performing Effects • IO and ST are inherently different • IO is for real-world objects that cannot be created on the fly (e.g. printer, speaker, keyboard, etc.). • ST is for mutable data structures that can be allocated dynamically. • Both types of effects require sequencing, which the monadic structure provides.

30. Sequencing • How can we do sequencing in FRP without monads? • Normal Haskell variables are time-invariant, but FRP values are conceptually time-varying. • Thus, the sequencing provided by monads can be achieved in FRP by using the ordering of eventsin an event stream. • Indeed, we have seen this already with midiIn and midiOut, using resource types to ensure safety. • But what about mutable data types?

31. Mutable Datatypes in FRP • An example of a mutable data structure: • Mutable array can be represented by a signal functions that handle mutation requests: • Each sfArraywill produce a new mutable array: • We do not need resource types. • We do not need a phantom type. (Event Response) (Event Request) sfArray

32. Mutable Datatype in FRP • An even simpler mutable data structure example would be a single cell of memory. • The input would be the new value to store and the output, the previous value in the cell: • Essentially, this acts like a delay function. We call it initin reference to our previous work on Causal Commutative Arrows. init

33. Splitting an Atom • Let’s examine initmore closely: • What if we split the reading and writing? it in

34. Splitting an Atom • Let’s examine init more closely: • What if we split the reading and writing. • We call this a wormhole. • The blackholeis for writing • The whiteholeis for reading blackhole whitehole

35. Wormholes • We can use wormholes as non-local, one-way communication channels. SF 1 SF 2

36. Wormholes • We can use wormholes as non-local, one-way communication channels. SF 1 SF 2

37. Wormholes • We can use wormholes as non-local, one-way communication channels. whitehole SF 1 blackhole SF 2

38. Wormholes • We can use wormholes as non-local, one-way communication channels.

39. “Bottles” • Composed and implemented by Donya Quick, a grad student in CS • Done entirely in Euterpea • Demonstrates: • High-level composition • Notes, repetitions, structure • Micro-tonal support • Low-level signal processing • Both “struck” bottle and “blown” bottle sounds,using physical modeling • Some additive synthesis as well • Reverb and audio processing

40. Thank You!! (any questions?) For more information about Euterpeaand our PL research, see: haskell.cs.yale.edu

41. Implementing Signals • As “yet another abstraction”, arrows might seem to introduce yet more computational overhead. • But in fact arrows eliminate certain important classes of space leaks. • Furthermore, causal commutative arrows (CCA) can improve performance dramatically: • Any CCA expression can be normalized into an expression with one loop and one vector of state variables. • This results in a factor of 50 improvement over conventional implementation methods. • Good enough for non-trivial real-time performance.