ChucK! Part 1: Basics of digital audio and ChucK

1. ChucK! Part 1: Basics of digital audio and ChucK Rebecca Fiebrink Princeton University

2. About me • PhD student in Computer Science at Princeton • background in computer science, classical flute, and music technology • Research: ChucK, live computer performance, human-computer interaction, and applied machine learning in music • Performer and assistant director of the Princeton Laptop Orchestra

4. Goals • Familiarity with digital audio basics (may already be familiar to some people) • Comfort with basic ChucK syntax and language constructs • Understanding what is possible to do with ChucK and associated tools • Knowledge of how start building things, where to go for more information and help • Don’t be afraid to ask questions throughout the workshop!

5. Outline of Part 1 • Introduction • Audio basics • Digital audio basics • ChucK programming basics • A ChucK sound synthesis tutorial: thinking about creating and working with sound at a low level • Coming in Part 2+: PLOrk, interaction with devices and other software, audio analysis, applied machine learning

6. Audio Basics What is sound? “Pressure wave”

7. What do we hear? • Frequency of sound pressure waves: pitch • 20-20,000Hz • Amplitude of pressure waves: loudness

8. Sounds include many frequencies • All “real-world” sounds contain many different frequencies • Harmonic frequencies are present in pitched sounds (violin, voice) • A violin playing an ‘A’ actually plays frequencies of 440Hz, 2x440=880Hz, 3x440=1320Hz,etc. • Inharmonic frequencies are present in other sounds (percussive, noisy) • A drum hit might include frequencies of 300Hz, 412Hz, 1192Hz, 4300Hz, etc. • Pitched instruments also typically produce inharmonic frequencies. • In music: identities and strengths of frequencies present contribute to the timbre. (We’ll see examples of this later.) • In speech, these differentiate vowels and consonants. (This suggests one method of synthesizing speech.)

9. Time-domain & Frequency-domain representations • Fourier theorem: We can represent any sound as a summation of sine waves at different frequencies, with different magnitudes and phase offsets • If we know the frequency, magnitude and phase of every component for all points in time, this is an equivalent representation to the wave itself

10. Demo: sndpeek • Run sndpeek (on my website, or at http://soundlab.cs.princeton.edu/software/sndpeek/ ) • Press ‘w’ to toggle waterfall display, ‘q’ to quit

11. Digital audio • Sound pressure in the real world is continuous (analog) in time and real-valued in amplitude. • Computers require a discrete (non-continuous) representation in both time and amplitude. • Analog to digital conversion: • Sample to get discrete time values • Quantize to get discrete amplitude values

12. Sampling • Converting a continuous wave in time into “samples” taken at specific moments in time. • Nyquist’s theorem: We can perfectly reproduce a sampled sound if the original contains frequencies no higher than ½ the sample freq. • CD quality: 44100 Hz (represents all frequencies up to 22050 Hz) Example: 44.1kHz, 4kHz

13. Another sampling picture

14. Quantization • Converting a real-valued amplitude value into a value represented by some finite number of bits. • Important: Higher numbers of bits allow us to represent more dynamic range between soft and loud • CD quality: 16 bits (per channel)

15. A/D and D/A Conversion • A/D conversion • E.g., recording from a microphone into a computer • D/A conversion • E.g., loading a .wav file and playing through a speaker, or synthesizing samples from scratch and playing through a speaker • Done in hardware • In digital audio, we can manipulate the samples directly and not worry (too much) about the analog domain • Typically have DAC and ADC objects we reference from the code

16. Music programming languages • With a high enough sample rate, and ability to manipulate samples, we can create any sound imaginable! • But how should we manipulate the samples?? • Music programming languages have evolved to provide precise control over the tools for sound synthesis and composition • 1960’s-70’s: First formal languages (MUSIC-N, CARL) • 1980’s-90’s: Evolving languages and libraries; real-time control (Max/MSP, CSOUND, STK, Nyquist) • 2000’s: Live-coding, rich interactivity (Supercollider, ChucK)

17. Freq = 100 Cut-off frequency=500, Q Square wave Low-pass filter DAC Unit generators • Each “algorithm” for producing or modifying sound is a unit generator • E.g. sine wave, square wave, filters • A synthesis algorithm can be represented by one or more unit generators chained together.

18. Let’s ChucK! • All examples are online at http://www.cs.princeton.edu/~fiebrink/hk_workshop/ • Feel free to steal them as starting points for your own projects. • See resources.doc for list of ChucK resources! • Next up: • Chuck’s unit generator paradigm, syntax • Time in Chuck • A look at common synthesis methods via ChucK

19. Freq = 440 Sine wave DAC A very simple UGen network See: instantiate.ck

20. The ChucK Operator • Use 1: Virtual “patch cord” between unit generators • SinOsc s => dac; • Noise n => LPF filter => dac; • Use 2: Assignment operator • 5.0 => float f; // Just like “float f = 5.0;” in C, Java, etc. See: time.ck

21. Time in ChucK • Two jobs of the virtual machine: • 1. Interpret ChucK code line by line, following instructions of the code • 2. Output a stream of audio samples to the system DAC’s buffer • ChucK ONLY outputs audio samples when time is “advanced”!! • E.g., 1::second => now; or 1::samp => now; • Advancing time ensures that exactly the number of audio samples are produced • All the other work of ChucK (step 1 above) happens in between outputting samples, as if in a single instant in time See: while-loop.ck

22. Parameters: pos, rate, loop, other SndBuf DAC Playing back stored audio • SndBuf is the UGen for reading a file and playing it back. • See sndbuf.ck • Also note: randomization using Std.rand2f

23. OO basics in ChucK • ChucK is object-oriented (like C++, Java) • Basic components are objects (e.g., SinOsc is a type of object) • You must instantiate objects, creating instances (e.g., “s”) • Objects have methods which can be called to modify or access their parameters (also called member variables) • E.g., s.freq() • ChucK syntax: • Get a value: s.freq() => float freq; • Set a value: 400 => s.freq; Or: f1 => s.freq; //For an existing float called f1 Or: s.freq(400); //Alternative Java-style syntax • Find out about UGen methods on the ChucK webpage: http://chuck.cs.princeton.edu/doc/program/ugen.html See: while-loop.ck

24. Physical modeling in ChucK • Physical models capture essential process of how physical objects produce sound • To model a mandolin: • What happens when you pluck a string? • How is the sound influenced by body size, pluck location, pluck force, etc.? • The Synthesis Toolkit (STK): a set of physical models by Perry Cook and Gary Scavone, ported to ChucK • Allows you to make “decent” sound immediately using default model parameters • All StkInstruments implement noteOn() and noteOff(), as well as instrument-specific methods (see webpage). See: mandolin.ck, clarinet.ck, mandolin2.ck

25. Additive synthesis • Add together several waves with different frequencies and amplitudes (and phases). • Recall: All sounds can be decomposed into a set of sine waves (Fourier) • Our ear tends to group different waves into one sound event under certain conditions • Harmonic frequencies, synchronous onsets, synchronous vibrato • Examples: additive_noarrays.ck, additive_sines_array.ck, additive_with_vibrato.ck

26. Spectral analysis • Use a Fourier transform / FFT to analyze the frequency content of a sound • Output: a spectrum indicating the strengths of all component frequencies (and their phases) • Can perform FFT in Audacity and ChucK (among others) Singing bowl strike: Peaks at 528, 1413, 2630, 4127, and 5877 Hz (add_bowl.ck)

27. Envelopes • Time-domain behavior is important, too! • Singing bowl in time domain: exponential decay • Ex: add_bowl_exp_envelope.ck • More advanced: Consider decay of individual partials • Ex: add_bowl_exp_envelope_per_component.ck Frequency  Time 

28. Built-in Envelopes • Example: ADSR: attack, decay, sustain, release Parameterss See: sine_ADSR.ck, add_bowl_with_ADSR Oscillator (e.g. SinOsc) Envelope DAC

29. Subtractive synthesis and filters • Filters modify the spectrum (frequency content) of a sound • Low-pass, band-pass, high-pass, and band-stop filter out some frequencies • Filters can also boost some frequencies (e.g. add resonances) • Often would like to start with a sound with a rich spectrum: Noise, Impulse train, Square wave, etc. • lpf.ck, lpf_mouse.ck, resonance.ck, resonance_mouse.ck, run_clix.sh (run in terminal by typing ./20_run_clix.sh)

30. Simple speech synthesis • Filter noise or an impulse train by adding resonances at formant locations • See: voice_synth.ck “Ahh” “Ooh”

31. Cross-synthesis • Filter the spectrum of one sound using the spectrum of another sound • Ex: xsynth_robot123.ck, xsynth_guitar123.ck • Voice spectrum taken from:

32. Getting audio input and using it • adc is another important UGen (Analog-Digital Converter) • Grab audio samples from a mic or line in • You might have to configure miniAudicle to use the correct input (miniAudicle->Preferences) • Audio example: Plork beat science flute processing: • Example: echo.ck • Be careful of feedback! External mic is best. • Example: trueman_munger.ck, adc_munger.ck • Live sampling: Chop up input into small grains, playback a cloud of grains whose pitch/amplitude have been modified in some way • Xsynth_guitar_live, xsynth_robot_live, and xsynth_noise_live do cross-synthesis from live mic

33. FM synthesis • Discovered by John Chowning in 1967-68 • Basis for Yamaha DX7 synthesizer • Algorithm: Use the output of one sine wave to modulate the frequency of another sine wave • Outputs a very rich sound palette, highly sensitive to parameters See: FM_byhand, FM_mouse, FM_stk

34. Putting the pieces together • Mix and match UGens (e.g., filter a SndBuf, apply envelopes to STK instruments, etc.) • Add multiple voices or parts in two ways: • “Add shred” in miniAudicle • See: Moe/Larry/Curly, OTF ChucK • Also note: OTF synchronization method • Call spork~ to spork a new child shred • Important: A child shred only stays “alive” as long as its parent (don’t forget to keep advancing time in the parent) • Wee: shred1.ck to shred4.c • Also note: defining your own functions

35. What makes ChucK unique? • Sample-level control • E.g. see FM_byhand.ck, while-loop.ck • Concurrency for free • Spork or add as many shreds as you want; they’ll remain in synch with each other • E.g., see OTF examples, shred4.ck • Support for audio analysis (Unit Analyzers) within a music language • Leads to support for real-time AI / machine learning and music information retrieval • More on this in Part 3!

36. Coming up in Part 2 • Making ChucK interactive!