Sound Synthesis

1. Sound Synthesis Part I: Introduction & Fundamentals Nicolas Pugeault n.pugeault@surrey.ac.uk http://info.ee.surrey.ac.uk/Personal/N.Pugeault/

2. Introduction • Instruments can be made in a variety of ways: think guitar, piano, organs, etc. • Use electronic devices to create sounds: synthesisers. • Can either • Recreate an existing timbre • … or something different.

3. Introduction • Producing a sound by sending an electrical signal to a speaker is trivial. • The question is what are the relevant and desirable properties of the signal to ensure that the resulting sound is as desired (eg, similar to a real instrument).

4. Applications • Musical instruments • Computer games • Sound effects for films • Multimedia, computer system sounds • Mobile phones • Speech synthesis • Toys • Effects

5. Sound Synthesis Plan • Synthesis I: Fundamentals • Synthesis II: Additive • Synthesis III: Filtering and distortion • Synthesis IV: Other approaches • Post-processing, pitch correction (autotunes) • Sound perception

6. Sound Synthesis Part I: Fundamentals Nicolas Pugeault n.pugeault@surrey.ac.uk http://info.ee.surrey.ac.uk/Personal/N.Pugeault/

7. Lecture Plan • Introduction to sound synthesis • Perception of sound • Loudness • Pitch • Timbre • Sound synthesis – Fundamentals • Summary

8. Sound (cont’d) • Sound is a pattern of compression and depression of the air • Record it using microphones • Perceive it from our ears • Generate it by speaking or using speakers • Energy per m2 decreases with the square of distance...

9. Sound is a waveform • Sound is a waveform, • Can be reflected when hitting a non-transmissive surface • If the surface is flat, reflected in cohesive way • Otherwise depends on frequency and surface texture Sound proof studio wall, for absorbing high frequencies

10. Attributes of sound

11. The simplest sound: Pure tone • Sinusoidal wave (440Hz) Periode p=1/f0

12. Reminder: Fourier Transform • Idea: “All functions can be decomposed in a (possibly infinite) sum of sinusoidal functions of varying frequencies.” • Transforms a function from time domain to frequency domain. • Eg, right, for a square wave. First component First two components First three components First four components

13. Loudness • Often measured in decibels (dB) R=20*log10(A/A0) • A0 is a reference amplitude, often taken as the threshold of audibility. • Logarithmicperception of loudness. •  A change in 6dB means a doublingof amplitude! • Range of audibility: ~120dB (1 to 1million)

14. Perception of Loudness • Correlated with amplitude. • Here: • constant frequency (f0=440Hz) • Varying amplitude (A = 0.2, 0.5 or 1.0)

15. Loudness (cont’d) • However • Perception of Loudness is frequency dependent. • Sound X and Y have the same amplitude, which is louder, X or Y? • X (100 Hz, A=1) • Y (3,500Hz, A=1) • Considering only amplitudes, sound Y should be the same loudness as sound X. • However, Y is louderthan X. Why?

16. Loudness (cont’d) • Fletcher Munson (1933) • Subjects listen to pure tones • Various frequencies • amplitude inc. per 10dB • Robinson & Dadson (1956) • more accurate • Basis for standard ISO-226 • Perceived Loudness (Phons) • 1 Phon = 1dB SPL @ 1kHz • British Standard BS ISO 226 (2003) (source wikipedia)

17. Loudness (cont’d) • There is a difference between sensory loudness and perceptual loudness! (Emmet, 1992) • For the design of a synthesiser with large dynamical range, changing only amplitude is a poor choice since signal may clip. • Solution: use spectral variation: • Broad spectrum will likely result in a loud sound. • Narrow spectrum will be perceived as quiet.

18. Perception of Pitch • Frequency correlated with pitch • Here: 3 examples of pure tones. • What if sounds are more complex? • Range: 20Hz-20kHz • Best acuity: 200Hz-2kHz

19. Pitch: Fundamental & Harmonics • Real sounds are not pure – more complex! • The ear assumes that multiple frequency components form one sound. • Harmonically related -> fuse into single pitch at Fundamental Frequency (f0largest common divisor) • Each sinusoid is called a Harmonic partial of the sound (fk = N*f0)

20. Fundamental & Harmonics (2) Fundamental f0 First harmonic f1 = 2*f0 Second harmonic f2 = 3*f0 Third harmonic f3 = 4*f0 ... Seventh harmonic f7 = 7*f0

21. Fundamental & Harmonics (3) • The pitch is correlated with the Fundamental frequency. • Although in this example the fundamental is missing, the pitch is the same. The timbre is different.

22. Timbre • All those sounds have the same pitch (A4, 440Hz) • Flute A4 • Tuning fork A4 • Violin A4 • Singer A4 • They differ in timbre.

23. Defining Timbre • Definition (American Standard Association):“That attribute of sensation in terms of which a listener can judge that two sounds having the same loudness and pitch are dissimilar.”(ASA, 1960 ; Wikipedia, 2011) • Has a “wastebasket” quality (Dixon Ward, 1965): • What is neither loudness nor pitch... • Synonyms:Tone quality or colour, texture... • Affected by a sound’s envelope.

24. Timbre (cont’d) • What physical parameters relate to timbre? • Static spectrum (transient) • Envelope of spectrum (transient) • Dynamic spectrum (time-evolving) • Phase • This list is not exhaustive. • cf “wastebasket” quality!

25. Timbre: Static Spectrum (220Hz)

26. Timbre: Envelope of Spectrum

27. Timbre: Envelope (cont’d) Flute • Difference in envelope (same note, 440Hz fundamental) • Top: Flute • Bottom: Violin •  Envelope differs! • Conclusion:Envelope is instrument-specific. Violin

28. Timbre: Envelope (cont’d) • Arrows indicate formants. • This slide indicates two speech vowels (i and u) • Formants not only determine timbre but helps distinguishing vowels. • (used in speech recognition)

29. Timbre: Dynamic Spectrum A • Will those two sounds have the same timbre? • No, same average spectrum, but different timbre! • Difference: • Top: original sound • Bottom: time reversed. • Conclusion: Temporal variation of spectrum impacts timbre! B

30. Timbre: Dynamic Spectrum (cont’d)

31. Timbre: spectrogram Frequency (Hz) Time (s.)

32. Timbre: Dynamic Spectrum (cont’d) A) Normal • This slides shows the long term (average) spectrum for two sounds (top: original and bottom: time reversed) • Spectrum is identical; timbre is totally different  very misleading! • Conclusion:it is important to know how the spectrum evolves in time. • The timbre does not only depends on the harmonic structure but on the way spectrum varies in time. B) Time reversed

33. Time envelope (ADSR) • Time Envelope (ADSR) • Attackis the time from nil to peak. • Decayis the time from peak to the sustain level. • Sustainis the level during the main sequence of the sound’s duration, until key is released. • Releaseis the time to decay from sustain level to zero.

34. Time Envelope (example)

35. Time Envelope (example) • Example of the same sound with and without attack • Attack cut at 0.7s. • With (blue+green): • Without (green):

36. Timbre: Phase? • Sound A • Sound B Are A and B of different timbres?

37. Timbre: Phase? Sound A: Square wave, fundamental 500Hz, 9 harmonics. • Timbre depends (weakly) on phase relationship between harmonics. • BUT waveforms are totally different, magnitude spectra identical, and timbre are (almost) identical! • Conclusion:Human hearing is not sensitive to phase differences. Sound B: Square wave, fundamental 500Hz, 9 harmonics, every second harmonic phase shifted by 90 degrees.

38. Summary 1: Loudness Control • In order to control loudness in synthetic sounds: • Modify the spectral content: • more energy at high frequency  louder(see right). • Modify the amplitude • Higher amplitude  louder

39. Summary 2: Pitch Control Fundamental frequency • In order to control pitch in synthetics sounds: • Modify the fundamental frequency. • High fundamental frequency  high pitch.

40. Summary 3: Timbre Control • In order to control timbre in synthetic sounds, modify • Spectral content • Spectral envelope • Spectrum in time • Spectrum evolution during transient states

41. Plan • Introduction to sound synthesis • Perception of sound • Loudness • Pitch • Timbre • Sound synthesis – Fundamentals • Summary

42. Fundamental Definitions • Computer Instrument: An algorithm that realizes (performs) a musical event. • Unit Generator: A high-level “building block” in an instrument.

43. Important Terms Two types of synthesisers monophonic polyphonic You can only play onenote at a time. If you play several keys together, only one note will be generated  no chords! You can play several notes at the same time  can play chords!

44. Types of synthesis

45. Plan • Introduction to sound synthesis • Perception of sound • Loudness • Pitch • Timbre • Sound synthesis – Fundamentals • Summary

46. Additional Reading • C. Dodge, C., & Jerse, T. A. (1997). Computer Music: Synthesis, Composition, and Performance.Schrimer, UK.(see chapters 2 and 4)