Sound History

1. Sound History

2. A History of Game Sound • A simple sound generator • PC sound (1980) • Amiga/Sega/Nintendo Chip Music • Midi • Wave • MP3

3. A History of Game Sound • Why are we looking at history? • Because it repeats itself. • We thought we’d seen the last of PC sound generators when PCs got sound cards. Then we got mobile phones with PC-style ring tones. Now mobile phones have gone to mp3... but what comes next?

4. A Simple Sound Generator Parallel Data From PC Output Register LSB MSB Gnd

5. A Simple Sound Generator • The CPU changes the data on the output port to alternate the output LSB (least significant bit) between one and zero. • This puts a square wave onto the loudspeaker. • This technique is so CPU-intensive that it is seldom used (even though the hardware is minimal)

6. PC Sound • Can play a square wave (sounds a bit like a violin) • PC programmed frequency (pitch) and whether the sound was on or off • Could only play one note at a time • Referred to as PC beeper or PC squeaker • Introduced in original IBM PC in 1980 • Still there on every motherboard.

7. PC Sound Parallel Data From PC System Clock (4-15 MHz) Intel 8253 Programmable Divider Chip Square wave output Gnd

8. PC Sound • One line of music is not as restrictive as it looks: • (star1.mid)

9. PC Sound • We can add an accompaniment: • (star2.mid)

10. Chip Music • In an attempt to brighten up the background music for games, a music chip was added to early games computers, such as Nintendo, Amiga, Sega Megadrive etc. • Typical chips were: General instruments AY-3-8910 or 8912,Yamaha YM2612 or YM3812 • They were programmed by the CPU.

11. Chip Music • Most chip sounds are synthesised by simply dividing a clock square wave to get a square wave of desired frequency • LFOs (low frequency oscillators) are used to control or influence a sound parameter such as pitch or filters in a repeating cycle to give tremolo or vibrato.

12. Chip Music • There were three different sound sources on the same chip, each of which could be programmed to give a note (for melody or accompaniment) or filtered white noise (for percussion) • Generally chip tunes consist of basic waveforms, such as sine waves, square waves and sawtooth or triangle waves, and basic percussion, often generated from white noise going through an envelope–controlled synthesizer.

13. Chip Music • The white noise for percussion sounds is generated on-chip using a random number generator, • The random number generator is programmed into the hardware. • A popular design for a random number generator is an m-sequence generator. (see lecture on random numbers)

14. Chip Music • The classic chiptune 8-bit sound can be recognised from: • Its synthesised square or pulse wave instruments, • Simple white noise percussion • Heavy use of ultra-fast arpeggios to emulate chords of three or four notes on a single channel (due to hardware limitations, several notes must be placed on the same channel).

15. Chip Music • There is now a nostalgic music genre called “Chiptune”. • Composers programme music chips of the sort found in a Gameboy. • (Sabrepulse - Gare de Europe) • Vibrato using LFOs • Filtered noise as percussion • Intricate music hides that only 3 generators

16. MIDI • Musical Instrument Digital Interface • (Also used to control theatre lighting) • It’s a little like the music rolls you can get for a steam-organ, or the cylinder of a musical box. • It gives instructions “switch this note on at this volume” and “switch this note off”.

17. MIDI • One minute of: • WAV: ~10.2 Megabytes • mp3: ~1.1 Megabytes • Midi: ~ 17 kilobytes • In other words, a midi file is less than 1/500 of the size of the equivalent WAV(an mp3 is about 1/10 of the size of a WAV.)

18. MIDI • There are 16 “Channels”. Each channel can be programmed to correspond to a different musical instrument. • There are 127 instruments to choose from • How these instruments are rendered (played) depends entirely on the computer that is playing the file. It might be an accurate rendition. It might be awful.

19. MIDI • To change the instrument of a channel takes 2 bytes: • eg: “#C0” means “change channel 1” • eg: “#0D” means “Xylophone” • So “change channel 1 to xylophone is “#C00D” = 1100 0000 0000 1101

20. MIDI • There is a byte to turn on a note • There are 127 notes available, over 9 octaves • Each note is given a volume of 0 - 127 (0 - #7F) • Instead of volume, midi people talk about velocity - how fast the key on the keyboard was pressed. Fast means loud.

21. MIDI • “Switch on a note on Channel 5” = “#94” • Let it be middle C = “#3C” ( • Let it be of medium volume = “#40” • So you send “#943C40”

22. MIDI • ...so we’ve got a “C” running on channel 5. It will run until we stop it. This is how we stop it: • “Stop note #3C on channel 5”: “#843C” • (there is also a command for “stop everything”)

23. MIDI • There is also a midi wiring standard. • Midi connectors are 5-pin DIN connectors. • It is quite common to run a midi cable between a piano accordion and a midi synthesiser box. • The accordion has sensors and a rudimentary serial interface. The box has all the fancy electronics.

24. MIDI • In olden days, midi synthesisers converted music by imitating instruments. They would produce a square or triangular wave and frequency modulate it with another wave. (Like they did with chip music.) • The result - FM synthesis - sounds LIKE the instrument it was trying to imitate, but it wouldn’t fool anybody.

25. MIDI • Nowadays, MIDI synthesisers use wave synthesis. • In wave synthesis, someone would record a real piano - perhaps even the actual piano at Abbey Road Studios that was used to make so many tracks • This sample would be looped to give an authentic instrument sound.

26. MIDI • The point is that the composition is divorced from the rendition • The midi file is like sheet music - it says what to play, but it gives no guarantee of quality • There is a rich variety of midi playback software and hardware. Quality of playback ranges from breathtakingly good to spectacularly awful.

27. MIDI • Q: Why am I saying so much about MIDI? • A: Because the Java platform you are working with will cheerfully handle MIDI tunes. • You don’t need to write your own (unless you want to - I do.) You can find them all over the internet. • Hint: look for rags.

28. WAV Sound Files • We hear things because sound pressure waves move our eardrums backwards and forwards • Microphones can convert these pressure waves into voltages • The voltages fluctuate at frequencies between 0 ... 16000 Hz. (Hz used to be called cycles per second, which made far more sense.)

29. WAV Sound Files • We convert the voltage from the microphone into digital form by repeatedly measuring the voltage height.

30. WAV Sound Files • A WAV file is simply a list of the heights of the waves. • Nyquist determined that if you want to accurately record audio to a bandwidth of 16kHz (hi-fi), then you need to take measurements at 32kHz. • Some applications only need a bandwidth of 8kHz, needing a sampling rate of 16kHz.

31. WAV Sound Files • Why am I telling you all this? • Well the Java platform for mobile games will play back WAV files. • Wav files are good for sound effects. (Or to put it another way, MIDI is useless for sound effects.) • However, one minute of Wav file needs about 10 Megabytes of memory, so you can’t store much in a mobile phone.

32. MP3 • MPEG-1 Audio Layer 3, more commonly referred to as MP3 • Digital audio encoding format using a form of lossy data compression. • Human ears take in sound vibrations as a time-domain waveform...

33. MP3 • ... and the Cochlea in the ear turns it into a frequency representation

34. MP3 • (voice signal and its spectrum)

35. MP3 • Nerves from the cochlea carry the signal frequency information in massively parallel form to the brain. • The brain hears frequencies, not waveforms. The frequencies change slowly compared to the actual waveform. • Storing a sound as frequency information reduces the data needed for storage.

36. MP3 • There is also the masking effect: quiet sounds close in frequency to a loud sound are simply not heard... so you don’t have to store them.

37. MP3 • The upshot is that by storing sound as frequency information, and only storing what will be heard, the data is cut to 1/10.

38. Granular Synthesis • Usually background sound is pre-recorded and looped. • This works for music as long as you provide new music every level - otherwise it gets repetitive and irritating. • Soundscapes and sound effects are trickier - after a while you know what comes next. Repeated effects become irritating.

39. Granular Synthesis Solution: • Short samples of effects and music • Assembled randomly or according to an algorithm • Changed in pitch or duration using signal processing techniques (usually something to do with a Fourier transform) • Granular Synthesis

40. Granular Synthesis Dialogue • You’ve got an effect of a guy shouting: “Hey, over here” • After a while you and everybody else notices that it is shouted in exactly the same way every time • You can randomly change the speed and pitch of delivery

41. Granular Synthesis Crowd Scenes • You could make up a crowd from a number of individuals • A less computationally expensive way is to take samples from a crowd and play them in a random order • You can also randomise the stereo panning to distribute them around the environment

42. Granular Synthesis Ambiences: eg Jungle • River, wind and insect background is built from a handful of similar samples, mixed in random order • Birds and monkeys are distinctive - they are played at random and with randomised parameters

43. Granular Synthesis Ambiences: eg Jungle • Soundscape can be made responsive to game events • After you fire your rifle, everything goes quieter for a while, then builds back up. • Prior to the Unexpected Challenge, the sound can quieten down for a while to build the suspense

44. Granular Synthesis Music • Matching sections of music are reordered by an algorithm or at random. • In “Diner Dash” there are four major themes. Each theme consists of a small number of smaller pieces (2, 4 or 8 bars long), which go together no matter how they are arranged. What piece to play is decided by the game-state!! • (dinerdash.mpg)