CS6022 Principles of Interactive Media - May 4th

1. Virtual Marimba Creating a novel musical instrument that produces visual representations of sounds. An Interactive Media Thesis by Chris Hackett & David Delaney CS6022 Principles of Interactive Media - May 4th

2. profiles • Business Studies Graduate • Multi-Instrumental Musician • Collaborative Musician (Palestinian/South African,etc) • Sound Effects/Soundtracking • Focus: Interactive Media in Tradtional Musical Instruments. Sound Synthesis & Manipulation David Delaney CS6022 Principles of Interactive Media - May 4th

3. profiles Chris Hackett • Computer Systems Graduate • Specialism: Web/Graphic Design • Works with: Dreamweaver, Photoshop, Flash, Sketch-Up • Interests: Graphics, Animation,Music, Audio-visual Art, Film, Photography • Focus: Graphics, Visual Representations of Sound. CS6022 Principles of Interactive Media - May 4th

4. inspiration • Interest in Music • Similar Thesis Proposals • Existing artefacts e.g. Marimba Lumina • Ailbhe’s ‘Rhythm Manifesto’ • Composer/Designers e.g. Todd Winkler • Audio - Visual Art Culture CS6022 Principles of Interactive Media - May 4th

5. design process • User-Centred Design • Research • Requirements Gathering • Design • Prototyping • Evaluation CS6022 Principles of Interactive Media - May 4th

6. visual music • ‘A means of expressing music in visual form requiring the active involvement of an artist, designer, or director to interpret the music and find the means to express it visually.’ (Friedlander, ‘98) • ‘A visualization of music which is the translation of a specific musical composition(or sound) into a visual language, with the original syntax being emulated in the new visual rendition.’ (Ox, J and Keefer, C, ‘06) • Origins: Pythagoras, Aristotle, Sir Isaac Newton Correspondences of the colour spectrum and sound waves, music and colour, sound and light. • 18th Century - Castel’s Clavecin Oculaire • Late 19th/Early 20th - Experimentation, abstract film • 20th Century - Art e.g. Scott Drave’s Electric Sheep • Present - Artists, composers, designer, virtual djs CS6022 Principles of Interactive Media - May 4th

7. background reading g • VJ: Audio-visual Art and VJ Culture (Faulkner, M) • On Curating Recent Digital Abstract Visual Music (Ox, J & Keefer C) • Amplifying Reflective Thinking in Musical Performance (Johnson, A et al) • Visual Music-Searching for an Aesthetic (DeWitt, T) • Exploration of the correspondence between visual and acoustic parameter spaces (Gerhard et al) • Issues for Designing a flexible expressive audiovisual system for real-time performance and composition (Franco et al) • Rhythm Manifesto – Ailbhe Keogan CS6022 Principles of Interactive Media - May 4th

8. benchmarks • malletKat • Marimba Lumina • ‘Music Plays Images X Images Play Music’ MPI X IPM Ryuichi Sakamoto • ‘Falling Up’ - Todd Winkler • ‘Rhythm Manifesto’ - Ailbhe Keogan CS6022 Principles of Interactive Media - May 4th

9. websites • NIME www.nime.org • The MEGA Project www.megaproject.org • InfoMus Lab infomus.dist.unige.it • SARC www.sarc.qub.ac.uk • The s2S2 www.s2s2.org • ConGas www.cost287.org • www.centerforvisualmusic.org • www.howstuffworks.com • www.acm.org CS6022 Principles of Interactive Media - May 4th

10. empirical work undertaken • Benchmark Research e.g. malletKat • Informal discussions with classmates • Interview with Ailbhe Keogan, 27.02 • Interview with David Merrill, MIT, 16.03 • Early sketches, prototyping, scenarios CS6022 Principles of Interactive Media - May 4th

11. further empirical work • Interviews • Cormac Byrne, Marimba player • Evelyn Glennie, Killarney 12.07 • Micheal O Suilleabhain, Musician • Music Academics, e.g. Jurgen Simpson • Further sketches, prototyping, scenarios

12. artefact

13. artefact

14. layout CS6022 Principles of Interactive Media - May 4th

15. layout CS6022 Principles of Interactive Media - May 4th

16. projection screens CS6022 Principles of Interactive Media - May 4th

17. visuals • Mapping Techniques • Colour - 7 colours to 7 units of the scale • Vibrato – sine wave • Interrupted Cadence – expectation • Relative Minor – sinister, dark, gothic • Timbre of Sound • Mikael’s Manifesto • PD & GEM • 3Ds Max, Photoshop, Flash CS6022 Principles of Interactive Media - May 4th

18. architecture Sound (Speakers) Images (Projector) Electronic Marimba App Max/MSP, Pure Data, VVVV, Director OS (Windows) Device Drivers BIOS Camera Sound Parameters can be manipulated Device Sensors PC Hardware (USB, Firewire or Wireless) CS6022 Principles of Interactive Media - May 4th

19. hardware required • Arduino board – prototyping • Piezo sensors • Pico device • Projector – possibly 3 • Webcamera • Computer (PC/Mac) • Under discussion – whether to build device from scratch or use existing commercial product CS6022 Principles of Interactive Media - May 4th

20. software required • Pure Data and GEM - probably • MAX/MSP - not so likely • VVVV - not so likely • Director & Flash (Lo-Fi Prototyping) • Photoshop (Graphics) • 3D studio Max (Modelling, Graphics) • Google Sketch-Up (sketches) • Logic Pro (Synthetic Sounds for lo-fi’s) CS6022 Principles of Interactive Media - May 4th

21. extra scope for instrument • Multi Instrumental • Display Dynamic Visuals • Pitch bend, vibrato • Velocity Sensitivity • Pressure Sensitivity (Aftertouch) • Recording and Playback of loops. CS6022 Principles of Interactive Media - May 4th

22. schedule CS6022 Principles of Interactive Media - May 4th

23. documenting the process http://0030511.blogspot.com CS6022 Principles of Interactive Media - May 4th

24. concerns • Finalising concept • Building device Vs Existing product • Real-time Vs Pre-Rendered • Latency Problems • Pressure/Velocity Sensitivity • Number of Projectors Needed • Software Skills • Cost • Time CS6022 Principles of Interactive Media - May 4th

25. issues for discussion • Suitable & novel interface? • Existing products • Ideas for graphics • Thesis Title • Anyone know PD, GEM? • Have a good weekend! CS6022 Principles of Interactive Media - May 4th

26. Q&A CS6022 Principles of Interactive Media - May 4th