1 / 46

Music and shape: Exploring cross-modal representation

Music and shape: Exploring cross-modal representation. Examples, a model and analyses Professor Adam Ockelford University of Roehampton 11th March 2010. Music and shape. How can the two relate? One set of dynamic relationships can be created through movement:

barr
Download Presentation

Music and shape: Exploring cross-modal representation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Music and shape:Exploring cross-modal representation Examples, a model and analyses Professor Adam Ockelford University of Roehampton 11th March 2010

  2. Music and shape How can the two relate? One set of dynamic relationships can be created through movement: • more or less freely associated with sound (eg, dance, gestures in performance) • creating, causing or controlling sound (eg, playing instruments, conducting)

  3. Music and shape Another set of relationships can arise through the visual representation of sound – the focus of this presentation • Seven examples • Attempt to construct common theoretical framework • Indicative analyses • Discussion

  4. Children’s ‘picture scores’ 7 year old’s representation of a rhythm (after Bamberger, 1982)

  5. Children’s ‘picture scores’ 7 year old’s representation of a rhythm (after Bamberger, 1982)

  6. Blind children’s drawings on ‘German Film’ After Welch (1981)

  7. Blind children’s drawings on ‘German Film’

  8. Conventional Westernmusic notation Any features in common with children’s informal notations?

  9. Braille music Key differences with print?

  10. Guitar chord tablature What processes are at work here? Similarities and differences with previous examples?

  11. Graphic notation Excerpt from Stockhausen, Electronic Study No. 2 (1954)

  12. Score produced through synaesthetic response to music Jamie Roberts’s score of Jean Michel Jarre’s Oxygene (Track 4)

  13. How does it all work? A common underlying principle or principles? Or fundamentally different processes at work?

  14. Perceptual domains • ‘Domains’ = way of modelling the fact that one sensation can have different ‘states’ • Touch: eg, temperature and pressure. • Vision: eg, hue, saturation and brightness.

  15. Perceptual domains In music: • Pitch: (chroma, octave), harmony, tonality • Perceived time: onset, duration • Timbre • Loudness • Perceived location of sound source

  16. Different categories of domain Can be conceived as being single dimensional, bidimensional or multidimensional and as being linear or cyclical or both, or complex

  17. Examples … Pitch Duration

  18. ‘Values’ • Each state in which a domain is perceived to exist can be conceived of as a ‘value’. • May be simple or complex (like the domains to which they pertain) • Example:

  19. Relationships within domains • Musical structure is not to do with individual values, but the relationships between them within each domain - particularly pitch and perceived time • We can assume that the relationships between values normally pass listeners by as series of qualitative experiences • However, they may be conceptualised (culture specifically?) as differences or ratios, or they may reflect the complexity of the values to which they pertain

  20. Examples of intra-domain relationships ‘Primary’ relationships

  21. Examples of intra-domain relationships ‘Secondary’ relationships

  22. Relationships between domains • Two main types • ‘Regular’ (isomorphic in the domain of ‘shape’) • ‘Irregular’ (through association)

  23. Regular inter-domain relationships Guiding principle: to map values systematically between domains, a relationship must involve a function that is not specific to the perceptual domains concerned.

  24. Regular inter-domain relationships For example, differences are domain-specific, so single values of difference cannot be mapped systematically between domains.

  25. Regular inter-domain relationships Whereas, ratios are abstract, are not bound by the context in which they occur. Hence they permit regular mapping between domains.

  26. Regular inter-domain relationships Ratios may occur between differences (as secondary relationships). Hence the regular inter-domain relationships would be tertiary.

  27. ‘Irregular’ inter-domain relationships Guiding principle: formed through association … either through repetition or indirect connection

  28. Irregular inter-domain relationships Different types: (a) arbitrary (repetition) (b) indirect (indirect connection) (c) synaesthetic (indirect connection)

  29. Arbitrary inter-domain relationships Any relationship between a sound and its visual representation (shape) is possible. For example: accent staccato fermata > •

  30. Arbitrary inter-domain relationships Example (‘staccato’) – relationship created through repetition (or verbal explanation = proxy repetition):

  31. Indirect inter-domain relationships Example (‘note cluster’) – relationship created through action / object as common visual and auditory source Can be reinforced through repetition

  32. Synaesthetic inter-domain relationships Example – green corresponds toEbmajor (and vice versa) Again, can be reinforced (for others) through repetition

  33. From theory to analysis Children’s ‘picture score’ Simplest interpretation: arbitrary relationships – given shape (circle) maps onto note

  34. Children’s ‘picture score’ More advanced interpretation: regular mapping of distance and duration; sequence and temporal order

  35. Score on German Film of pitch glide Coordinated regular mapping of vertical and horizontal distances with pitch and perceived time (implied tertiary relationships)

  36. Print score Arbitrary and semi-regular mappings combine (the latter reinforcing the former)

  37. Braille score Only arbitrary mappings survive transcription into Braille (which makes using this medium more challenging)

  38. Guitar tablature Combines both indirect and regular representations of pitch

  39. Stockhausen score: comprises regular representations of pitch, perceived time and loudness

  40. Jamie’s score: comprises regular and indirect representations of pitch, perceived time and timbre

  41. Conclusions • Evidence from children’s untutored representations of music suggests that sophisticated inter-domain mapping between sound and shape occurs early and intuitively • This view is reinforced by blind children’s representations of changing pitch – which are also sophisticated, and exist in the absence of any visual model to guide them

  42. Conclusions • The precise nature of inter-domain mapping may vary from individual to individual, and according to cultural convention. • However, it appears to be relatively easy to grasp regularmappings (consider, for example, widespread use of horizontal dimension for time and vertical dimension for pitch) or arbitrarysymbols (as in standard notations)

  43. Conclusions • Some inter-domain mappings may be hard-wired through synaesthesia. • These seem to vary from individual to individual. • However, such representations can be learnt and appreciated by others

  44. Conclusions • What next? • More exhaustive search of different forms of symbolic visual representation of sound to check broad applicability of the model • Use the model to generate new forms of sound / shape installations • More research into how mappings aries and are learnt, and into sonic/visual synaesthesia

  45. Adam Ockelford a.ockelford@roehampton.ac.uk 07818-456 472

More Related