1 / 50

The cAitlIn system Musical Program Auralisation & External Representations of Pascal

The cAitlIn system Musical Program Auralisation & External Representations of Pascal. http://computing.unn.ac.uk/staff/cgpv1/caitlin Dr Paul Vickers School of Informatics, Northumbria University Professor J.L. Alty Dean of Faculty of Science, Loughborough University. Introduction.

raiden
Download Presentation

The cAitlIn system Musical Program Auralisation & External Representations of Pascal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The cAitlIn systemMusical Program Auralisation& External Representations of Pascal http://computing.unn.ac.uk/staff/cgpv1/caitlin Dr Paul VickersSchool of Informatics, Northumbria University Professor J.L. AltyDean of Faculty of Science, Loughborough University Musical Program Auralisation

  2. Introduction • Major features • Background to the project • Sound in HCI • A case for music • The cAitlIn system • Demonstration • Future work Musical Program Auralisation

  3. Major features • With cAitlIn we have: • A musical equivalent for various syntactic components of a programming language = musical grammar • Extended the earcon technique (q.v.) • An audio program structure diagram (execution path) Musical Program Auralisation

  4. Major features • A MPAS usable by non-musicians • Shown that music can be used to communicate information about program state and structure • In turn, this info can assist bug location • Empirical evaluation • Organising principles • Opened new line of enquiry & research Auditory External Representations Musical Program Auralisation

  5. Background • Interests in music, computing, & programming • Hunch that music could be used to assist in the programming process • Develop and test a musical framework for communication of program information Musical Program Auralisation

  6. Sound in HCI • Increasing research focus since mid ‘80s. • A belief that non-speech audio useful as interaction medium • A nascent field called auditory display spawns techniques like: • Sonification • Audification • Auralisation Musical Program Auralisation

  7. Examples • Auditory icons • Earcons • Data visualisation (sonification & audification) • Assistive technologies Musical Program Auralisation

  8. Program auralisation • Mapping of program data and events to sound • Main examples • LogoMedia • Sonnet • ADSL • LISTEN • Largely untested Musical Program Auralisation

  9. Why music? • Gives multiple audio streams in parallel • Structured and organised according to specific rules • Time-ordered. Visual techniques are spatial. • Can music’s grammars be used to carry real information? Musical Program Auralisation

  10. cAitlIn system • Computer Audio Interface to Locate Incorrect Nonsense • Uses musical motifs to render Pascal language constructs at run time • Motif design is hierarchic as is organisation of constructs Musical Program Auralisation

  11. Hierarchic motif design Musical Program Auralisation

  12. Musical grammars Musical Program Auralisation

  13. Musical grammars • We map existing grammar (e.g. Pascal syntax, English, etc.) onto a musical grammar • Transliteration or translation? • We may get emergent properties leading to a new musical grammar Musical Program Auralisation

  14. cAitlIn musical grammar • Uses motif classes and variations to render Pascal constructs • Allows • abstraction • nesting (tonality & harmony preserved across motifs) • Uses grammatical rules (e.g. a tonal music system) Musical Program Auralisation

  15. cAitlIn musical grammar cont • Renders information not directly communicated by source code or other visualisation techniques •  a new grammar, conveys extra meaning Musical Program Auralisation

  16. Advantages • Over earcons • do not show state • Over Auditory Icons • symbolic mappings = poor system aural ecology • Syntactic design  expect retention • Displays both flow and state Musical Program Auralisation

  17. Design of Tunes • Points of Interest • Construct Entry • Evaluation of condition • Execution of body • Exit from construct • Boolean True=major, False=minor • Drones Musical Program Auralisation

  18. Selection Tunes • Metaphorically asking & answering a question. • Tune rises on entry and descends on exit Musical Program Auralisation

  19. Iteration Tunes • Major/Minor chord to show when condition evaluated • Heard at start of WHILE loops and at end of REPEAT loops • FOR loops increase/decrease the pitch each time round the loop Musical Program Auralisation

  20. Variations • All selections use same basic tune, modified for individual constructs • All iterations use another basic tune, again modified Musical Program Auralisation

  21. Auditory Parentheses • As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs Musical Program Auralisation

  22. Other Statements • All other statements are currently represented by a drum beat, one per statement. Musical Program Auralisation

  23. Demonstration Musical Program Auralisation

  24. Examples • Generic selection motif • Generic iteration motif Musical Program Auralisation

  25. IF (true) a := 10 ; IF a = 10 THEN Writeln (‘A is 10’) ; Musical Program Auralisation

  26. IF (false) a := 0 ; IF a = 10 THEN Writeln (‘A is 10’) ; Musical Program Auralisation

  27. IF…ELSE (true) a := 10 ; IF a = 10 THEN Writeln (‘A is 10’) ELSE Writeln (‘A is not 10’) ; Musical Program Auralisation

  28. IF…ELSE (false) a := 1 ; IF a = 10 THEN Writeln (‘A is 10’) ELSE Writeln (‘A is not 10’) ; Musical Program Auralisation

  29. CASE match a := 3 ; CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; END ; Musical Program Auralisation

  30. CASE no match a := 100 ; CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; END ; Musical Program Auralisation

  31. CASE…ELSE match a := 3 ; CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ; END ; Musical Program Auralisation

  32. CASE…ELSE no match a := 100 ; CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ; END ; Musical Program Auralisation

  33. WHILE 2 iters cntr := 1; WHILE cntr <= 2 DO cntr := cntr + 1 ; Musical Program Auralisation

  34. REPEAT 3 iterations cntr := 1 ; REPEAT cntr := cntr + 1 ; UNTIL cntr > 3 ; Musical Program Auralisation

  35. FOR…TO 6 iters FOR cntr := 1 TO 6 DO Writeln (cntr) ; Musical Program Auralisation

  36. FOR…DOWNTO 6 iters FOR cntr := 6 DOWNTO 1 DO Writeln (cntr) ; Musical Program Auralisation

  37. Sequence & Nesting • Clue to nesting is in the drone • If you hear a continuous note then you are inside a construct. • Each construct has a drone. • For nested constructs you hear multiple drones simultaneously • Drone pitch increases as nesting deepens Musical Program Auralisation

  38. Example program Program Exemplar VAR counter : Integer ; BEGIN counter := 1 ; WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ; END. Musical Program Auralisation

  39. Score Musical Program Auralisation

  40. Score Musical Program Auralisation

  41. Structure Diagram Musical Program Auralisation

  42. Experiment 1 • Listening test to assess comprehensibility of motifs • 22 subjects, 60 auralisations • Recognition rate better than by chance and with very little training • Hierarchic organisation allowed class level identification & abstraction Musical Program Auralisation

  43. Experiment 2 • Debugging exercise • 22 subjects, 8 programs, 4 auralised, 4 normal • Results suggest technique is useful, especially for complex programs and where program output gives few clues • Identified avenues for further research Musical Program Auralisation

  44. Limitations • No comparison with visual task yet • Cognitive latency of motifs? • No longitudinal results Musical Program Auralisation

  45. Future work • Follow-up studies • Extend the grammar to include assignment statements and procedure calls • How far? Bugs not manifested in program flow are v. hard to detect and generally require variable inspection to locate, which is impractical for auralisation techniques Musical Program Auralisation

  46. Future work • Port cAitlIn to MS-Windows and conduct a longitudinal study • Assistive technologies. How can MPA aid blind programmers? Musical Program Auralisation

  47. Future work • Preparing EPSRC bid to investigate Auditory External Representations • (c.f. EPSRC CRUSADE project at COGS in Sussex - du Boulay et al) Musical Program Auralisation

  48. Publications–Journals & Reports • Vickers, P. and Alty, J. L. (2002). When Bugs Sing, Interacting with Computers in press. • Vickers, P. and Alty, J. L. (2002). Musical Program Auralisation: A Structured Approach to Motif Design, Interacting with Computers in press. • Vickers, P. and Alty, J. L. (2002). Using Music to Communicate Computing Information, Interacting with Computers in press. • Vickers, P. (2001). Using the World-Wide Web as a Platform for an Interactive Experiment, Liverpool: Liverpool John Moores University, Technical Report CMS-21, ISBN 1-902560-050. Musical Program Auralisation

  49. Publications–Conferences • Vickers, P. and Alty, J. L. (2000). Musical Program Auralisation: Empirical Studies, in Proc. ICAD 2000 Sixth International Conference on Auditory Display (Atlanta, GA, 2-5 April, 2000), International Community for Auditory Display, pp. 157-166 ISBN 0-9670904-1-5 • Vickers, P. and Alty, J. L. (1998). Towards some Organising Principles for Musical Program Auralisation, in Proc. ICAD '98 Fifth International Conference on Auditory Display (Glasgow, Nov 1-4, 1998), British Computer Society ISBN 1 902505 05 0 • Alty, J. L. and Vickers, P. (1997). The CAITLIN Auralization System: Hierarchical Leitmotif Design as a Clue to Program Comprehension, in Proc. ICAD '97 Fourth International Conference on Auditory Display (Palo Alto, November 3-5, 1997), Xerox PARC, Palo Alto, CA 94304, pp. 89-96 Musical Program Auralisation

  50. Publications–Conferences cont. • Alty, J. L., Vickers, P. and Rigas, D. (1997). Using Music as a Communication Medium, in Proc. Refereed Demonstrations, CHI97 Conference on Human Factors in Computing Systems (Atlanta, GA, March 22-27, 1997), ACM Press, pp. 30-31 ISBN 0-8979-926-2 • Vickers, P. and Alty, J. L. (1996). CAITLIN: A Musical Program Auralisation Tool to Assist Novice Programmers with Debugging, in Proc. ICAD '96 Third International Conference on Auditory Display (Palo Alto, Nov 4-6, 1996), Xerox PARC, Palo Alto, CA 94304, pp. 17-24 Musical Program Auralisation

More Related