MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS One

1. MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS One Multimodal Methodologies Part I and II Jeff Bezemer, Carey Jewitt and Gemma Moss http://mode.ioe.ac.uk

2. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS The aim of today is to provide: • An understanding of the origins and scope of multimodality • Explications of multimodal frameworks and key concepts • Knowledge of the different ways in which multimodality has been taken up by a range of perspectives • An outline of the procedures of multimodal analysis

3. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Focusing questions: • Why take a multimodal approach? • What concepts and methods can be used to analyse interaction in digital environments? • How can we get at how the use of different digital technologies shapes understandings and experiences? • What challenges does multimodality raise for researching digital data and environments?

4. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Program: Part I • 9.15 Introduction • 9.30 Origins, Scope and Key concepts (Carey Jewitt) • 11.15 Multimodal perspectives: video data analysis (Jeff Bezemer) 12.45 lunch Part II • 16.00 Presentation and data workshop: Analyzing multimodal digital texts(Gemma Moss)

5. MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Multimodality: origins, scope and key concepts http://mode.ioe.ac.uk

8. Concepts: the level of mode MODE MEANING FUNCTIONS SEMIOTIC RESOURCE SEMIOTIC (MEANING) POTENTIAL MATERIALITY AFFORDANCE PROVENANCE MODAL LOGICS FUNCTIONAL SPECIALISM MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

9. Concepts: Level of modal-interaction MULTIMODAL ENSEMBEL INTER-MODAL RELATIONS COHERENCE DISSONANCE MODAL DENSITY MODAL INTENSITY EQUAL-UNEQUAL DOMINANT – SUBORINATE ELABORATION EXPANSION REPETITION-EMPHASIS SALIENCE December 2011, Norway

10. Concepts: level of process of sign making SIGN MAKER INTEREST MULTIMODAL ORCHESTRATION MULTIMODAL DESIGN TRANSFORMATION TRANSDUCTION TRANSMODAL MOMENT TEXTUAL CYCLE MODAL ORIENTATION CONTEXT December 2011, Norway

11. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Four main perspectives on multimodality: • Social Semiotic Multimodal Analysis • A Systemic Functional Approach (Multimodal Discourse Analysis) • Multimodal Interactional Analysis • Multimodal conversation analysis

12. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Similarities • fine grained analysis: the need to understand the details of texts and interactions • focus on meaning making as social and semiotic • the demand to broaden conceptions of communication beyond language • shared interests, and shared histories • externally material focus

13. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Differences • Historical origins, influences and directions • Extent of emphasis on context • Degree of emphasis on modes and modal systems • Attention to the agentive work • The scale of attention

14. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Conversation analysis & multimodality: analytical focus • Organizational geography • Temporality of events • Recognizing events: patterns and customs • Detecting incongruities and breakdowns • Sequences of interaction, before, after and immediately following one another

15. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

16. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Systemic Functional Linguistics (Anders Bjorkvall, 2009 in HMA)

17. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

18. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Social Semiotic Multimodal Analysis

19. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Analyzing Multimodal Interaction Norris (2004)

20. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Norris (2009) Modal Denisity in HMA

21. MODEmultimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Other approaches • Sensory anthropology (e.g. David Howes) • Socio-cultural approaches and multimodality (e.g. Barbara Rogoff, Roger Saljo) • Eye-tracking (Jana Holsvana) • New literacy Studies (Kate Phal, Jennifer Rowsell) • Historical analysis with a multimodal focus (e.g. Grosvenor)

22. mode@ioe.ac.ukhttp://mode.ioe.ac.uk