Reading Visual Representations

Prof. Marc Davis, Prof. Peter Lyman, and danah boyd UC Berkeley SIMS Tuesday and Thursday 2:00 pm – 3:30 pm Spring 2005 http://www.sims.berkeley.edu/academics/courses/is146/s05/ Reading Visual Representations IS146: Foundations of New Media

Lecture Overview • Review of Last Time • Social Uses of Mobile Phones • Today • Reading Visual Representations • Preview of Next Time • Midterm Exam

Why Does Texting Work In Japan? • Over 10 years since popular mobile texting beginning with youth pager adoption • Dense and information-rich pedestrian urban ecologies • Limited private space • Highly regulated public transportation • High print and visual literacy rates • Vibrant popular techno-cultures surrounding portable electronics • Text the dominant modality of mobile communication Slide from Ito

Diary Study • What is a diary study? • Record every interaction with the mobile • Indicate meta commentary • Complements data record • When are diary studies useful? • Interactions over extended periods of time • Want to know what people are thinking during interaction before they forget

Technosocial Situations • Social setting as merging of technology, practice, place, and social relations • Goffman in the networked world • ….or ethnography of the technosocial • Keitai use as technosocial practice • Merging of remote/mediated (virtual) relations and physically co-present (real) relations • New practices and competencies for navigating different situations and stages • Keitai messaging technosocial situations • Mobile Text Chat • Augmented Flesh Meet • Ambient Virtual Co-Presence Slide from Ito

Emergent Technosocial Orders • Changing sense of co-presence • Pulsating motion between background and foreground awareness and engagement • Couple communication via mobile email particularly distinctive • Move from serendipitous to intentional contact • Individual selectivity in communication and contact • Networked individualism (Wellman) • Full-time intimate community Yoshii et al) • Tele-cocooning (Habuchi) • Portable practices • Layering of personal/intimate and place-based meanings • Urban space is personalized,no longer anonymous/alienated • We carry more of our identities and social connections around with us Slide from Ito

Cheskin Research: Methods and Goals • Methods • Quantitative primary research with teens, age 13-18, and young adults, age 19-24 • Interviews with industry experts • Secondary research • Motivations for wireless study • Youth influence larger consumer trends • Figure out new wireless product and service offerings • Goals • Understand the phenomena being observed • Advise customers about what actions to take based on that understanding • Sell these services and the value of their approach

Social connectivity and entertainment will be the primary defining characteristics of wireless devices in the youth market, and likely the consumer market at large Young people will build relationships via wireless devices Multitasking capabilities will flourish within the youth market Personal security and convenience will continue to be motivating factors for first time mobile phone consumers Personalization of design, function, and interface will be a common expectation Wireless entertainment and information applications will become favored "gap-fillers" Strategic convergence will define the most successful wireless devices Entertainment will drive the development of wireless cross-platform content Cheskin Research: The Wireless Future

Ito and Cheskin on Mobile Youth • How do their methods, motivations, and goals compare? • How do their findings compare? • What factors affect the similarities and differences?

Nick Reid on Ito and Okabe • How do “social spaces” intersect? What I mean is, if there is a situation where a group of people are together, and through telepresence, how does this outside party enter into a group? How is their presence felt by people who are there or are not there, would it really seem like the person is in the next room?

Nick Reid on Ito and Okabe • A question that would be fun to discuss is what counts as “contact”? Is contact a hug or is contact a SMS? Does a communication not being “physical” demean the communication? “When a situation is predictable there is no information present.” Another question would be, does contact actually have to transmit information?

Steven Lybeck on Ito and Okabe • Ito and Okabe show that new technologies are spawning the creation of virtual social spaces that are quite analogous to physical ones. Could these virtual spaces supersede or even replace interaction in physical spaces? Why or why not?

Steven Lybeck on Ito and Okabe • Are there any examples of virtual spaces constructed without the use of new media technologies?

Questions For Today • What does it mean that images are constructed? • How do we read images? • Why are most of us not visually literate? • How do we become visually literate? • What do the previous questions mean for how we use, understand, and design cameras and photographs?

Yukaghir Epistle

Airplane Instructions

Reading Images • Visual communication is always coded • “It seems transparent only because we know the code already, at least passively — but without knowing what it is we know, without having the means for talking about what it is we do when we read an image.” • Our culture is moving from textual to visual • “Until now, language, especially written language, was the most highly valued, the most frequently analyzed, the most prescriptively taught and the most meticulously policed code in our society.” • Visual “literacy” is not taught and needs to be • “If schools are to equip students adequately for the new semiotic order, if they are not to produce people unable to use the 'new writing' actively and effectively, then the old boundaries between 'writing' on the one hand, traditionally the form of literacy without which people cannot adequately function as citizens, and, on the other hand, the 'visual arts', a marginal subject for the specially gifted, and 'technical drawing', a technical subject with limited and specialized application, should be redrawn.”

Semiotic Landscape • Relationship between • The range of forms or modes of public communication available in a society • The uses and valuations of these forms or modes of public communication available in a society

Semiotics Review • What are the signifier, the signified, and the sign? • What are the similarities and differences between linguistic signs and visual signs? Signified “dog” “dog” Signifier dog

SaussureLinguistic Sign • Sign, Signified, Signifier • The linguistic sign is the unity of the signifier (a sound-image) and the signified (a concept) Concept Sound-Image

The Linguistic Sign “dog” dog

The Visual Sign “dog”

Arbitrariness of the Visual Sign • Theories of visual denotation • Iconic (i.e., onomatopoetic) • Images are an analogous reproduction of what they represent • Arbitrary • Images construct an arbitrary relationship between signifier and signified • Motivated • The relationship between the signifier and signified is motivated, but by what? • A “natural” analogy between image and the world? • By the conventions of visual language?

Social Semiotic Theory: Metafunctions • Ideational metafunction • To represent, in a referential or pseudo-referential sense, aspects of the experiential world outside its particular system of signs • Interpersonal metafunction • To project the social relations between the producer of a sign or complex sign, and the receiver/reproducer of that sign • The textual metafunction • To form texts, complexes of signs which cohere both internally and with the context in and for which they were produced

“Portrait”

Natalie Torin on Visual Representations • Kress and van Leeuwen wrote that "the dominant visual language is now controlled by the global cultural/technological empires of the mass media..... which exerts a 'normalizing' rather than explicitly 'normative' influence on visual communication across the world." Do you agree with this statement? How might this notion have changed in the last decade, considering that the article was written in 1995?

Natalie Torin on Visual Representations • Roland Barthes describes the special status of the photographic image as a "message without a code." Why do photographs have this special status yet pictorial images need a qualification?

Natalie Torin on Visual Representations • Any good semiotic system has to be able to project a particular social relation between the producer, the viewer, and the object represented. Can you think of an example of a very clear semiotic system? A very unclear semiotic system?

Natalie Torin on Visual Representations • Kress and van Leeuwen wrote that "some things can be said only visually, and others only verbally." Stemming from the notions of Ong's orality and literacy, is it possible to represent and understand something equally in either fashion? What gets lost in the process of converting from visual to verbal or vice versa?

Review Session • 6:30 pm – 8:00 pm • 202 South Hall • NO sections next week!

Reading Visual Representations