Transforming Journalism: How AR & VR are impacting the news

1. Transforming Journalism:How AR& VR are impacting the news John V. Pavlik, Ph.D. Professor, Journalism and Media Studies, SC&I Rutgers, the State University of New Jersey jpavlik@comminfo.rutgers.edu

2. Outline of My Remarks • Defining AR, VR • Qualities of AR, VR • Brief history of AR, VR • Use of AR, VR in journalism • AR, VR are Global • Conclusions: Key Questions and EM Effects.

3. I. Defining AR, VR (experiential media) • Augmented Reality (AR), or digital content layered in near real-time onto user’s real-world experience via mobile or wearable device, • Virtual Reality (VR), or supplanting user’s direct world experience with computer-mediated via headset or volumetric display.

4. II. Qualities of AR, VR: • Interactive, • Immersive, • Multi-Sensory, • Algorithm- and Data-Driven, • First-Person in Perspective, and, • Designed with a Natural User Interface.

5. VR-Journalism Confluence • Oculus Rift(wearable platform for immersive games, news stories, cinematic VR) • Founder Palmer Luckey was journalism major (2012 launch on Kickstarter$2M; 2014 Facebook buys $2.5B) • Interned with immersive journalist Nonny de la Pena • Called Godmother of VR • Considers VR an empathy machine, can make every news consumer a witness to events

6. 1. Interactivity…two forms • Interaction via network between and among public, journalists and other media story tellers (e.g., Facebook, Snapchat) • Interaction between user and content itself

7. A social media example:John Pavlik with augmented reality

8. Content interaction • Clicking to access embedded content or other story elements • Engaging artificial intelligence-enabled features

9. Interactive Documentaries

10. Interactive billboard

11. 2. Immersive • Enveloping the user via headworn display or other wearable device (e.g., via audio, imagery/video, 3D content; psychological immersion), • Ubiquitous content.

12. NYTimes Launches VR Nov. 5 2015:Shipped Google Cardboard free to 1 Million subscribers

13. First Story, then Daily 360 reporting Runs on VRSE app; VRSE is a VR production co.

14. CBS’s Jeff Glor walks the (virtual) plank

15. 3. Multi-Sensory

16. Haptics plus sight, sound

17. 4. Data & Algorithm-generated content • Algorithms are generating an increasing array of media content, from news to entertainment including EM • Narrative Science • Sample lead: Friona fell 10-8 to Boys Ranch in five innings on Monday at Friona despite racking up seven hits and eight runs. • Wordsmith from Automated Insights (Associated Press uses for automated financial reporting) • Algorithms stitch UHD video into 360 format

18. 5. 1st person, via HoloLens: MR in your home

19. 6. Natural User Interface: Digital audio assistants • Immersive, 360 degree audio • Amazon Echo with Alexa, Google Home • High fidelity microphones and speakers • Cloud-based interaction and control • Alexa, play Pandora • Alexa, what’s the news • Alexa, tell me a joke • Alexa, lower the volume • Alex, play Bosch (an LAPD detective case • Alexa, stop

20. Traditional Model Linear Passive Single/dual modality Static Episodic 3rd-person narrative: telling and showing Experiential Media Model Non-linear Interactive Multi-Sensory Dynamic Contextualized 1st-person point of view: immersive (3D) and experiential Transforming Storytelling in Media

21. VR: 3 DOF to 6 DOF Three DOF allows user to look about 360 image or video in any direction Six Degrees of Freedom: user can move up, down, left, right, forward, backward. Quality audio and video essential: Ultra high definition and high fidelity spatial audio creates highly lifelike user experience.

22. III. A Brief History of AR, VR • L. Frank Baum in 1901 imagines electronic “character marker”. • Ray Bradbury envisions electronic nursery in 1949 short story The Veldt. • Martin Heiliginvents in 1957 the ‘Sensorama” allowing four people to experience “the illusion of reality using a 3-D motion picture with smell, stereo sound, vibrations of the seat, and wind in the hair to create the illusion.” • Ivan Sutherland in 1968 develops first head-worn display. • JaronLanier in 1987 coins phrase Virtual Reality, designs first commercial VR headset. • Steve Mann in 1989 develops first functional and networked wearable computer. • Tom Caudell in 1990 coins the term augmented reality (AR)

23. IV: AR, VR in Journalism Pavlik & Nayar students create first 360 video journalism, 1997: ILGO Protest in St. Patrick’s Day parade

24. Höllerer, Feiner & Pavlik (1999):Immersive news via AR, the first Situated Documentary • Using AR/VR to layer digital content, usually geo-located, onto user’s real-world experience: the situated documentary

25. Pavlik & Bridges (2013): AR in art, news via HP Reveal

26. AR and 360 Video Use in News Media • By 2010, news media around the world experiment with AR in news. • In 2015, The New York Times launches VR news reporting with The Displaced, shipping 1 million Google Cardboard viewers to subscribers who can then immersively experience 360 video.

27. Many news media use AR & VR in reporting • The New York Times • USA Today • The Wall Street Journal • Philadelphia Inquirer • The Guardian • Associated Press • CBS News • Vice News • Many others around the world

28. Vice News live streamed news via AR eyewear

30. Lessons from NYT’s Daily 360 in First Year, 2017 • 360 video from 57 countries • 200 NYT journalists • 94 million views on Facebook • 2 million views on YouTube • 360 video from all sources captured 4.5 billion views on Facebook and YouTube (source: Tubular Data); growing but still small portion • Top ten video publishers on Facebook captured 14.5billion views on Facebook in Sept. 2017

31. February 2018: AR and journalism converge • The New York Times debuts AR journalism anchored in real world with Olympics coverage, viewed on smartphone. • Next slides: The user experience • The “story”: Nathan Chen, US Figure skater, in AR

32. NYTimeslaunches AR Journalism Feb. 2, 2018.

33. 3D objects virtually on flat surface.

34. Try it on an iPhone: it’s just the beginning.

36. Anchored AR is 3D

37. Enter Thai Rescue Cave Virtually

38. V. AR, VR are global • HTC (Taiwan), Huawei and Xiaomiinvesting in China’s AR & VR sectors, developing AR eye-wear and tethered and untethered VR head-mounted displays (HMD), including HTC VIve. • Also entering EM market: China’s Baidu, Tencent and Alibaba (owns the South China Morning Post). • South Korea’s Samsung manufactures and distributes Samsung Gear VR HMD • Oculus Go is among the untethered HMDs

39. VI. Conclusions—Key Questions. Do AR, VR: • Produce a sense of presence? • Transform public from passive audience to active participant? • Increase user engagement and generate greater empathy? • Increase recall, intention to act? Research says yes.

40. Looking Ahead, will AR, VR offer: • Disintermediatedmulti-sensory experiences via volumetric displays featuring 3D photorealistic, interactive animation and live action? • Will AR, VR content become virtually indistinguishable from reality? • Will neuroreality advance engagement? • Will AR, VR become unobtrusive, ubiquitous and persistent? • What are ethical and policy concerns: Threats to privacy, security, health (addiction), misinformation (or other crime)? Economic opportunity?

41. AR, VR & News: A Clash of Rights…and Wrongs? Freedom of speech and press (1st Amendment) Right to privacy (4th Amendment) Journalist Edward R. Murrow once said TV “can teach, it can illuminate; yes, and it can even inspire” (Murrow, 1958). It is up to humans to determine whether it is more than just “lights in a box.” Today, might we say something similar about AR, VR in news? Will AR, VR in news bring us closer to the truth…or will fake news transform into artificial reality?

42. Do you want to experience God? Well, unfortunately, I can’t help you with that. But through this 12/23/18 EM report from the NYT you can virtually experience the “God particle,” AKA the Higgs boson.