Development of an Objective, Language-Based Measure of Perceived Video Quality

1. Development of an Objective, Language-Based Measure of Perceived Video Quality Daniel R. Saunders, Peter J. Bex, and Russell L. Woods Schepens Eye Research Institute and Harvard Medical School Boston, Massachusetts

2. The Schepens Eye Research Institute • Dedicated to fighting blindness and visual impairment • Treatments and assistive technology for eye disorders • Example: Age-related Macular Degeneration • Affects at least 1.75 million in the U.S. • Central vision loss, leading to reduced contrast sensitivity and visual acuity • No cure Daniel Saunders

3. Digital video enhancement Contrast enhancement (Peli and Peli, 1984; DigiVision CE-3000) Daniel Saunders

4. Digital video enhancement Digital magnification Daniel Saunders

5. How to evaluate enhancements? • Easy to measure preference: • Dual monitor • Rating Adaptive contrast enhancement has been shown to be preferred in many cases. (Kim et al., 2004; Fullerton et al., 2008) But subjective. An objective measure would derive from how much it helps people perform a task. Reading, driving, watching TV. Daniel Saunders

6. Information transfer • Principle: The better the perceived quality of a video clip, the more (and more accurate) information the viewer should be able to report about it. • Impaired perception should result in reduced amount of accurate information. • Use of multiple choice quiz questions has problems • Creation of questions is subjective • Labour intensive • Also may have low sensitivity (Peli, 2005) Daniel Saunders

7. Using free recall to measure information transfer “Describe this video as if to someone who hasn't seen it.” Standard set of 200 video clips, 30 seconds each, from Hollywood films and nature documentaries. Daniel Saunders

8. Collecting “normal” responses • 4000 responses from Amazon.com Mechanical Turk workers (20 per clip) working from home • 2400 responses from normally-sighted in a lab setting, stratified by age and gender (in progress) Examples: A spotlight displays the Batman symbol. Batman arrives on a rooftop and a beautiful redhead appears from behind the spotlight wearing a black negligee and robe. She flirtatiously walks over to the caped crusader and they converse. Batman swoops on. As he lands a woman comes out and starts talking to him. There is some plan afoot that Batman must thwart and she seems helpful by her look. Batman flies onto a roof top and starts talking to a blonde woman. The scene is very dark. We first see a bat spotlight, then Batman flies into the room. A woman with long hair comes out and speaks with Batman. It is a very dark room with only a "Batman" Logo shining into the night. This is how Batman is called upon. Once he arrives, it is in a darken room with a very seductress Nicole Kidman to greet him there. It starts out with a huge bright spotlight with Batman's stencil on it shining up into the night. It's a very dark scene, when Batman arrives. He is then approached by a young woman with long wavy red hair in a black robe and slinky nightgown. There is a slight breeze blowing her hair. A giant floodlight has a cutout of Batman in it. Batman appears out of nowhere and walks by the spotlight. A woman wearing a sexy dress walks into the room and talks with Batman. Daniel Saunders

9. Evaluate responses in terms of their discrepancy A spotlight displays the Batman symbol. Batman arrives on a rooftop and a beautiful redhead appears from behind the spotlight wearing a black negligee and robe. She flirtatiously walks over to the caped crusader and they converse. The scene is very dark. We first see a bat spotlight, then Batman flies into the room. A woman with long hair comes out and speaks with Batman. It is a very dark room with only a "Batman" Logo shining into the night. This is how Batman is called upon. Once he arrives, it is in a darken room with a very seductress Nicole Kidman to greet him there. It starts out with a huge bright spotlight with Batman's stencil on it shining up into the night. It's a very dark scene, when Batman arrives. He is then approached by a young woman with long wavy red hair in a black robe and slinky nightgown. There is a slight breeze blowing her hair. A giant floodlight has a cutout of Batman in it. Batman appears out of nowhere and walks by the spotlight. A woman wearing a sexy dress walks into the room and talks with Batman. Batman swoops on. As he lands a woman comes out and starts talking to him. There is some plan afoot that Batman must thwart and she seems helpful by her look. Batman flies onto a roof top and starts talking to a blonde woman. • Score for an individual is average distance of their responses from the normally-sighted responses. • A high score indicates poor vision/perceived video quality. Daniel Saunders

10. Text passage similarity metrics • Baseline: shared word count (common word removed) • Performance (Mechanical Turk): Mean shared words within a clip = 6.8, outside a clip = 1.7. Accuracy at matching responses to clips = 95%. • Latent Semantic Analysis could deal with synonyms etc. Used in essay grading and finding articles by matching abstracts. • Not our wheelhouse! Daniel Saunders

11. Assessing validity of the measure Perceived video quality = Perception + video quality. Measure can be used to assess video, with applications in e.g. video compression algorithms. Future studies: • Normally-sighted participants watch blurred video. • Greater blur should cause larger discrepancy scores • Within-subject design, so compensates for variability • Visually-impaired participants watch enhanced video. Daniel Saunders

12. What we have: • An application (evaluating the benefit of video enhancement to the visually impaired). • A dataset (free descriptions of 200 video clips). • What we need: • A metric of text passage discrepancy. (LSA?) • Thanks! Daniel Saunders

13. Mechanical Turk for data collection • 3000 responses in 30 days Daniel Saunders

14. No objective measure of perceived video quality • Most are based on ratings or side-by-side comparisons • Relevant to perception as well as to videos. Daniel Saunders

15. Age-related macular degeneration • Affects more than 1.75 million people in the United States(Friedman et al., 2004), 3 million by 2020. • Causes 50% of all cases of blindness in the United States (WHO, 2002). • No cure, only aids. • Which aids really help? (the same question as perceived visual quality) Daniel Saunders

16. TV is an important activity • How can we help people watch TV? • How can we evaluate whether we’ve helped? 20% of seniors who went in for a visual impairment exam said that one of their main motivations was watching TV. 69% of people with central field loss reported at least “some” difficulty with watching TV Daniel Saunders