16-899A Pixels to Percepts

1. 16-899A Pixels to Percepts • Instructors: • Alexei (Alyosha) Efros, 225 Smith Hall, CMU • .Lavanya Sharan, Disney Research Pittsburgh • Web Page: • http://graphics.cs.cmu.edu/courses/P2P/

2. Today • Instructors • Why This Course? • Why Perception? • Administrative stuff • Overview of the course Old Lady or Young Girl?

3. Lavanya Sharan Postdoctoral Researcher Computer Graphics Computer Vision Human Visual Perception Visual perception, both behavioral and computational aspects.

4. Light or dark? Material category? RECENT WORK Emotion? Real or fake Oreo? GRADUATE WORK Where does he land?

5. Alexei “Alyosha” Efros • Associate Professor, Robotics & CSD • Ph.D 2003, from UC Berkeley (signed by Arnie!) • Research Fellow, University of Oxford, ’03-’04 • Research Interests • Computer Vision, Computer Graphics, Data-driven Approaches

6. PhD Thesis on Texture and Action Synthesis Smart Erase button in Microsoft Digital Image Pro:

7. Why This Course? • A perfect storm: • I wanted to do this for a long time • but felt I wasn’t qualified • Lavanya happened to be in Pittsburgh • and she is more than qualified • The first-ever textbook on Perception for CS types (Thompson et al) is coming out • We will be giving out printouts soon • There are great perception courses in Psych dept (e.g. by Klatzky), but our aims are different: • The aim to be much more applied • There are real problems in vision/graphics that we can solve • Exposure for these who might never take perception otherwise

8. Why study Perception? • It’s fascinating! • Ok, but so is astronomy • It’s still in the “early days” • Relatively easy to get up to speed • Still lots of low-hanging fruit • OK, OK, but why should I care? • I just want to create a seeing robot • and/or create the next “Avatars” • Two answers: 1) a classic one and 2) a real one

9. Classic Answer • Understanding Human Visual Perception should help us design better vision algorithms • After all, human is the only vision system known to work • Just like understanding how birds fly should help us design airplanes • …WAIT A MINUTE • Ok, so there is a fine line between “helping” and “mindless copying” • Beware of papers claiming to be “biologically inspired”

10. The Real Answer • Human Perception is an integral part of most vision and graphics endeavors • i.e. you can’t get away from it!

11. “What does it mean, to see? The plain man's answer (and Aristotle's, too). would be, to know what is where by looking.” -- David Marr, Vision (1982)

12. depth map Vision: a split personality • “What does it mean, to see? The plain man's answer (and Aristotle's, too). would be, to know what is where by looking. In other words, vision is the process of discovering from images what is present in the world, and where it is.” • Answer #1: pixel of brightness 243 at position (124,54) • …and depth .7 meters • Answer #2: looks like bottom edge of whiteboard showing at the top of the image • Which answer is better? • Is the difference just a matter of scale?

13. Measurement vs. Perception

14. Measurement vs. Perception Proof!

15. Measurement vs. Perception Pablo Picasso The Guitar Player

16. Vision as Measurement Real-time stereo on Mars Physics-based Vision Virtualized Reality Structure from Motion

17. Vision as Understanding • Object / Scene Recognition • Action / Activity Recognition • even scene geometry

18. Perception in Graphics • One of central goals of graphics is to simulate visual experience as realistically as possible • So, if we could simulate the physics of light perfectly, why bother with perception? • 1) Practical Reasons • 2) Artistic Reasons

19. Complexity • Amazingly real • But so sterile, lifeless… Why?

20. Device Limitations • Low Dynamic Range • Limited modalities (visual + audio) • No smell, taste, temperature • Limited field of view

21. Artistic Reasons • Reality is boring • Visual content creators (film directors, game designers) want hyper-reality

22. How to study Perception? • Ok, you convinced me: Perception is useful. • But why not just “open up the brain” and figure out how it works? • Two most popular methods in physiology: • Cell recordings • Recordings from small number of cells (1-100, out of millions) • fMRI • Overall activity in the brain as function of time • Very low spatial resolution

23. Tale of Martians with an old PC Cell recordings fMRI

24. Instead, in this course…

25. Course Goals • Read some interesting papers together • Learn something new: both you and us! • Get up to speed on big chunk of perception research • Use perception / human studies in your own research • Try your hand at a perception study • Learn how to speak • Learn how think critically about papers

26. Course Motto: 50/50 • 50% Lectures / 50% student presentations • 50% textbook / 50% research papers • 50% from perception side / 50% from vision/graphics side • 50% Alyosha / 50% Lavanya

27. Course Organization • Requirements: • Class Participation (25%) • Keep annotated bibliography • Post on the Class Blog before each class • Ask questions / debate / flight / be involved! • Three Projects (75%) • Two Analysis Projects (25% + 25%) • Implement / Evaluate a paper and present it in class • 1 CS paper, 1 perception paper • Synthesis (Final) Project (25%) • Produce a publishable result (study or implementation) • Can be continuation of analysis project or something new • Can be done solo or in groups of 2

28. Class Participation • Keep annotated bibliography of readings (always a good idea!). The format is up to you. At least, it needs to have: • Summary of key points • A few Interesting insights, “aha moments”, keen observations, etc. • Weaknesses. Unanswered questions. Areas of further investigation, improvement. • Before each class: • Submit your summary for current readings in hard copy (printout/xerox) • Submit a comment on the Class Blog • ask a question, answer a question, post your thoughts, praise, criticism, start a discussion, etc.

29. Analysis Project • Pick a paper / set of papers from the list • Understand it as if you were the author • e.g. Re-implement it; try to replicate the study • If there is code, understand the code completely • Run it on data the same data (you can contact authors for data and even code sometimes) • Understand it better than the author • Run it on LOTS of new data (e.g. LabelMe dataset, Flickr dataset, etc, etc), or try to replicate the study with new data • Figure out how it succeeds, how it fails, where it fails, and, most importantly WHY it fails • Look at which parts of the code do the real work, and which parts are just window-dressing • Maybe suggest directions for improvement. • Prepare an amazing 45 min presentation • Discuss with us twice – once when you start the project, and 3 days before the presentation

30. Synthesis Project • Can grow out of analysis project, or your own research • 1-2 people per project. • Project proposals mid-semester • Project presentations at the end of semester. • Results presented as a CVPR-format paper. • Hopefully, a few papers may be submitted to CVPR / VSS etc

31. End of Semester Awards • We will vote for: • Best Analysis Project • Best Synthesis Project • Best Blog Comment  • Prize: dinner in a French restaurant in Paris (transportation not included!) or some other worthy prizes

32. Course Outline (cont.) • Faces (week 11) • Visual Attention (week 12) • Motion Perception (week 13) • Visual Realism (week 14) • Perception and Art (week 15) • Final Project Presentations • Subject to Change

33. Perception & Art Livingstone