Dagstuhl 6-06

1. Function-Based Object Recognition Louise Stark Electrical & Computer Engineering University of the Pacific Dagstuhl 6-06 1

2. Function-Based Research • Dr. Melanie Sutton Univ of West Florida • Dr. Kevin Bowyer Univ of Notre Dame Dagstuhl 6-06 2

3. Function-Based Research • Dr. Louise Stark Univ of the Pacific Stockton, CA Dagstuhl 6-06 3

4. University of the Pacific Dagstuhl 6-06 4

5. University of the Pacific Dagstuhl 6-06 5

6. University of the Pacific • Movies Filmed at University of the Pacific or in Stockton • 1949 All the Kings Men - Broderick Crawford, John Derek, Joanne Dru, John Ireland, Mercedes McCambridge • 1955 Blood Alley - John Wayne, Lauren Bacall • 1958 God’s Little Acre - Robert Ryan, Aldo Ray, • Tina Louise, Buddy Hackett • 1959 Porgy & Bess - Sidney Poitier, Dorothy Dandridge, Sammy Davis Jr. • 1960 High Time - Bing Crosby, Tuesday Weld, Fabian • 1967 Cool Hand Luke - Paul Newman, George Kennedy, • Jo Van Fleet • 1970 RPM (Revolutions Per Minute) - Gary Lockwood, • Ann Margaret, Anthony Quinn Dagstuhl 6-06 6

7. University of the Pacific Movies Filmed at University of the Pacific or in Stockton 1972 Fat City - Stacy Keach, Jeff Bridges, Susan Tyrell 1973 World’s Greatest Athlete - Tim Conway, Jan Michael Vincent, John Amos 1973 Poetry in Motion - Larry Hagman 1981 Raiders of the Lost Ark - Harrison Ford 1984 Dreamscape - Eddie Albert, Dennis Quaid 1984 The Sure Thing - John Cusack, Tim Robbins 1988 Glory Days - Robert Conrad Dagstuhl 6-06 7

8. University of the Pacific Movies Filmed at University of the Pacific or in Stockton 1996 Inventing the Abbotts - Joaquin Phoenix, Liv Tyler 1996 Flubber - Robin Williams 1997 Dead Men on Campus - Mark Paul Gosselear, Tom Everett Scott Dagstuhl 6-06 8

9. Overview • Relationship to Seminar • Basics of the Function-based Approach • Framework for Context-based Reasoning • Experimental Results • Questions Posed Dagstuhl 6-06 9

10. Relationship to Seminar • Ontology • Affordance • Computer Vision • Object representation Dagstuhl 6-06 10

11. Ontology onhtolhohgy According to Webster… 1) <philosophy> A systematic account of existence Dagstuhl 6-06 11

12. Ontology onhtolhohgy According to Webster… 2) <artificial intelligence> (from philosophy) An explicit formal specification of how to represent the objects, concepts and other entities that are assumed to exist in some area of interest and the relationships that hold among them. Dagstuhl 6-06 12

13. Ontology onhtolhohgy According to Webster… 3) <information science>The hierarchical structuring of knowledge about things by subcategorizing them according to their essential (or at least relevant and/or cognitive) qualities. Dagstuhl 6-06 13

14. Affordance According to Webster… 1) <graphics> A visual clue to the function of an object. Dagstuhl 6-06 14

15. Gibson’s Theory of Affordances • “Perhaps the composition and layout of surfaces constitutes what they afford” Example: “..a terrestrial surface is nearly horizontal…, nearly flat…, and sufficiently extended (relative to the size of the animal) and if its substance is rigid (relative to the weight of the animal), then the surface affords support” The Ecological Approach to Visual Perception, J.J. Gibson Dagstuhl 6-06 15

16. Gibson’s Theory of Affordances • Properties noted: • horizontal • flat • extended • rigid Physical properties, measured relative to the animal. The Ecological Approach to Visual Perception, J.J. Gibson Dagstuhl 6-06 16

17. Gibson’s Theory of Affordances • “If a surface of support is knee-high above the ground, it affords sitting on. • We call it a seat in general. • If it can be discriminated as having just these properties, it should look sit-on-able. • If it does, the affordance is perceived visually.” The Ecological Approach to Visual Perception, J.J. Gibson Dagstuhl 6-06 17

18. Gibson’s Theory of Affordances • “To perceive an affordance, is not to classify an object.” • “If you know what can be done with… an object, what it can be used for, you can call it whatever you please.” • “You do not have to classify and label things in order to perceive what they afford.” The Ecological Approach to Visual Perception, J.J. Gibson Dagstuhl 6-06 18

19. Computer Vision? • Deriving meaningful descriptions of the environment from images • Descriptions needed for • - recognition • - manipulation • - reasoning about objects Dagstuhl 6-06 19

20. A Representation is always needed... Forms of representation commonly used • pictures of objects you need to recognize (restricted) • complete 3-D models - rotate the object around, project view from different positions, does it match? • structured models (recognition by parts, geons) • parameterized models, allows variability, but still held to the specified parts All successful in some way for a limited domain. Dagstuhl 6-06 20

21. Earlier Work • Roberts • “Machine perception of three dimensional solids” 1965 • Analyze intensity image • Extract edge information • Match against library of geometric models • - “Model-based vision” paradigm • - “Single arbitrary view 3-D object recognition” paradigm Dagstuhl 6-06 21

22. Generic Representation Schemes • Parameterized 2-D feature configurations • Parameterized 3-D geometric models • Parameterized 3-D structural models Dagstuhl 6-06 22

23. Motivation Parameterized Model Structural Model Could these be recognized? Dagstuhl 6-06 23

24. What is the goal? Develop alternative approaches to generic object recognition & manipulation - concentrate on man made objects (artifacts) Human artifacts – existence or non/existence of properties can be deduced by analyzing the shape of an object For any particular object category – there is some set of functional properties shared by ALL objects in that category. Dagstuhl 6-06 24

25. What is the goal? Question – How do we recognize objects we have never previously encountered? - we don'thave a model (or do we?) Essentially- We categorize objects using some type of "model" Dagstuhl 6-06 25

26. More Terminology • Category – using Rosch’s terminology • Basic Level Category – those that carry the most information, most differentiated from one another • Subcategory– Subordinate Categories • Each subcategory has its own set of functional attributes that may overlap with other subcategories Dagstuhl 6-06 26

27. CAD-based vs Function-based • Model-based vision • Proceeds from the assumption at a model is available for each object • Function-based vision • - Models correspond to generic categories of objects Dagstuhl 6-06 27

28. Earlier Work • Winston, Binford, Katz and Lowry • “Learning physical descriptions from functional definitions, examples and precedents” 1984 • Discussed used of function-based definitions of object categories • Infinity of individual physical descriptions of objects in a category… • Single functional description to represent all (cup example) Dagstuhl 6-06 28

29. Earlier Work Binford “Survey of model-based image analysis systems” 1982 “The essential definition of object class is functional. … Object classes have an associated 3-D form: form equals function. … Dagstuhl 6-06 29

30. Earlier Work Binford “Survey of model-based image analysis systems” 1982 “An object’s function is often a geometric function. The function of a room is to be an enclosing volume. … The function of a chair… is to be a flat surface at a comfortable height for sitting….” Dagstuhl 6-06 30

31. Earlier Work • Brady, Agre, Braunegg and Connell • “The mechanics mate” 1985 • Connell and Brady • “Generating and generalizing models of visual objects” 1987 • Discussed relation between geometric structure and functional significance • Generalized structural description learned from sequence of examples Dagstuhl 6-06 31

32. Earlier Work • Ho • “Representing and using functional definitions for visual recognition”, 1987 • Considered 2 functional concepts: chair and support – what is needed to represent function for recognition • 2D cross section of object used Dagstuhl 6-06 32

33. Earlier Work • DiManzo, Trucco, Giunchiglia, Ricci • “FUR: Understanding Functional Reasoning”, 1989 • Utilized functional knowledge within an expert system framework • Primitives defined as individual expert systems that evaluate 3D information Dagstuhl 6-06 33

34. Earlier Work • Rivlin and Rosenfeld • “Navigational Functionalities”, 1995 • Explored functionality as it relates to mobile robots • Navigating agent may classify objects in its environment in functional terms as “threat,” “landmark” and so on. Dagstuhl 6-06 34

35. Background Minsky “The Society of Mind”, 1985 “… The solution is that we need to combine at least two different kinds of descriptions. On one side, we need structural descriptions for recognizing chairs when we see them. ” Dagstuhl 6-06 35

36. Background Minsky “The Society of Mind”, 1985 “… On the other side we need functional descriptions in order to know what we can do with them… we need connections between parts of the chair structure and the requirements of the human body that those parts are supposed to serve. “ Dagstuhl 6-06 36

37. What do I mean by "function-based"recognition? "Recognition" categorization of the input shape as a member of a class of objects (e.g. dishes, furniture, handtools) "Function-based" classes of objects can be defined by functional requirements (e.g. provides_containment, provides_sittable_surface) Dagstuhl 6-06 37

38. GRUFF Generic Recognition Using Form and Function chair(cher) n. - a piece of furniture for one person to sit on Dagstuhl 6-06 38

39. Approach to the Problem Derive the format of my function-based representation Confirm feasibility of appoach test domain- perfect input - planar face models Expand the domains Test real data Interact to confirm functionality Exploit contextual information Dagstuhl 6-06 39

40. Knowledge in GRUFF is of three types: A category hierarchy which specifies superordinate / basic / subordinate categories furniture  chair  arm chair Functional properties that define each catgory (provides_sittable_surface, provides_stability,...) Knowledge primitives used to reason about shape (dimensions, relative orientation, ...) All organized into a "category definition tree" which is GRUFF's knowldge about the world. Dagstuhl 6-06 40

41. Category Representation Tree Conventional Chair Provides Sittable Surface Provides Stable Support Dagstuhl 6-06 41

42. We imagine the definition of a generic object category to be something like... straight_back_chair ::= provides_seating_surface + provides_stability + provides_back_support_surface and recognition is conceptualized as ... Provides_back_support provides_arm_support Provides_sittable_surface provides_stable_support Dagstuhl 6-06 42

43. Shape-based Knowledge Primitives A functional requirement such as : provides_sittable_surface is implemented as a sequence of calls to shape-based operators. dimensions(shape_element, dimensions_type, range_parameters) relative_orientation(normal 1,normal 2, range_parameters) clearance(shape_element clearance_volume) Dagstuhl 6-06 43

44. Knowledge Primitives Abstract shape reasoning • Metric dimensions (width, depth, height, area, contiguous surface, volume • Proximity • Relative orientation • Clearance • Stability • Enclosure Dagstuhl 6-06 44

45. Knowledge Primitives Physical interaction reasoning • Change orientation • Apply force • Observe deformation Dagstuhl 6-06 45

46. Evaluation Measures Value returned from knowledge primitive invocation 1.0 Evaluation Measure 0.0 least low high greatest ideal ideal Values of Shape Property Dagstuhl 6-06 46

47. Combining Evidence Combine required measurements using probabilistic AND (0-1) Combine descendent subcategory node measure using probabilistic OR Dagstuhl 6-06 47

48. Recognition Process • Category representation graph is control structure • Structural Constraint Propagation – subcategory nodes constrained by what was found for the parent Dagstuhl 6-06 48

49. Recognition Stage 2 approaches 1. Check all known categories in the knowledge base 2. Confirm/deny object can/cannot function as a specified (sub)category Dagstuhl 6-06 49

50. Valid Chairs Recognized by GRUFF Dagstuhl 6-06 50