Using the Web to Interactively Learn to Find Objects

1. ΕΘΝΙΚΟ ΚΑΙ ΚΑΠΟΔΙΣΤΡΙΑΚΟ ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ ΜΕΤΑΠΤΥΧΙΑΚΟ ΜΑΘΗΜΑ : ΠΡΟΗΓΜΕΝΗ ΤΕΧΝΗΤΗ ΝΟΗΜΟΣΥΝΗ ΚΟΚΚΙΝΟΥ ΑΝΝΑ AM :Μ1253 Using the Web to Interactively Learn to Find Objects Mehdi Samadi, Thomas Kollar, Manuela Veloso

2. INTRODUCTION AIM: make mobile robots that are able to intelligently perform tasks APPROACHES: • Manually define the knowledge of the robot • Enable robot to actively query the WWW

3. RELATED WORK • Yamauchi 1997 (Baseline search) • Stachniss,Grisetti and Bulgar 2005 (focus on balancing the trade-off between localization quality and the need to explore new areas) • Meger 2008 (describe an integrated robotic platform) • Kollar and Roy 2009 (integration between robots and Web) • Sjoo 2009, Aydemir 2011, Velez 2011 ,Joho- Senk and Bulgar 2011 (focus on visual object search that does not leverage the Web)

4. Object Eval (OE) • An approach ,which addresses the challenges of the find and deliver task by querying the Web. (stages of find and deliver task: receive command, go to a location, ask for object, get object ,deliver object) PROCCESS: • Downloading and categorizing a set of web pages • Learning the probability that a location (ex.kitchen) will contain an object (ex. Coffee) Probability => Utility Function

5. FIND AND DELIVER TASK

6. UTILITY FUNCTION (part1) • Input: Object Name (ex. papers) and Destination Room • Output: A plan consisting of locations that robot should visit U( plan | O)= ∑ p( plani |O) × R ( plani|O) (1) U: Utility function O: Object name p(plani|O) : probability of finding an object in a location R: the reward that robot receives when it executes i-th step

7. UTILITY FUNCTION (part2) The problem can be formulated as finding the plan that maximizes the utility argplanmax U(plan|O) (2) R (plani|O) = D (plani) × I (plani) × F (plani) (3) R: the reward that robot receives when it executes i-th step D: the reward which is dependent on the distance I: the reward which is dependent on the number of interactions F: the reward which uses previous searches to help search for objects ( F = 1 or F = 0.5 or F = 0 )

8. UTILITY FUNCTION (part3) The probability will be computed as p(plani |O) ≈ [ ∏ ( 1 – p(lj| O))]× p(li| O) (4) where, p(lj=kitchen| O=coffee)=p(locationHasObject(kitchen,coffee)

9. QUERYING THE WEB • Objects present in a location will be found frequently on the Web • Object Eval converts predicate instances into a search query (ex: { papers, printer room} ) • The text on the web that is most relevant to a predicate instance will be near the search terms • OE extracts text snippets from each of the web pages ( up to 10 words before, after and between the terms). • Multiple text snippets from the same page are merged into one • Each text snippet is transformed into a feature vector

10. To find which location type should be assigned to a page, we train an SVM classifier (input :feature vector, output :location type) • OE only needs positive examples of the predicate

11. OE EVALUATION-SIMULATION • OE searches over candidate plans to maximize the utility function (beam search ,W=10 ,N=10 no loops) • At each step a new plani is added to overall plan • Query the Web to predict the probability distribution p(li | O) • A set of simulated commands were executed for three floors of an office building (80 object types)

13. RESULTS OE achieves a high F1 value even when it uses a few training examples The search for 20 objects is repeated 5 times starting from different initial location to obtain 100 runs.

15. Thank you!