CrowdSearch : Exploiting Crowds for Accurate Real-Time Image Search on Mobile Phones

1. CrowdSearch: Exploiting Crowds for Accurate Real-Time Image Search on Mobile Phones Original work by Yan, Kumar & Ganesan Presented by Tim Calloway

2. Roadmap • Problem Description • What is “crowdsourcing”? • System Architecture • The CrowdSearchAlgorithms • Delay Prediction • Validation Prediction • Experimental Evaluation • Discussion/Criticism • Questions

3. The Perceived Problem • Text-based search is easy…

4. The Perceived Problem • Mobile-based search will become more important in the future. • More than 70% of smart phone users perform searches. • Expected to be more mobile searches than non-mobile searches soon • Text-based mobile searches are easy as well…

5. The Perceived Problem • But how does a mobile phone user search for this? • No visible words/letters; too far away to know the address.

6. The Perceived Problem • Ways to find out what that building is: • Ask random people on the street • Travel to the building to see the address/sign • Take a picture of the building with your mobile device and send to a search engine… • How easy is image searching on a mobile phone though?

7. The Perceived Problem • Image search is a non-trivial problem – have to deal with variations in lighting, texture, image quality, etc. • Even when results are returned, scrolling through multiple pages on a mobile device is cumbersome. • Search should be precise and return very few erroneous results. • Multimedia searches require significant • Memory • Storage • Computing resources

8. The Proposed Solution • CrowdSearch – Attempts to provide an accurate, image search system for mobile devices by combining… • Automated image search and • Real-time human validation of search results • Leverage crowdsourcing through Amazon Mechanical Turk (AMT)

9. The Proposed Solution • Humans are good at comparing images • Could an automated search determine these two images are of the same building? • Crowdsourcing increases search result accuracy.

10. Balancing Tradeoffs • Result delay • Should minimize delay or at least keep it within a user-provided bound • Result accuracy • Strive for high (i.e., ≥ 95%) accuracy • Monetary cost • Low cost is better than high cost • Energy • Should consume minimal battery power

11. System Architecture

12. System Architecture • Three main components: • Mobile Device • Initiates queries • Displays responses • Performs local image processing (maybe) • Remote Server • Performs automated image search • Triggers image validation tasks • Crowdsourcing System (AMT) • Validates image search results

13. System Operation Overview

14. System Operation Overview

15. System Operation Overview • How do we minimize delay and cost while maximizing accuracy?

16. Accuracy Considerations • How many validations are required for 95% accuracy? • Requiring at leastthree validationsout of five achieves≥ 95% accuracy.

17. Optimizing Delay • Utilize parallel posting • Post all candidate images to the crowdsourcing system at the same time. • But this approach increases cost! 5 cents 5 cents 5 cents 5 cents = 20 cents

18. Optimizing Cost • Utilize serial posting • Post top-ranked candidate first, wait for responses, then post next candidate if necessary. • This approach increases delay!

19. CrowdSearch Delay/Cost Optimization • Combine elements of parallel and serial posting • Prediction requires delay and validation models • Goal: want at least one verified result by the deadline.

20. CrowdSearch Delay/Cost Optimization

21. Delay Prediction Model • The delay of a single response is the combination of acceptance delay and submission delay. • Both of these follow an exponential distribution with an offset. • Thus, overall delay is the convolution of these delays.

22. Delay Prediction Model Performance

23. Validation Model • Given a response set S, want to compute probability of positive validation result. • Use training data to set these probabilities • If the probability of a positiveresult is less than somethreshold, send the nextcandidate to validation. • In this example, if the threshold were set to < 76%, the server would post the next candidate image to AMT.

24. Power Considerations • Should some image processing occur on the local device or should it be outsourced to the server? • It depends! • Use remoteprocessing when WiFi is available. • Use local processingwhen only 3G is available

25. Experimental Results • Any of the crowdsourcing schemes lead to better results! • Some types of imagesare easier for automated searchesto handle than others

26. Experimental Results • CrowdSearch leads to (given a long enough deadline)… • Behavior close to parallel posting for recall • Behavior close to serial posting for search cost

27. Thoughts/Criticism • The limited nature of the solution • Limitation to the four categories • Buildings • Books • Flowers • Faces • Only 1000 images in the backend database. • Would increasing the number of automated search images increase total task time in a significant way?

28. Thoughts/Criticism • How useful is this anyway? • Are people willing to go through the trouble to set up a payment account and pay 5-20 cents for a search? • How much effort would it usually take for someone to find out what the object is through traditional means? • Especially for books! • Privacy concerns • People utilizing CrowdSearch must accept the fact that random strangers know what they are looking at and searching for. • Additionally, their GPS information might be provided to the CrowdSearch servers. • What about the privacy of the object of the search? • Undercover police officers

29. Questions?