Business Identification : Local Neighborhood

1. Business Identification:Local Neighborhood Alexander Darino

2. Outline • Where Am I? project obtains geolocation of camera from image  • Objective: Obtain the geolocation and address of Businesses in image • Assume Business is nearby, eg. < 100m from camera • Compare methods of obtaining this information

3. Outline Business Name Matching Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses Business Identification Image OCR Detected Text

4. Outline • This Week: • Finding Local Businesses via Geocode Search • Finding Local Addresses via Reverse Geocoding • Extracting Identifying Text (ie. store names) via Optical Character Recognition (OCR) • Matching OCR text to Business Names • Next Steps/Weekend Objectives • Acknowledgements

5. Obtaining Business Names ✓

6. Local Businesses: Geocode Search • Used Three Place-Search APIs: • Yelp API - detailed yellow page-type results • Google Places API - "Skinny" + Reference to more information • CityGrid API - minimal yellow page-type results • Used by Yellow Pages, Super Pages  • At present, only interested in business names  • Aggregated names from all three APIs  • Example (next slide)

7. Local Businesses: Geocode Search 40.441127247181797 -80.002821624487595 Denham & Company Salon Ullrich's Shoe Repairing Nicholas Coffee Co Bella Sera On the Square A & J Ribs Starbucks Coffee Jenny Lee Bakery Galardi's 30 Minute Cleaners Jimmy John's Gourmet Sandwiches Charley's Grilled Subs Fresh Corner Lagondola Pizzeria & Restaurant Camera Repair Service Inc Pittsburgh Cigar Bar Original Oyster House MixStirs 1902 Tavern Costanzo's Pittsburgh Silver Llc Graeme St Galardi's 30 Minute Cleaners Denham & Co Salon Bruegger's Bagel Bakery Nicholas Coffee Co Market Square Fat Tommy's Pizzeria Mixstirs Cafe Giggles Rycon Construction Inc Garbera, Dennis C, Dds - Emmert Dental Assoc Bella Sera on the Square Mancini's Bread Co Las Velas Ciao Baby Washington Reprographics Inc Highmark Life Insurance Co Fischer, Donald R, Md - Highmark Life Insurance Co Jimmy John's Lynx Energy Partners Inc Emmert Dental Assoc

8. Local Businesses: Geocode Search Results: 12 Success, 3 Partial Q9: First Presbyterian Church [turns out it wasn't a cathedral] (SUCCESS) Q28: Moe's (SUCCESS) Q34: Bruegger's Bagels (SUCCESS) Q35: Breuggers, Tavern, Nicholas (SUCCESS) Q42: Tavern, Nicholas, Constanzo's [in distance] (SUCCESS) Q57: Tambellini (SUCCESS) Q63: Benedum Center (SUCCESS) Q141: Roberts/7-Eleven (PARTIAL - misses Roberts) Q200: Goodyear (SUCCESS) Q238: Far from Bruegger's, Tavern, Nicholas (PARTIAL - misses Tavern) Q246: Some theater (can't read it) (SUCCESS) Q249: George Aikens (SUCCESS) Q260: Dogs Dun Wright, Cherrie's diner (SUCCESS) Q300: Giggles, Bruegger's, Tavern (in distance) (SUCCESS) Q318: Fifth Avenue Place, Wines & Spirits (PARTIAL - misses Wine & Spirits)

9. Local Businesses: Geocode Search • Strengths • Aggregated results almost always found Business of interest • Weaknesses • Each API limits query result set size - this is why we aggregate • Contacted Yelp, Google, CityGrid for extended API Access. • Heard back from CityGrid; conference call next week. • Only businesses listed • Not all businesses listed • All but one "Partial" result were for unlisted businesses • Limitations • Have only tested for 15 Pittsburgh images - unknown result quality for rural areas.

10. Local Businesses: Geocode Search ✓ ✓* ✓ * Implicitly verified: APIs can search by latitude/longitude OR address

11. Local Addresses: Reverse Geocoding • Used Two Reverse-Geocoding APIs • Google: provides a range of addresses on the same road • Usually the road is correct, but sometime's it's slightly off • Sometimes the road is correct, but the actual address number is not in the range • Bing: provides one or two proximate addresses • Rates it's own confidence. Even 'Medium' confidences are very accurate • Address is never exact, but is almost always adjacent to correct address • Results returned never consistent: always returns one or the other or both of the two addresses regardless of confidence level

12. Local Addresses: Reverse Geocoding • Intent: Get up to ~500 nearby addresses • No Address Search API Available ✓ ✓* ✓ ✗

13. Extracting Identifying Text: OCR Business Name Matching Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses Business Identification Image OCR Detected Text

14. Extracting Identifying Text: OCR • Given: • List of nearby businesses (names, addresses, etc) • Image containing businesses with visible names • Objective: • Extract name of businesses from image • Identify businesses located in image • Match names extracted from image to names in business list

15. Extracting Identifying Text: OCR • Used Two OCR APIs: • GNU OCR (Ocrad) • GOCR • OCR APIs highly sensitive to: • Font (only works well with roman font) • Perspective • Scale • Binarization Threshold • Dark on Light vs. Light on Dark (inversion)

16. Extracting Identifying Text: OCR • OCR API evaluations • Ocrad - could not yield any meaningful data across over 200 scale/threshold/inversion combinations • GOCR - produced good results across 10 scales with and without inversion using threshold automatically determined by Otsu's method • Examples of GOCR output (next slides)

17. Extracting Identifying Text: OCR

18. Extracting Identifying Text: OCR n.c.......o.a...u..............oU..D.oa..e......_RuEGGE..KERy..J...w...........L........M.II.....c.. ...i.......l..J.t...llt...lSHA.P.It..tllt.........._.l...Jy._.c_...._tt.._....t.._.r.........t.t_t.._.._.l..J.r.r.I.

19. Extracting Identifying Text: OCR

20. Extracting Identifying Text: OCR u..........._nq......eoR.E.l.e...í....e...n... .n....n.e.R.E...e....o. _....E.R.E.IKE........I.ltlO.. .......rE..o......E.....I.K.E.o..... J.n....c...E.R.E.I.E...... .M..E.R.E...E...a J...Gu.ge..geE.F .._.....E..gE.D... fUlI..lll.lll.IIi.l..Xl..

21. Extracting Identifying Text: OCR

22. Extracting Identifying Text: OCR ..e_..w.._......D.........uJ.....J.................n......n..........n_..r.l_d..J.ec.m._..n.......J.n.._...tn..ct..._.................D.u.v...e.n....u.. Y.._w.n.n....Jn.......G..o..r..._........J...ml.t..l.tt.l.._w....................._....l....t........j..ilI.i..

23. Extracting Identifying Text: OCR

24. Extracting Identifying Text: OCR __.ncu_.l..._..._J...ne......._n._..v.....ra......d_..._.............i..n..UllREsT.unAN...r.c.....r...Tt.rJll......m...c.....n....... ...Jn.I..c...r.rESTAU.ANT.r.O....c.cc. Note: Even though "Tambellini" is a roman font, it is too stretched to be picked up by GOCR

25. Extracting Identifying Text: OCR • Strengths • Applicable to expected input of orthogonal images • Output can be run through word similarity matching algorithms • Weaknesses • Only works well(-ish) for strictly roman font • Limitations • Will perform poorly for artistic fonts and business signs • Conclusion • By itself, OCR is not the best approach towards Business identification (poor recognition, franchises, perspective, etc) • OCR could be used as part of Business identification voting scheme 

26. Matching OCR Text to Business Names Business Name Matching Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses Business Identification Image OCR Detected Text

27. Matching OCR Text to Business Names • Fuzzy String Matching: TRE Package • Approximate Regular Expression Matching • Returns edit-distance of matched text • Filter OCR text • Trimming • Chunking • Uselessness (ie. Less than two letters) • Developing algorithm to rate confidence of business name appearing in image

28. Matching OCR Text to Business Names

29. Matching OCR Text to Business Names

30. Next Steps/Weekend Objectives • Implement ‘chunking’ to OCR output • Evaluate and refine algorithm against multiple inputs • Detect location of text in image

31. Acknowledgements • Subh • Directed us to the Ocrad and GOCR OCR packages • Provided feedback on how to calibrate OCR packages to extract meaningful text (eg. scaling, inversion, etc)

32. Thank You