Veggie Vision by IBM

1. Veggie Vision by IBM Ideas about a practical system to make more efficient the selling and inventory of produce in a grocery store. CSE 803 Fall 2008 Stockman

2. Problem is recognizing produce • properly charge customer • do inventory • save customer and checker time CSE 803 Fall 2008 Stockman

3. 15+ years of R&D now This information was shared by IBM researchers. Since that time, the system has been tested in small markets and has been modified according to that experience. CSE 803 Fall 2008 Stockman

4. Up to 400 produce types CSE 803 Fall 2008 Stockman

5. CSE 803 Fall 2008 Stockman

6. Practical problems of application environment CSE 803 Fall 2008 Stockman

7. Engineering the solution CSE 803 Fall 2008 Stockman

8. System to operate inside the usual checkout station • together with bar code scanner • together with scale • together with accounting • together with inventory • together with employee • within typical store environment • * figure shows system asking for help from the cashier in making final decision on touch screen CSE 803 Fall 2008 Stockman

9. Modifying the scale CSE 803 Fall 2008 Stockman

10. Need careful lighting engineering CSE 803 Fall 2008 Stockman

11. Need to segment product from background, even through plastic CSE 803 Fall 2008 Stockman

12. Previously published thresholding decision CSE 803 Fall 2008 Stockman

13. Quality segmented image obtained CSE 803 Fall 2008 Stockman

14. Design of pattern recognition paradigm (from 1997) FEATURES are: color, texture, shape, and size all represented uniformly by HISTOGRAMS Histograms capture statistical properties of regions – any number of regions. CSE 803 Fall 2008 Stockman

15. Matching procedure • Sample product represented by concatenated histograms: about 400 D • 350 produce items x 10 samples = 3500 feature vectors of 400D each • Have about 2 seconds to compare an unknown sample to 3500 stored samples (3500 dot products) • Analyze the k nearest: if closest 2 are from one class, recognize that class (sure) CSE 803 Fall 2008 Stockman

16. HSI for pixel color: 6 bits for hue, 5 for saturation and intensity For each pixel quantify H HIST[H]++ same for S&I CSE 803 Fall 2008 Stockman

17. Histograms of 2 limes versus 3 lemons • Distribution or population concept adds robustness: • to size of objects • to number of objects • to small variations of color (texture, shape, size) CSE 803 Fall 2008 Stockman

18. Texture: histogram results of LOG filter[s] on produce pixels Leafy produce B Leafy produce A CSE 803 Fall 2008 Stockman

19. Shape: histogram of curvature of boundary of produce CSE 803 Fall 2008 Stockman

20. Banana versus lemon or cucumber versus lime Small range of curvatures indicates roundish object Large range of curvatures indicates complex object CSE 803 Fall 2008 Stockman

21. Size is also represented by a histogram CSE 803 Fall 2008 Stockman

22. Each pixel gets a “size” as the minimum distance to boundary Chinese eggplants Purple grapes CSE 803 Fall 2008 Stockman

23. Learning and adaptation • System “easy” to train: show it produce samples and tell it the labels. • During service: age out oldest sample; replace last used sample with newly identified one. • When multiple labeled samples match the unknown, system asks cashier to select from the possible choices. CSE 803 Fall 2008 Stockman

24. Where is Veggie Vision today? • http://www.internetnews.com/xSP/article.php/3642386 • System uses almost all color features • Installed in few places: many stores have self-checkout, putting work on customer. • IBM has a “shopping research” unit http://www.usatoday.com/tech/news/techinnovations/2003-09-26-future-grocery-shop_x.htm • Customers will tolerate a higher human error than a machine error CSE 803 Fall 2008 Stockman