SAVI: Shopping Assistant for Visually Impaired Individuals

1. SAVI Shopping assistant for the visually impaired Andrew EbaughSaurav Chatterjee

2. Basic concept of SAVI • What does it do? • SAVI allows blind people to locate and identify items via RFID technology. • What is it good for? • Blind people can shop by themselves. • Facilitates in identifying items once they get home. • It can substitute for Braille signs and tags.

3. Why is SAVI needed? Currently, blind individuals cannot navigate a grocery store independently. • Have to depend on others to help with shopping (Braille stickers on products) • Little choice on prices and brand of items being purchased • Have to depend on others when checking out at the register • Organizing their items once they return home is a problem

4. Product Demo • We will display three active pieces: Personal Server, ear piece, & scanning glove • Simple demo idea: Identify common grocery items with our system without using sight. • Plan to set up a display of several tagged items on shelves • Generate a grocery list from among the several possible items • Locate the correct items by handling the objects and scanning the tags • As our audio board also transmits over FM, we’ll have a small stereo set up to broadcast what the device user will be hearing

5. Product Demo, cont. • Our original scenario was pretty close to this • it was for an actual user in a grocery store • It showed a user receiving audio feedback at the register, which we will not demo • It also showed the user putting away groceries at home, which we will not demo

6. Implementation – Overall diagram Input Output Processing

7. Implementation, cont. • RFID scanner – mote running TinyOS transmits RFID tag scan to a mote in a CF card on personal server • CF card mote – accessible as a serial port on personal server, /dev/tts/3 • Serial forwarder service – dumps whatever is received on the serial port to listeners on a network port (4444 TCP) • ODBS listener service – Connects to network port, and registers as a listening client, accepting forwarded packets.

8. Implementation, cont. • ODBS – Multi-threaded: port listening thread, database query thread, output thread • Product information is stored in SQL database • Upon receiving a RFID scan, the service queries the product database • Additionally queries for brand, price, sale information • Each piece of information returns an associated audio clip; from all of these, a presentation “playlist” is generated

9. Implementation, cont. • Database is provided by sqlite and accessed by sqlite JDBC-connector • Contains 7 tables: product, brand, measure, location, sale, alphanum Product table pseudo-SQL : CREATE TABLE product (rfid, upc, brand_id, name, price, price_unit_id, price_quantity, sale_type, sale_quantity, sale_price, quantity, quantity_unit_id, location_id, section_id, clip) Ex:INSERT INTO product VALUES ('E00781BCC1957A75', '04482008', 9, 'Oreo Cookies', '3.69', NULL, NULL, 2, NULL, NULL, '1.13', 0, 9, 18, 'product-11');

10. Implementation, cont. • Presentation clip – Once the clip has been generated, ODBS makes a call to Runtime.exec() to execute an mp3 audio player • Execution call takes the form of: madplay /mp3/brand-04.mp3 /mp3/product-02.mp3 /mp3/alphanum-39.mp3 • MP3 player talks to audio board as /dev/dsp

11. Implementation, cont. • Slappy board – Audio is provided by board that plugs into personal server • Board drivers are modular and with dependent modules (soundcore, ac97_codec) provide standard linux audio interface to be accessed by applications • In addition to standard stereo output and line input, provides FM transmission on configureable frequency. • All this together gives a range of options, from enhanced ODBS access of audio board, to a multitude of output devices

12. Implementation, cont. • What is done? • All the pieces described so far are working together • System works end-to-end, from tag scan to audio • What isn’t? • Mainly finishing things up and work on aesthetics • Glove and personal server aren’t currently something one would want to wear • Last minute hardware glitches

13. Evaluation • How well does it work? • Once the kinks are worked out, the system works pretty well. The key metrics: • Distance glove needs to be from tag • Length of time glove needs to be reading tag • Delay between successful scan and start of audio(i.e. processing time) • Length and relevance of audio clip

14. Evaluation, cont. – metrics • Distance glove needs to be from tag • Depends on how charged the batteries are, and how sensitive the antenna is • For our glove reader with a Mylar antenna in palm and full batteries, the distance is 3-4 inches. • Length of time glove needs to be reading tag • We have toyed with different reading frequencies to tune this metric • Currently check for a tag 4 times per second. At this speed, the reader can detect a tag when moved by it at an acceptable speed (i.e. as if you were feeling around the object)

15. Evaluation, cont. – metrics • Delay between successful scan and start of audio • This was our main concern, that doing the processing and database queries would take time • Actually, this time is minimal • From successful scan, audio will start in under a second • The longest portion of this is starting up the mp3 player, suggesting a possible point for improvement • Length and relevance of audio clip • Probably needs the most work • The overall clip is built on subclips. Currently output is: brand, name, price, any sale information • It would be best if this was configurable, because playing this information can take some time

16. Future Work • Lots to be done, lots of possibilities • Database contains a lot of information not currently used: UPC code, location, quantity • There should really be an audio output component always loaded, rather than making an external call • The product database could be a central store database accessed over wi-fi or similar • Could integrate a shopping list or store navigation aid • Would like to have a scanner at the checkout register that relates product name, price, subtotal as rung up • The glove we have is definitely a prototype: should have a lightweight, chargeable battery, and several buttons • Possible text-to-speech for audio generation, and linking with existing UPC-based product databases