The Traveler's Companion

1. The Traveler's Companion *This image is for illustration only

2. What is it ?Why would I want it ? The traveler’s companion is a simple compact digital camera that can translate foreign languages on signs and posters. • Park your car without worrying if this is a tow zone. • Find out what that “50% off” sale sign is referring to. • Read the names on the street and road signs.

3. For example: You find a great parking spot downtown, but you see this sign looming right above you... and you can’t read Hebrew or Arabic. Would you hesitate ?

4. Point - Click - Select and read With the Traveler’s companion, all you would need to do is… • Point the viewfinder at the sign, then Click. (just like an ordinary camera) • Select the area of interest. ...and then read the translation that appears on the back panel screen.

5. This is what you would see: BACK SENSITIVITY parking place regulated by parking meter or parking card from 08:00 until 18:00 the parking for three hours.

6. The User Interface • Image acquisition - works just like an automatic compact camera. • Selection - the user simply taps the area of interest on the screen. • Result display - printed in 4 lines on the LCD screen and has a scroll wheel placed immediately to its right.

7. A look under the hood- The hardware components: • A good autofocus/zoom system • A moderate-speed (~100 MHz) fixed-point DSP [e.g.TMS320VC549] • A cheap color CCD chip (~1 Mpixels) • EPROM (~2MB) containing the built in algorithms. • ROM card (~1MB) containing the dictionary (~5000 words), basic grammar rules and local fonts (~20 alphabets). • Large touch-LCD display (~4 sqr. inches) • Rugged rain-proof plastic/rubber casing.

8. Power consumption: • DSP - 800 mW • LCD panel - 250 mW Total: 1.2 W • CCD - 150 mW Power available from 2xAA battery pack: 7.2 WH Battery-life with continuous use: 6 hours Battery-life with anticipated use: over 1000 snapshots (assuming ~20 seconds per average snapshot)

9. The software components: 1. The border location algorithm - finds “keystone” areas in the image, given a manually selected sensitivity. 2. Sign parsing - divides the user selected area into individual lines, words and letters. 3. OCR/spellchecker- performs a simple OCR of the text, with wildcard dictionary lookup.

10. A look under the hood- 1. The border location algorithm: raise the contrast edge detection #1 “keystone” filter #2

11. A look under the hood- 2. Sign Parsing: The run-lengths minimization algorithm:

12. A look under the hood- 3. OCR/spellchecker : Parsing: ?קום חניה מוסרר באמצעו? OCR: מקום חניה מוסדר באמצעות Spellchecker: parking place regulated by Translation*: * unrecognized words will be transliterated

13. DSP operation count breakdown: • Boarder location- contrast [O(N)]  10^7 ops. edge detection [O(N)]  10^7 ops. “keystone” filter [O(sqrt(N))]  10^5 ops. Total: ~0.2 sec • Sign parsing- horizontal scanning [O(N)]  10^7 ops. vertical sub-scanning [O(N)]  10^7 ops. Total: ~0.2 sec

14. DSP operation count breakdown: • OCR/spellchecker- OCR: [#alphabets] x [alphabet len.] x [font size] x [offset alignments] = = 20 x 26 x 16 x 64 = 5 x 10^5 ops./character lookup: [#words in sign] x [sign word len.] x [dic. size] x [dic. word len.] = = 20 x 4 x 5000 x 5 = 2 x 10^6 ops. spell check: [#lookup ops.] x [error fraction] x [best-guess retries] = = 10^7 x 0.15 x 6 = 10^7 ops. Total: ~0.7 sec

15. Some other examples:

16. Images with problematic borders: Non-keystone Multiple overlapping signs False borders

17. Images with problematic text: Non-horizontal text Hand writing

18. Possible implementation approaches: • The shortcut:Software within an existing PDA, with a camera added-on. • Collaboration:Add software to an existing digital camera, using its DSP. • Independent: Fabricate the whole product, including dedicated optics.