Reinventing Printed Document Keeping digital as digital with uPen

1. Reinventing Printed DocumentKeeping digital as digital with uPen Jian Wang Senior Researcher Microsoft Research Asia July 19, 2005

2. Outline • Challenges and Approach • Technologies • Embedded Interaction Code(EIC) system • Computer vision system • HW/SW architecture • Hardware prototype design • EIC SDK: system design and APIs • Demos

3. Challenges • ``non-digital way of doing things'' • like paper and pencils • PCs still create non-digital document • Such as printed document • Keyboard/mouse confine people to desktop • Mobile interaction device

4. Goal • Digital original • Keep digital as digital • Makes every printed document digital • Brings digital computing to places where things are still being done in non-digital ways

5. Approach: EIC System • High data-embedding capacity to provide a gigantic address space up to 2192 bits. • A family of EIC visual patterns to adapt to various hardware implementations, applications and surfaces, and to meet the needs for usability and readability. • Multiple data-embedding channels to embed x and y coordinates, and both global (such as document ID) and local metadata in a digital document.

6. 0 1 EIC: Absolute X, Y Coordinates 0 1 0 1 1 0 0 0 1 1 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 EIC Pattern

7. Window Property of m-sequence A pseudo-random 1D array of 0’s and 1’s with “window” property An example with primitive polynomial of order 6: h(x)=x6+x+1, code length = 26 - 1=63 000001000011000101001111010001110010010110111011001101010111111 The window property ensures the unique decodability

8. 000001000011000101001111010001110010010110111011001101010111111000001000011000101001111010001110010010110111011001101010111111 m1 = 9 0 0 0 1 0 0 1 0 1 0 0 1 1 1 1 1 0 1 1 0 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 1 0 1 0 1 0 1 1 m2=7 0 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 0 Construction of m-array • Create a 2D array with size of m1*m2, where gcd(m1,m2) =1 • Folding the sequence • The 2D array retains the window property. If L is the bit location in m-sequence, then its location in m-array: • x= L mod m1 • y= L mod m2 0 0

9. Data-embedding

10. Bounding box EIC pattern Action Non-decodable area Decodable area EIC + Encoded content Biding EIC with Document

11. Hardware Architecture • Mechanical • Size: 14x17x160mm • Ink cartage • Pressure sensor • Optical • Camera • Lens • IR LED • Electrical • Processor/Memory • Bluetooth • Battery

12. Computer Vision System • To obtain high resolution (> 400 dpi), accurate x and y positions and metadata on printed digital documents (papers). • To obtain actual x and y position of the pen tip regardless of pen posture (tilting and rotating). Calibration error < 0.15mm. • To obtain sufficient spatial and temporal resolution (> 400 dpi x 120 Hz) of uPen digital ink (so that it’s compatible with the resolution of Tablet PC ink).

13. adaptive threshold of document content at (i, j) adaptive threshold of EIC pattern at (i, j) Document/EIC pattern segmentation T: document content A: EIC pattern Z: background area

14. ì n n j i å å = - Q ( | X Y | ) / N ï 0 , , 0 i j i j ï = = 1 1 i j ï n n j i å å ì + + j 1 = - i 1 Q ( | X Y | ) / N ï å å = SB ( min( X , Y )) / n ï + 1 , , 1 1 i j i j ï i , j l , k l , k ï = = 1 1 i j = - = - l i 1 k j 1 í í + + j 1 i 1 n n å å ï j ï i = å å SW ( max( X , Y )) / n = - Q ( | X Y | ) / N i , j l , k l , k ï î ï + + 2 1 , , 1 2 i j i j = - = - l i 1 k j 1 = = 1 1 i j ï n n j i ï å å = - Q ( | X Y | ) / N + ï 3 1 , , 3 i j i j î = = 1 1 i j n k å = - * Î = - * < d 2 J ( y x b ) , i S { j || y x b | } k i i k k k j j i - k k k 1 = i 1 k ( ) - 1 = ' ' b X X X Y k k k k k EIC Pattern Processing • Determine EIC pattern grid • Determine orientation of pattern • Determine the value of each bits Bits in camera FOV:  98.5bits in average Extracted bits:  93% in average

15. : transformation matrix from camera coordinate to document image coordinate :search radius. :Maximum grid alignment error. Efficient Image Matching

16. EIC SDK Architecture uPen prototype/third-party hardware Applications EIC SDK EIC Pattern Metadata EIC Document EIC Embedded SDK .Net Compact Framework uPen Firmware

17. Demos

18. Thank you