360 likes | 469 Views
LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera. Supervisor: Prof. Michael R.Lyu Group Member: Leung Man Kin, Stephen Ng Ying Kit, Tommy. Outline. Introduction Approach Stitching Algorithm Geometric Transformation Progress Conclusion Future Plan. Outline.
E N D
LYU0503Document Image Restorationon Mobile PhoneUsing Onboard Camera Supervisor: Prof. Michael R.Lyu Group Member: Leung Man Kin, Stephen Ng Ying Kit, Tommy
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Introduction • Use mobile phone to capture photo of document. • The document can be stored in the phone or sent to other people by several means.
Objective • Use camera to take photo of different parts of a document. • Use suitable algorithm to merge the photos back to a whole document. • Transmit the document by several means • e.g. Fax, MMS, etc.
Motivation • Most of phones nowadays have built-in camera. • Resolution of camera is not enough for document • The text on the document would be blurred • Symbian phone can provide several transmission function. E.g. Fax, MMS • Fax at any time!
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Approach • Take photos of different part of document • Use stitching algorithm to merge the photos into a photo • Stitching algorithm is our major research area of our project
Assumption • The document consists mainly of text. • The camera has macro mode. • Sharp Close Up.
Approach • Procedure of our design • Take a photo of full document first. • Then ask the user to take photos of different parts • Overlay a rough picture on the screen of the phone.
Approach A4 document
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Stitching Algorithm • Stitching needs to find corresponding points in two images • two different approaches • Direct alignment • Compare images pixel by pixel • e.g. • Feature-based registration • Use some methods to find feature points • Match features in two images
Stitching Algorithm • Feature-based registration is chosen. • Usually, direct alignment is less efficient. • Need to consider different scale, rotation, etc. • There are several methods to extract feature. • E.g. corner detector. • Scalable Invariant Feature Transform (SIFT) is used to extract the feature. • The features are invariant to image scale and rotation.
Stitching Algorithm • How to get the feature point? • Identifies key locations in scale space by looking for locations that are maxima or minima of a difference-of-Gaussian function.
Stitching Algorithm • Varying σ • Repeat the above process using subsampled image for several times • Decrease the resolution by 2.
Stitching Algorithm • Find the maxima or minima by comparing 26 neighbors. • They are the key feature points.
Stitching Algorithm • Property of the feature keypoints • Achieve partial invariance to local variations like affine or 3D projections. • Need to filter feature keypoints further • Reject keypoints that have low contrast.
Stitching Algorithm • How to store the feature? • For each keypoint, image gradient magnitudes and orientations are sampled to store the feature.
Stitching Algorithm • Compare the features in two images • Get the position to stitch • However, the photo taken may suffer from geometric distortion • e.g. Lens distortion, taking photo with different angles
Stitching Algorithm • Use the matched feature to calculate the transformation parameters. • Transform images to the correct orientation before merging
Stitching Algorithm SIFT Extract feature keypoint Matching keypoints in two images Find transformation parameter by matched keypoints Transform images using the parameter Merge all the images
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Lens Correction Transformation
Image1 Image2 Transformation parameters ai Transformation parameters bi Image3
Take several overlapping images of the document……
High Resolution Document Image Restored!
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Porting to Symbian Phone • On the PC, the stitching algorithm is implemented using C# and Matlab. • We should port the algorithm to the Symbian phone. • Symbian OS use C++ as the language.
Porting to Symbian Phone • Difficulty: • Their syntaxes are not the same. • e.g. set / get keyword in C# • Symbian and C# provide different API. • Math library, Array library. • Some of data types are different in Symbian C++.
Porting to Symbian Phone • Difficulty: • The processor on Symbian phone is not powerful. • The memory space on Symbian phone is not large. • We may need modify the algorithm so that it would suit the phone.
Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Conclusion • We have studied different stitching algorithms • We have implemented the stitching algorithm on PC • We are porting the stitching algorithm on Symbian phone
Outline • Introduction • Objective • Motivation • Approach • Stitching algorithm • Geometric Transformation • Progress • Conclusion • Future Plan
Future Plan • Implement the stitching on the Symbian phone. • Modify SIFT or design other algorithms for more efficient performance on mobile phone
Future Plan • Improve the algorithm of merging. • Study super-resolution to make text clearer.