1 / 30

sensors in mobile devices for capturing panoramas and environment maps

Maarten Van Lier 2 e Master Computerwetenschappen. sensors in mobile devices for capturing panoramas and environment maps. Use Case. Found an awesome view View too large for one picture Take pictures with smartphone app 360° left to right / full spherical view

finna
Download Presentation

sensors in mobile devices for capturing panoramas and environment maps

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Maarten Van Lier 2eMaster Computerwetenschappen sensors in mobile devicesforcapturingpanoramas and environment maps

  2. Use Case • Foundanawesome view • View toolargeforone picture • Takepictureswithsmartphoneapp • 360° left to right / full spherical view • App combines picturesinto panorama • Reasonablyfast

  3. ProblemDescription • Makepanoramas and environment maps • Findalignmentbetween image pairs • Combine pictures

  4. ProblemDescription (2) • Onsmartphone! • Processing power = low • Efficiency = necessary! • Interactivity • Help user takingpictures • View resultonsmartphone • Within a reasonable time

  5. Standard approach • Takepictures • Partially overlapping • Findneighboringpictures • Findalignmentbetweenneighbors • Intensitybasedalignment • Feature basedalignment • Compositepictures

  6. FindAlignment Pixel Based Feature Based • Findtransformation • Withlowestmisregistration • Withhighestintensity match • Minimizeerrorfunction • Search options • Full search • Hierarchical • Incremental • Detect features • Recognizablepoints • Caracterizedwith vector • SIFT, SURF, … • Match features • Findsamepoints in images • Usingdistancebetweenvectors • Findtransformation • Transforms features to corresponing features • RANSAC foroutliers • FindHomography

  7. Smartphone Sensors • Butsmartphones have sensors! • Accelerometer, compass, gyroscope • Determineorientation of device • Useaccelerometer & compass • Useorientation! • For estimated picture location • (and forreal time “preview”)

  8. The Overlap Approach • Takepictures • Guide user with 3D preview of estimated panorama • Save sensor data onshutter • Find overlapping regions • Usingsaved sensor data • Detect & extract features • From overlap regions • Match features • Betweencorrespondingoverlaps • Findalignment • Compositepictures

  9. TakingPictures

  10. TakingPictures

  11. TakingPictures

  12. TakingPictures

  13. Find Overlap Regions • Usingestimated picture locations • To findneighboringpictures • To determineestimated overlap • Boundingcirclearound picture center • Bounding box around overlap region • Axisalignedvs non axisaligned

  14. Find Overlap Regions (2)

  15. Find Overlap Regions (3)

  16. Find Overlap Regions (4)

  17. Find Overlap Regions (5)

  18. Find Overlap Regions (6)

  19. Detect & Extract Features • Onlyfrom overlapping regions of image • Large overlap • Many features => goodalignment • Expensivedetection & extraction • Small overlap • Fewer features => bad alignment • Cheaperdetection & extraction • For actualpanoramas & envmaps • Expected: large overlap regions • Actual speed gainmaynotbeverylarge

  20. Match Features • Neighboringpictures • Usingestimatedlocations (sensor data) • Features in same overlap region • Features estimated to be close to eachother • Useestimated 3D orpolar feature locations • Butnotyetimplemented

  21. Match Features Likely match Possiblefalse match Unlikely match Likely match

  22. Overlap Regions Test

  23. Overlap Regions Test (2)

  24. Overlap Region Test (3) • Recommendedfor most panorama apps: • About 20% oneachside=> about 70-80% whenon all sides • Butonlyanestimate=> needs to beinvestigated & tested! • Here: on average 62% overlap • Because overlap at (nearly) all sides • Expected: a 30-40% drop in time whenextracting features onlyfrom overlap regions

  25. Results: Timings (6 pictures) Standard Approach Overlap Approach • Find features • 2383 msfor 4998 features • Matching features & calculatehomographies • 18661 ms • Total: 21044 ms • Calculate Overlap Regions • 9 msfor 12 overlaps • Find Features • 1617 msfor 3153 features • Matching features & calculatehomographies • 8252 ms • Total: 9878 => 53% less!

  26. Results Standard Approach

  27. Results Overlap Approach

  28. So… What’sNext? • Improvestitchedresult • Increase overlap region • Paper! • Second semester: • Optimizations • Port to smartphone • Check timingsonsmartphone • More testing & results • Trysomethingsimilarfor pixel basedalignment • Initiallocationusing sensors + incrementalrefinement • And maybe HDR • Writing thesis text, making poster

  29. References • Image Alingment and Stitching: A Tutorial • (Richard Szeliski, 2006) • SURF: Speeded Up Robust Features • (HerbertBay, TineTuytelaars, Luc Van Gool, 2006) • RecognisingPanoramas • (M. Brown, D. G. Lowe, 2003) • BoofCV • http://boofcv.org

  30. Questions and suggestions?

More Related