Devices Synchronization for modeling « 3D plane »

1. DevicesSynchronization for modeling « 3D plane » Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

2. Summary • Introduction • Context • ProblemModeling • DevicesSynchronization • Computer vision • Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

3. Introduction • More and more augmented reality applications • Development of more performant technologies • Best performances • Reliabilitytools for users Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

4. Context • Smartphones • More than 100 applications • 2 knowntechniques • Geolocalization and camera • Camera, gyro, accelerometer • Embedded systemsconstraints • « EyeTape » • prototype • Head up display • Display information superimposed on vision Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

5. Context • The project • Based on a CCD camera • Taking frames in video • 2 kinds of information to • Static (buildings, urbanenvironment…) • Dynamic (people, cars…) Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

6. Context • UbiquitousComputing • Mobile systemsaroundusers • Interactswith the environment • Devicesused in parallel • Mobile system • Videotreatment • 2 informations flow • Embedded systemsproblematic • Light mobile system • Light CPU power • Memory constraints Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

7. Context • Smartphone (iPhone 4) composed of • CCD 5mPixels (video 720p à 30fps) • Gyro L3G4200D Digital 3-axis • Accelerometer ST Micro LIS331DLH 3-axis • CPUApple A4 APL0398 (ARM Cortex A8) • DMA Memory • RAM  512Mo DRAM Samsung Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

8. Context • Devices synchronisation Apple iOS • Apple iPhone OS • Based on a BSD Kernel • Mac OS X withcellphone services Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

9. Context • Video frames overlap • Video’sframeratepreset • Purpose Reduce or increaseframerate • Original  30fps • Smartphone motion in space • Slow motion : video frames canoverlapsignificantly • need for suppressinguselessones • Need for fusioningdevices datas • Allowselection in video frames Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

10. Context • 2 knownmethods • Hardware • Can exist if it’sbuilt-in • Smartphones dont' have thischaracteristic • Software • Can beimplemented in the OS • Require an personalizedalgorithm Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

11. Summary • Introduction • Context • ProblemModeling • DevicesSynchronization • Computer vision • Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

12. Problemmodeling • Eachdeviceshave different time delays • In retrieving datas • In transmitting datas • Requiremodelingeachdelaysrepresenting the time between the datas are required and the time wherethey are in memory Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

13. Device Controller gyro Gyroscope Device Controller accéléromètre Accéléromètre Buffer mémoire DMA Capteur CCD Device Controller CCD Problemmodeling • Basic system architecture Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

14. Problemmodeling • Accelerometer • 3 datas sendinto 1 block • 3 axis accelerationsensor Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

15. Problemmodeling Data Bus Adresses Bus Control Bus CPU Device Controller Mémoire Accéléromètre Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

16. Problemmodeling • Accelerometer • Retrieving datas delay • Devices bus output: fbus • Datas volume: Vdata • Datas acquisition delay: ∆tacqacc • Bus acquisition delay (if busy): delay • Writing in memorydelaydepending on datas volume : ∆twrite • ∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

17. Problemmodeling • Gyro • 3 datas sendinto 1 block • 3 axis rotations Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

18. Problemmodeling Data Bus Adresses Bus Control Bus CPU Device Controller Mémoire Gyroscope Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

19. Problemmodeling • Gyro • Retrieving datas delay • Devices bus output: fbus • Datas volume: Vdata • Datas acquisition delay: ∆tacqacc • Bus acquisition delay (if busy): delay • Writing in memorydelaydepending on datas volume : ∆twrite • ∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

20. Problemmodeling • CCD Camera • Treatment of frames beforethey are placed in memory • sRGBcolors • 720p • Red/ Green/ Blue • Heavy data volume • DMA Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

21. Problemmodeling Data Bus Adresses Bus Control Bus CPU DMA Device Controller Mémoire CCD Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

22. Problemmodeling • CCD Camera • Retrieving datas delay • Devices bus output: fbus • Datas volume: Vdata • Datas acquisition delay: ∆tacqacc • Bus acquisition delay (if busy): delay • Writing in memorydelaydepending on datas volume : ∆twrite • ∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

23. Summary • Introduction • Context • ProblemModeling • DevicesSynchronization • Computer vision • Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

24. Devicessynchronization Recordingat 30fps T0 Start memory scan window Windows of N scan in memory for finding frame T1 ∆t Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

25. Devicessynchronization ∆t Frame Spottedatti • ti : date of scan • ∆t : retrieveing image delay • ti - ∆t = real date of the frame capture Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

26. Devicessynchronization • In thiswindow • If a frame isspottedat pi • S∆ = pi – (N - 1) / 2 • Nextwindowstartat • T1 + S∆ • Windowplacedevery time nearthe last image wasspotted • Real time and embeddedconstraintscontext Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

27. Devicessynchronization • When the windowisstarted, creation of coordinate table • Accelerometer • ta[] = tspotted - ∆tacc • Gyro • tg[] = tspotted - ∆tgyr Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

28. Devicessynchronization • Coordinate are chosed • tCoordonnees = tImgSpotted - ∆tImage • Wetry to keep the nearestcoordinate Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

29. Summary • Introduction • Context • ProblemModeling • DevicesSynchronization • Computer vision • Conclusion Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

30. Computer Vision • A frame = spatial coordinates • Smartphone’s spatial localization • Spatial localization= frame plane • Creates frame 3D plane Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

31. Computer Vision Rémi Devinant DII5 / Devices synchronization for modeling 3D plane

32. Computer Vision • Modeling 3D plane Rémi Devinant DII5 / Devices synchronization for modeling 3D plane