Digitalization of Cultural Artifacts using Structured Lighting

1. Digitalization of Cultural Artifacts using Structured Lighting H.J. Chien 簡祥任 C.Y. Chen 陳佳妍 C.F. Chen 陳基發 Y.M. Su 蘇義明

2. Outline • Introduction • Structured Lighting • Calibration • Depth measuring • Stereo Correspondence • Dynamic Thresholding • Sub-stripe Resolution • Reconstructed Result • Conclusion and Future Work Vision and Graphics Lab, Dept. CSIE, NUK

3. Introduction • Heritage Preserve & Computer Vision • Digitalized shape or structure of cultural objects from 2D images • Reconstructed results are not relevant to modeling skill • comparing to CAD approaches • Automatic procedure • Fast acquisition • Highly detailed surface • Non-tactile, non-destructive scanning Vision and Graphics Lab, Dept. CSIE, NUK

4. Introduction • Related work • Stanford Digital MichelangeloProject (1999) • Statue Maddalena • Bronze Minerva of Arezzo • Castles in Northern Italy • Old city of Xanthi in Greek • Great Buddhas in Japan • … Bronze Minerva of Arezzo C. Rocchini, et al, “A Low Cost 3D Scanner based on Structured Light” Proc. Eurographics 2001, vol. 20(3), pp. 299-308, 2001. Stanford Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/ Vision and Graphics Lab, Dept. CSIE, NUK

5. Structured Lighting • Stem from passive stereo vision • One of paired cameras is replaced by a projector • Actively solve correspondence problem • Able to obtain dense depthmap on featureless surface • Challenges • How to calibrate a projector • How to establish correspondence • How to extract light patterns • Heterogeneity of image resolutions • e.g. 1600x1200 (camera) vs. 800x600 (projector) Vision and Graphics Lab, Dept. CSIE, NUK

6. Structured Lighting • Overall Strategy • Lens Calibration • Camera • Projector • Correspondence • Sequential scanning • Simple patterns • Coded patterns • Triangulation Vision and Graphics Lab, Dept. CSIE, NUK

7. Calibration - Camera • Tsai’s method • One-frame calibration • Need object with known geometry as reference • Construct linear equations using perspective projection model • Solve linear system • Refine result by non-linear optimization (aka bundle adjustment) • e.g. Gradient descent, Levenberg-Marquardt Vision and Graphics Lab, Dept. CSIE, NUK

8. Calibration - Projector • Treat projector as an inversed pinhole camera • Reconstruct projector view • from camera view and camera-projector correspondence • Apply Tsai’s method Projector View (partial) Camera View Vision and Graphics Lab, Dept. CSIE, NUK

9. Depth Measuring • Triangulation technique • Find intersection of the back-projected ray through camera and the corresponding projection plane Vision and Graphics Lab, Dept. CSIE, NUK

10. Depth Measuring • Visualized example Vision and Graphics Lab, Dept. CSIE, NUK

11. Stereo Correspondence • Plane-to-plane mapping • State how two image planes are related • Key prerequisite to reconstruction techniques based on multiple view geometry Vision and Graphics Lab, Dept. CSIE, NUK

12. Stereo Correspondence • Gray-coded light patterns • Time-multiplexed coding strategy • Use binary Gray codes to represent 1D coordinate of each unit stripe on projector’s image plane • N projections identify2n unit stripes Vision and Graphics Lab, Dept. CSIE, NUK

13. Binarization • Intensity-ratio thresholding • Store per-pixel intensity profile • Compute intensity ratio • Binarize each pixel base onits ratio • Invariant lightingcondition? Vision and Graphics Lab, Dept. CSIE, NUK

14. Observation Vision and Graphics Lab, Dept. CSIE, NUK

15. Observation Decision Boundary Pattern # Vision and Graphics Lab, Dept. CSIE, NUK

16. Dynamic Thresholding • Albedo-based thresholding fails • when the lighting condition of a pixel is away from the reference images • Need to update thresholds • Use an online updating mechanism • Probabilistic kernel-based dichotomizer weighting coefficient kernel function Vision and Graphics Lab, Dept. CSIE, NUK

17. Per-Pixel Classification K = Gaussian Kernel (σ: 0.1) Vision and Graphics Lab, Dept. CSIE, NUK

18. Fixed vs. Dynamic Thresholding • Result on a problematic pixel Pattern # Vision and Graphics Lab, Dept. CSIE, NUK

19. Sub-stripe Resolution • Information lost in camera-projector correspondence • due to… • heterogeneous resolution, improper placement, wrongly decoded index, noises, etc. • Cause measurement errors • Estimate missing mappings • By interpolation • Assign new values to unsureelements • Reference reliable neighbors Vision and Graphics Lab, Dept. CSIE, NUK

20. Sub-stripe Resolution • How to tell reliability of a pixel • Take certainty of classification into account • Defined as sum of square of P(Ri=1~k) for each pixel • Adjustment on correspondence • F: decoded correspondence • C: certainty of classification • D: function reflects spatial proximity • G: adjusted correspondence Vision and Graphics Lab, Dept. CSIE, NUK

21. Sub-stripe Resolution • Result CpX9 : X-coordinate correspondence established usingnine Gray-coded light patterns Z9 : Depth computed using CpX9, saw-toothed effects are significant Zconv : Improved result using densified correspondence Zspline : Improved result using spline interpolation Vision and Graphics Lab, Dept. CSIE, NUK

22. Reconstructed Result • Chosen artifact 布偶如來 • Has various surfaces in different material • Non-uniform reflectance • Specular and diffuse componentsare present • Hardware & Configuration Vision and Graphics Lab, Dept. CSIE, NUK

23. Reconstructed Result • Original surface Vision and Graphics Lab, Dept. CSIE, NUK

24. Reconstructed Result • Improved surface Vision and Graphics Lab, Dept. CSIE, NUK

25. Reconstructed Result • Improved surface (textured) Vision and Graphics Lab, Dept. CSIE, NUK

26. Conclusion & Future Work Vision and Graphics Lab, Dept. CSIE, NUK

27. Showcase http://ndap.csie.nuk.edu.tw Vision and Graphics Lab, Dept. CSIE, NUK

28. Vision and Graphics Lab, Dept. CSIE, NUK