Demosaicking for Multispectral Filter Array (MSFA)

1. Demosaicking for Multispectral Filter Array (MSFA) Lidan Miao AICIP Research August 24, 2004

2. Scene Camera Len MSFA Sensor Multispectral Image Reconstruction Mosaic Image Background • Challenge in acquisition of multispectral images. The CyberEye 2100is developed in SCP Laboratory at Syracuse University. A synchronized filter wheel and optical system provides 12 independent spectral bands from 400-1000 nm . http://www.cs.ucf.edu/~jlee/www-scp/html/real-time_multispectral_camera.html • Multispectral filter array technique. • System Structure http://www.dpreview.com/learn/?/Glossary/ Camera_System/Color_Filter_Array_01.htm

3. Terminology • MSFA samples • Mosaic Image • Demosaicking & Interpolation • Reconstructed image & Demosaicked image

4. CFA Demosaicking Review • Bilinear Interpolation • Constant Hue-based Interpolation • Hue is defined as the ratio or difference of red and green, blue and green (i.e. R-G,B-G or R/G, B/G) • The HVS is more sensitive to hue artifacts than luminance errors. • Gradient Based Interpolation • Interpolate along the edges instead of across them. • Weighted sum Interpolation

5. MSFA Review • Design requirement • Probability of appearance(POA) • Spectral consistency • Uniform distribution • A generic MSFA generation algorithm • Based on the binary tree decomposition. • Input: the number of spectral band and the probability of each band.

6. Decomposition 1 1 2 2 1 2 1 1 2 2 1 1 2 2 1 1 2 2 Subsampling 3 4 5 6 5 5 3 3 4 4 6 6 7 8 5 5 3 3 4 4 6 6 Decomposition 7 8 7 8 3 7 3 8 6 4 6 4 3 8 3 7 6 4 6 4 MSFA Review Resulted MSFA

7. Spectral Correlation • Why?? • Details are well preserved in spectral bands with more MSFA samples. • Different spectral bands possess similar edge information. • Validation metric: image similarity.

8. Spectral correlation:example Similarity between different spectral bands

9. Progressive Demosaicking Binary tree MSFA • Band selection • Pixel selection • Interpolation Pixel distribution of each band

10. Band Selection • Determine the order of selecting different spectral bands for interpolation. • Band selection is associated with the tree level of each leaf, i.e. the probability of appearance. • The spectral band with the highest POA is interpolated first.

11. Pixel Selection • Binary tree traversal from leaf nodes • Example: reconstruct spectral band “C”

12. p3 p1 e p2 p4 Interpolation • Gradient Based Interpolation • Pros. vs. Cons. • Adaptive edge-sensing Interpolation: weighted sum of neighboring pixels. p1 p3 e p4 p2

13. Edge-sensing Interpolation • Edge-likelihood in each direction of neighboring pixels. • Sobel edge detector and the second order derivative. • Weight estimator • Estimation of target pixel value

14. Case studies POA=1/2 POA=1/8 POA=1/4

15. Experiments-Data Description • Real Multispectral Data • 92AV3C9(9 bands) • FLC1(12 bands) • TIPJUL(7 bands) • Synthetic Multispectral Data • Eight objects

16. Validation • Original multispectral images are sampled using MSFA to generate mosaic images. • Reconstruct the mosaic images using • proposed method with and without binary tree • bilinear interpolation with and without binary tree. • Compare between the original multispectral images and the reconstructed images. • Evaluation metric: subject visual comparison, objective RMSE, classification accuracy.

17. Visual Comparison Proposed with binary tree Bilinear with binary tree Bilinear without binary tree Proposed without binary tree Original

18. Visual Comparison Original Bilinear with and without tree Proposed with and without tree

19. Experiments-RMSE • Root Mean Square Error where M and N are the size of image, and is the original image, represents the reconstructed image. The smaller the RMSE, the better the reconstructed image.

20. Experiments-RMSE Comparison Real multispectral data set Synthetic multispectral data set

21. Experiments-Pixel Classification • Real multispectral data • Each scene contains different classes. Choose the region we know the ground truth, from which we extract training and testing samples. • Knn classifier. • Synthetic multispectral data • Extract 20 training samples from each targets, all the targets pixels are testing samples. • Knn classifier

22. Pixel classification accuracy Real multispectral data set Synthetic multispectral data set

23. Experiments-Color Map Pixel classification of each targets, different colors represent different classification results Reconstructed Original Reconstructed Original Original Reconstructed

24. Experiments-Object classification • Compare the spatial and spectral features. • Spatial: hu-moments (7D) • Spectral: average of target pixels (7D) • Study the effect of viewing distance. A schematic representation for extraction of training and testing set

25. Object classification accuracy Original Bilinear Proposed

26. Conclusion • The proposed progressive method outperforms non-progressive and bilinear interpolation. • The classification accuracy using reconstructed images is comparable with the one generated using original multispectral images. • The MSFA technique is a feasible solution for multispectral cameras.

27. Thanks ?