270 likes | 447 Views
Image Repairing: Robust Image Synthesis by Adaptive N D Tensor Voting. IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Jiaya Jia, Chi-Keung Tang Computer Science Department The Hong Kong University of Science and Technology. Motivation.
E N D
Image Repairing: Robust Image Synthesis by Adaptive ND Tensor Voting IEEE Computer Society Conference on Computer Vision and Pattern Recognition Jiaya Jia, Chi-Keung Tang Computer Science Department The Hong Kong University of Science and Technology
Motivation • Main difficulties to repair a severely damaged image of natural scene • Mixture of texture and colors • Inhomogeneity of patterns • Regular object shapes
Motivation • Given as few as one image without additional knowledge, we address: • How much color and shape information in the existing part is needed to seamlessly fill the hole? • How good can we achieve in order to reduce possible visual artifact when the information available is not sufficient. • Robust Tensor Voting method is adopted
Tensor Voting Review • Tensors: compact representation of information • Tensor encoding: Ball tensor: uncertainty in all directions Stick tensor: certainty along two opposite directions 3D tensor Plate tensor: certainty of directions in a plate
Tensor Voting Review • Voting process is to propagate local information Osculating circle P
Image repairing system Complete Segmentation Input Damaged Image Texture-based Segmentation Curve Connection Statistical Region Merging Adaptive Scale Selection ND Tensor Voting Output Repaired Image Image synthesis
Segmentation • JSEG [Deng and Manjunath 2001] • color quantization • spatial segmentation • Mean shift [Comanicu and Meer 2002] • Deterministic Annealing Framework [Hofmann et al 1998]
Statistical Region Merge • (M + 1)D intensity vector for each region Pi, where M is the maximum color depth in the whole image. if histogram gradient
Why Region Merge? • Decrease the complexity of region topology • Relate separate regions P1 P5 P2 Damaged area P3 P4
Z P1 P5 P2 P3 P4 X Curve Connection • 2D tensor voting method P2 P4
Why Tensor Voting? • The parameter of the voting field can be used to control the smoothness of the resulting curve. • Adaptive to various hole shapes Without hole constraint Small Scale Large Scale With hole constraint
Connection Sequence • Topology of surrounding area of the hole can be very complex • Greedy algorithm • Always connect the most similar regions P1 P2 andP4 P5 P3 andP5 P2 P1 Damaged area P3 P4
Image repairing system Complete Segmentation Input Damaged Image Texture-based Segmentation Curve Connection Statistical Region Merging Adaptive Scale Selection ND Tensor Voting Output Repaired Image Image synthesis
ND Tensor Voting • Tensor encoding • Each pixel is encoded as a ND stick tensor 5 5 Scale N=26 Stick tensor
ND Tensor Voting • Voting process in ND space • An osculating circle becomes an osculating hypersphere. • ND stick voting field is uniform sampling of normal directions in the ND space. sample sample
Adaptive Scaling • texture inhomogeneity in images gives difficulty to assign only one global scale N[Lindeberg et al 1996]. • For each pixel iin images, we calculate: • trace(M) measures the average strength of the square of the gradient magnitude in the window of size Ni
Adaptive Scaling • For each sample seed: • Increase its scale Nifrom the lower bound to the upper bound • If trace( ) < trace() - αwhere αis a threshold to avoid small perturbation or noise interference, set Ni - 1 → Niand return • Otherwise, continue the loop until maxima or upper bound is reached
Limitations • Lack of samples. • Meaningful and semi-regular objects.
Conclusion • An automatic image repairing system. • Region partition and merging. • Curve connection by 2D tensor voting. • ND tensor voting based image synthesis. • Adaptive scale.