Image Inpainting Using Pre-trained Classification CNN

1. Image Inpainting Using Pre-trained Classification CNN By - Yaniv Kerzhner & Adar Elad Supervisor - Yaniv Romano

2. Achievements ! We wrote an academic paper which we will soon send to a conference.

3. Non-Blind Image Inpainting • The process of reconstructing lost parts of a given image. • A platform for interesting applications: Removal of undesired objects, replace a face with another, super resolution is a special case, compression, and more… ?

4. Project Scope • Goal: Solving the inpainting problem using Convolutional Neural Networks (CNN) • Starting point: Handwritten digits • Later: Facial images • Background on CNN • Define an architecture: • Convolution – ReLU – … – Fully-connected – Soft-max. • Minimize a loss function (example, label) using backpropagation and gradient descent. • We use MatConvNet environment • Our approach to tackle the inpainting problem.

5. Neural Networks (CNN) • We take an image, pass it through a series of convolutional, nonlinear, pooling (down-sampling), and fully connected layers, and get an output. That output can be a single class or a probability of classes that best describes the image. CNN for digit classification • The process of learning the parameters is called backpropagation. • Feed an image to the network, and compare the output of the desired class, and pay a Loss in case of error. • Update the parameters by minimizing the Loss. Requires access the gradient of the loss as function of the parameters.

6. abel Back-Prop (CNN) Feed Forward: X X’s classification 20% 012..9 60% 40% 70%

7. Back-Prop (CNN) label Expected classification Creating the Loss Taking the derivative of the penalty 0% 012.. 9 0% 100% 0% X’s classification CNN(X) 20% 012..9 60% 20% 01 2.. 9 60% 40% 90% 40% 90%

8. Background: Inpainting • The straightforward solution • Given a set of training pairs (original and corrupted images), train a network in a supervised fashion. CNN ℒ , [J. Xie et al. (‘12)] [K. Rolf et al. (‘14)] …

9. Project Objectives • The straightforward solution • Given a set of training pairs (original and corrupted images), train a network in a supervised fashion. In contrast, Our project offers an alternative way for inpainting • Can we leverage a network that was trained for classification to solve the in-painting problem? • We offer a novel approach to hallucinate missing data in images.

10. Block Diagram Pre-trained Network (CNN) MinX E(Y,CNN(X),label) M What is E(•) that will lead to the hoped result?

11. Defining the Cost Function The cost is a linear combination of several penalties, each of them represent a different force, which would lead to high quality restoration Its Importance Smooth Should be recognizable

12. The Penalties (Mathematically) Should be classified correctly: Smoothness (artifacts removal): TV Mask Horizontal Derivative Vertical Derivative

13. Minimizing the Cost • The gradient of E is the sum of the gradients of each penalty. • Psmooth is known and commonly used term. • The interesting penalty is Plabel, computing its gradient is similar to back-propagation!

14. Calculating P_label Calculating Plabel Feed Forward X’s classification X 20% 012..9 60% 40% 70%

15. Minimizing the Cost label Expected classification Creating the penalty Taking the derivative of the penalty 0% 0..5. 9 0% 100% 0% X’s classification CNN(X) 20% 012..9 60% 20% 01 2.. 9 60% 40% 90% 40% 90% × × ×

16. Minimizing the Cost label Expected classification Creating the penalty Taking the derivative of the penalty 0% 0..5. 9 0% 100% 0% CNN(X) x’s classification 20% 012..9 60% 10% 0..5. 9 7% 40% 90% 15% Better classification

17. Initialization (1) • Since we use the gradient descent method, we face an uncertainty in the minimum we reach. • In order to improve our chances of converging to a meaningful minimum that relates to our inpainting task, we should initialize the missing parts wisely.

18. Initialization (2) • The initialization methods we have explored include: • Completing the missing parts with the image after enlargement and reduction. In this way we create a diffusion of the hole's boundaries into the missing parts (this initialization worked best for digit inpaintings). • Completing the missing parts with the average image of the desired classification. This gives the network a successful starting guess to start from Initialization strategies for the algorithm. The right-most is the corrupted image to inpaint. The left-most is initialization (2) and the right-most is initialization (1)

19. Failure Cases: Past • We noticed that for some images the algorithm does not successfully fill the hole. • The network doesn't memorize the small pieces of the image content. Meaning, the network isn’t able to add content to the corrupted regions because the network hasn’t assimilated the visual characteristics of each label.

20. Failure Cases: Old Solution • The Problem: We noticed that for some images the algorithm does not successfully fill the hole. Solution Instead of forcing the image to be classified by the network we force the image to be close to the representation of another image from the database from the same class [L. Gatys et al. (‘15)] [A. Vedaldi et al. (‘16)] …

21. Results – Post solution The first row displays the corrupted images and in the second row we can see the inpainting results. These examples are taken from ImageNet and MNIST datasets.

22. Past Suggestments • Those are the main ideas and comments we received after the mid-semester presentation: • Train the network on small parts of images. • Inpaint by all labels and take the smallest energy as the best inpainting. Those comments made us rethink in order to improve the results. We also reviewed literature and found similar elements between your recommendations and the works we read.

23. Related Work • Interesting work has been done recently for solving the inpainting problem using Neural Network approaches. Here are some of them: • NianCai, et al., "Blind inpainting using the fully convolutional neural network", Springer 2015 • J. Xie, L. Xu, and E. Chen. "Image denoising and inpainting with deep neural networks". In NIPS, 2012. • D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. Efros. Context encoders: Feature learning by inpainting. 2016 • Raymond A. Yeh, Chen Chen, T. Yian Lim, A. G. Schwing, M. Hasegawa-Johnson, M. N. Do. "Semantic Image Inpainting with Deep Generative Models". 2017

24. Results - Mnist dataset The images in the first row are the original images from the database. The images in the second row are the original images above with different types of holes. The images in the third row are the results of the inpainting.

25. Results - Yale B dataset The images in the top row are the original images from the database, in the second row are the corrupted images and in the most bottom row displays the inpainting results.

26. Special Cases (1) - Visualization In this section we present and discuss some special and interesting experiments involving our inpainting algorithm. These experiments do not involve changes in the network nor the algorithm correctness. • A specific case of image inpainting where all the data is lost, is called illusion.

27. Special Cases (2) – Inpainting by all labels • We tried to inpaint a certain image as all labels and at the end we defined the best inpainting result to be the one which led to the lowest energy function. The most-right image is the corrupted image and the others are the best inpainting results To our surprise, when we performed this experiment, we received the best inpainting (according to the criterion we defined) for a different label than the original one (as shown in the figure)

28. Special Cases (3) - illumination • We noticed during the experiments with the Yale Extended B database that there are many images taken in the dark, the question then arises: could these images be illuminated with the assistant of our algorithm? We discovered that it is possible by defining the missing parts of the image as all the pixels in the image that are below a specific threshold. It is possible to distinguish that all the details of the original image still remains after the inpainting. Meaning, all we did is to add information to the dark parts.

29. Conclusion Our solution is not based on a network that has been taught to inpaint. Instead we present the concept of using a classification-oriented network for solving a completely different problem based on the data it has learned.

30. An alternative - modify Plabel • Instead of forcing the image to be classified by the network we force the image to be close to the representation of another image from the database from the same class Layer 8 Layer n

31. Result – Pfeature After Old Algorithm Holed image Reference Image After Algorithm Reference representation