By Rupesh Shet , K.H.Lai Dr. Eran Edirisinghe

1. Use of Neural Networks in Automatic Caricature Generation:An Approach based on Drawing Style Capture By Rupesh Shet , K.H.Lai Dr. Eran Edirisinghe

2. Agenda 1. Introduction to ACCR 2. Cascade Correlation Neural Network 3. Capturing the Drawing Style of a Caricaturist 4. Conclusion 5. Questions

3. Automated Caricature Creation Observation: • The understanding between the mapping of input and output which is not necessary. • Able to capture the non-linear relationships Now to do this we suggested to Use Neural Network

4. What has changed? Input Images Corresponding caricature (Cartoon) ∆S’ What is the distinctive feature? ∆S Why make this change? (Rules in the brain) Mean Face Corresponding Caricature Mean Face Input Image Formulization of Idea • Exaggerating the Difference from the Mean (EDFM) which is widely accepted among caricaturists to be the driving factor behind caricature generation

5. Cascade Correlation Neural Network

6. Cascade Correlation Neural Network • Artificial neural networks are the combination of artificial neurons • After testing and analysing various neural networks we found that the CCNN is the best for the application domain under consideration. • The CCNN is a new architecture and is a generative, feed forward, supervised learning algorithm. • An artificial neural network (ANN) is composed neurons, connections between neurons and layer. • Connection weights determine an organizational topology for a network and allow neurons to send activation to each other.

7. NN Terminology There are three layer: • Input layer the problem being presented to the network. • Output layer the network’s response to the input problem. • Hidden layer perform essential intermediate computations. • Input function is a linear component which computes the weighted sum of the units input values. • Activation function is a non-linear component which transforms the weighted sum in to final output value

8. CCNN Algorithm • Cascade-Correlation (CC) combines two ideas: • The first is the cascade architecture, in which hidden neuron are added only one at a time and do not change after they have been added. • The second is the learning algorithm, which creates and installs the new hidden neurons. For each new hidden neuron, the algorithm tries to maximize the magnitude of the correlation between the new neuron’s output and the residual error signal of the network.

9. CCNN Algorithm • The algorithm is realized in the following way: • CC starts with a minimal network consisting only of an input and an output layer but with no hidden neuron. Both layers are fully connected with an adjustable weight and bias input is permanently set to +1. • Train all the connections ending at an output neuron with a usual learning algorithm until the error of the network no longer decreases.

10. Trained Connection Frozen Connections Output Unit Activation Function Initial State No Hidden Units Input unit I II III IV V +1 Bias CCNN Algorithm

11. CCNN Algorithm • Generate the so-called candidate neurons. Every candidate neurons is connected with all input neuron and with all existing hidden neuron. Between the pool of candidate neuron and the output neuron there are no weights. Output Unit Add Hidden Unit 1 Input unit I II III IV V +1 bias

12. CCNN Algorithm • Try to maximize the correlation between the activation of the candidate neuron and the residual error of the net by training all the links leading to a candidate neuron. Learning takes place with an ordinary learning algorithm. The training is stopped when the correlation scores no longer improves. • Choose the candidate neuron with the maximum correlation freeze its incoming weights and add it to the network. To change the candidate neuron into a hidden neuron, generate links between the selected neuron and all the output neuron. Since the weights leading to the new hidden neuron are frozen, a new permanent feature detector is obtained. • Loop back to step 2. • This algorithm is repeated until the overall error of the net fall below a given value

13. A trained NN with Cascade Correlation Algorithm

14. Advantages of CCNN • In addition the CCNN has several other advantages namely: • It learns very quickly and is at least ten times faster than traditional back-propagation algorithms. • The network determines its own size and topology and retains the structure. • It is useful for incremental learning in which new information is added to the already trained network.

15. The Proposed Methodology: To capture the drawing style of a caricaturist

16. Caricature Image Original Facial Image Mean Face Generator Facial Component Extractor/ Separator Caricature Component Mean Component Original Component Cascaded Correlation Neural Network (CCNN) Training New Facial Component Automatically Generated Caricature Component The Proposed Methodology

17. Step 1:Generating Mean Face Generating Mean Face: • For the purpose of our present research which is focused only on a proof of concept, the mean face (and thus the facial components) was hand drawn for experimental use and analysis. However, in a real system one could use one of the many excellent mean face generator programs made available in the World Wide Web

18. FacialSketch Face Lip Nose Eyes Step 2: Extraction Facial Component Extraction/Separation: • To extract/separate various significant facial components such as, ears, eyes, nose and mouth from the original, mean and caricature facial images • Many such algorithms and commercial software packages exists that could identify facial components from images/sketches.

19. Node direction 3 4 2 1 5 (mm) Caricature Image X7 6 8 Original Image 7 Mean Image Step 3: Creating Training Data Sets Creating Data Sets for Training the Neural Network: • The original, mean and caricature images of the component under consideration are overlapped • Subsequently using cross sectional lines centred at the above point and placed at equal angular separations.

20. Original Mean Caricature Original Mean Caricature Original X1 25 37 13 X5 86 75 100 Y1 99 99 99 Y5 99 99 99 X2 47 53 37 X6 62 60 68 Y2 108 102 118 Y6 93 95 87 X3 56.8 56.8 56.8 X7 56.8 56.8 56.8 Y3 109 102 125 Y7 92 95 86 X4 66 59 76 X8 50 52 45 Y4 109 102 119 Y8 93 95 87 Step 4: Tabulating Data Sets Tabulating Data Sets: • The higher the number of cross sectional lines that are used, the more accurate the captured shape would be. • However for ease of experimentation, we have only used four cross sectional lines, which results in eight data sets

21. Step 5,6: NN Training & Setting NN Neural Network Training: • we consider the data points obtained from the caricature image above to be the output training dataset of the neural network. • The data sets obtained from the original and mean images to formulate input training dataset of the neural network. [This follows the widely accepted strategy used by the human brain to analyse a given facial image in comparison to a known mean facial image. ] Setting up the Neural Network Parameters: • We propose the use of the following training parameters for a simple, fast and efficient training process.

22. Parameter Choice Neural Network Name Cascade Correlation Training Function Name Levenberg-marquardt Performance Validation Function Mean squared error Number of Layers 2 Hidden Layer Transfer Function Tan-sigmoid with one neuron at the start Output Layer Transfer Function Pure-linear with eight neurons Step 7: Testing Testing • Once training has been successfully concluded as described above, the relevant facial component of a new original image is sampled and fed as input to the trained neural network along with the matching data from the corresponding mean component.

23. Experiments and Analysis

24. Experiment : 1 This experiment is design to investigate the CCNN is capable for predicting orientation and direction.

25. Experiment: 2 This experiment is designed to prove that it is able to accurately predict exaggeration in addition to the qualities tested under experiment 1

26. Experiment: 3 (Training) This experiment we test CCNN on a more complicated shape depicting a mouth (encloses lower and upper lips). Training1 Training 2 Training 3 Training 5 Training 6 Training 7

27. Experiment: 3 (Results) The results demonstrate that the successful training of the CCNN has resulted in it’s ability to accurately predict exaggeration of non-linear nature in all directions. Result 1 Result 6 Result 7 Note:An increase in the amount of the training data set would result in an increase of the prediction accuracy for a new set of test data.

28. Conclusion • In this research we have identified an important shortcoming of existing automatic caricature generation systems in that their inability to identify and act upon the unique drawing style of a given artist. • We have proposed a CCNN based approach to identify the drawing style of an artist by training the neural network on unique non-linear deformations made by an artist when producing caricature of individual facial objects. • The trained neural network has been subsequently used successfully to generate the caricature of the facial component automatically.

29. Conclusion • The above research is a part of a more advanced research project that is looking at fully automatic, realistic, caricature generation of complete facial figures. One major challenge faced by this project includes, non-linearities and unpredictabilities of deformations introduced in exaggerations done between different objects within the same facial figure, by the same artist. We are currently extending the work of this paper in combination with artificial intelligence technology to find an effective solution to the above problem.

30. References [1]Susan e Brennan “Caricature generator: the dynamic exaggeration of faces by computer”, LEONARDO, Vol. 18, No. 3, pp170-178, 1985 [2]Z. Mo, J.P.Lewis, and U. Neumann, Improved Automatic Caricature by Feature Normalization and Exaggeration, SIGGRAPH 2003 Sketches and Applications, San Diego, CA: ACM SIGGRAPH, July (2003). [3]P.J. Benson, D.I. Perrett. “Synthesising Continuous-tone Caricatures”, Image & Vision Computing, Vol 9, 1991, pp123-129. [4]H. Koshimizu, M. Tominaga, T. Fujiwara, K. Murakami. “On Kansei Facial Caricaturing System PICASSO”, Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, 1999, pp294-299. [5]G. Rhodes, T. Tremewan. “Averageness, Exaggeration and Facial Attractiveness”, Psychological Science, Vol 7, pp105-110, 1996. [6]J.H. Langlois, L.A. Roggman, L. Mussleman. “What Is Average and What Is Not Average About Attractive Faces”, Psychological Science, Vol 5, pp214-220, 1994.

31. References [7]L. Redman. “How to Draw Caricatures”, McGraw-Hill Publishers, 1984. [8]“Neural Network” http://library.thinkquest.org/C007395/tqweb/index.html(Access date 13th Oct 2004) [9]The Maths Works Inc, User’s Guide version 4, Neural Network Toolbox, MATLAB [10]http://www.cs.cornell.edu/boom/2004sp/ProjectArch/AppoNeuralNet/ BNL/NeuralNetworkCascadeCorrelation.htm (Access date 13th Oct 2004) [11]S. E. Fahlman, C. Lebiere. “The Cascade-Correlation Learning Architecture”, Technical Report CMU-CS-90- 100, School of Computer Science, Carnegie Mellon University, 1990. [12]Carling, A. (1992). “Introducing Neural Networks”. Wilmslow, UK: Sigma Press. [13]Fausett, L. (1994). “Fundamentals of Neural Networks”. New York: Prentice Hall.

32. Question???