Goodfellow: Chap 1 Introduction

1. Goodfellow: Chap 1 Introduction Dr. Charles Tappert The information here, although greatly condensed, comes almost entirely from the chapter content.

2. Artificial Intelligence • Early Days – AI solved problems easy for computers but difficult for humans • Problems described by formal, math rules • The AI challenge was to solve problems easy for humans but difficult to describe formally • Recognizing spoken words or faces in images • Deep learning is abut solving these more intuitive problems • Allowing computers to learn from experience • Building a hierarchy of concepts, each defined by simpler concepts

3. Machine Learning • Early AI successes were in formal environments • IBM’s Deep Blue beats world champion Kasparov • Knowledge-based AI attempted to hard-code world knowledge in formal languages • Difficulties faced suggested need for ability to acquire knowledge by extracting patterns from real-world raw data • This capability is known as machine learning • Examples are logistic regression and naïve Bayes

4. Data Representation • Performance of machine learning algorithms depends heavily on the data representation • The features/attributes characterizing the data • This dependence on representations is a general phenomenon in computer science and even in daily life • Many AI tasks can be solved by designating the right set of features

5. Representation Learning • One solution to the data representation problem is to have machine learning discover the representation • Example: Autoencoder – combination of encoder to convert the input data into another representation, and decoder to convert back to the original format • Goal is usually to separate the factors of variation that explain the observed data • To disentangle and discard those not of interest

6. Deep Learning • Deep learning solves the representation problem by introducing representations expressed in terms of simpler representations • Deep learning involves a hierarchy of concepts that allows the computer to learn complicated concepts by building them out of simpler ones • Graphically the concepts are built on top of each other with many layers • The quintessential example of a deep learning model is the feedforward deep network or MLP

7. Illustration of Deep Learning Model

8. Deep Learning • Deep learning is a machine learning approach to AI that allows computers to improve with experience and data • Deep learning achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined in relation to simpler ones • The relationship among these different AI disciplines is shown in the following figure

9. Venn Diagram: Deep Learning

10. Flow Chart: Deep Learning

11. Goodfellow Textbook • Part I • Basic math tools and machine learning concepts • Part II • Established deep learning algorithms • Part III • More speculative ideas for future research

12. Goodfellow Textbook Organization

13. Three Waves ofDevelopment of Deep Learning • 1940s-1960s: Early Neural Networks (Cybernetics?) • Rosenblatt’s perceptron – developed from Hebb’s synaptic strengthening ideas and McCulloch-Pitts Neuron • Key idea – variations of stochastic gradient descent • Wave killed by Minsky 1969, lead to “AI Winter” • 1980s-1990s: Connectionism • Rumelhart, et al. • Key idea – backpropagation • 2006-present: Deep Learning • Started with Hinton’s deep belief network • Key idea – hierarchy of many layers in the neural network

14. First Two Waves

15. Deep LearningIncreasing Dataset Sizes • Since the 1990s machine learning systems have been used successfully in commercial applications • But regarded as being more of an art than a technology • Deep learning is regarded more and more as a technology – the amount of development skill reduces as the amount of training data increases • The age of “Big Data” has made data collection easier • As of 2016, rule of thumb is that supervised deep learning algorithms need around 5k labeled samples per category • Performance is expected to match or exceed the human with a dataset of at least 10 million labeled samples

16. Deep LearningIncreasing Dataset Sizes

17. Deep LearningIncreasing Dataset SizesExample inputs from MNIST dataset

18. Deep LearningIncreasing Model Sizes • A key reason deep learning networks are wildly successful today is that greater computation resources have allowed rapidly increasing model sizes • Biological neurons are not especially densely connected and the number of connections per neuron in machine learning models have been within an order of magnitude of mammalian brains for decades • As for the increase in the total number of model neurons, which doubles roughly every 2.4 years, we have decades to go at this rate before reaching the number on neurons in the human brain

19. Deep LearningIncreasing Model Sizes: Connections per Neuron 9 = Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology, 2013

20. Deep LearningIncreasing Model Sizes: Number of Neurons 19 = Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology, 2013

21. Deep LearningIncreasing Accuracy, Complexity and Real-World Impact • Since the 1980s machine learning systems have consistently improved recognition accuracy • And applied with success to broader sets of applications • In terms of complexity • Early models recognized individual objects in small images • Today, we process large high-resolution photos and typically recognize 1000 different categories of objects • A dramatic moment in the meteoric rise of deep learning came when a convolutional network won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) for the first time – Krizhevsky, et al., 2012

22. Deep LearningIncreasing Accuracy, Complexity and Real-World Impact ImageNet Large Scale Visual Recognition Challenge (ILSVRC) now consistently won by deep networks