Statistical Learning & Inference

1. Statistical Learning & Inference Lecturer: Liqing Zhang Dept. Computer Science & Engineering, Shanghai Jiao Tong University

2. Books and References • Trevor HastieRobert TibshiraniJerome Friedman , The Elements of statistical Learning: Data Mining, Inference, and Prediction, 2001, Springer-Verlag • V. Cherkassky & F. Mulier, Learning From Data, Wiley,1998 • Vladimir N. Vapnik, The Nature of Statistical Learning Theory, 2nd ed., Springer, 2000 • M. Vidyasagar, Learning and generalization: with applications to neural networks, 2nd ed., Springer, 2003 • G. Casella & R. Berger, Statistical Inference, Thomson, 2002 • T. Cover & J. Thomas, Elements of Information Theory, Wiley Statistical Learning and Inference

3. Overview of the Course • Introduction • Overview of Supervised Learning • Linear Method for Regression and Classification • Basis Expansions and Regularization • Kernel Methods • Model Selections and Inference • Support Vector Machine • Bayesian Inference • Unsupervised Learning Statistical Learning and Inference

4. Why Statistical Learning? • 我们被信息淹没，但却缺乏知识。---- R. Roger • 恬静的统计学家改变了我们的世界；不是通过发现新的事实或者开发新技术，而是通过改变我们的推理、实验和观点的形成方式。---- I. Hacking • The quiet statisticians have changed our world; not by discovering new facts or technical developments, but by changing the ways that we reason, experiment and form our opinions ....。 –Ian Hacking Statistical Learning and Inference

5. Problems in Statistical Learning • Predict whether a patient, hospitalized due to a heart attack, will have a second heart attack. • Predict the price of a stock in 6 months from now, on the basis of company performance measures and economic data. • Identify the numbers in a handwritten ZIP code, from a digitized image. • Identify the risk factors for prostate cancer, based on clinical and demographic variables.

6. Cloud Computing

7. SARS Risk Age Gender Albumin Blood pO2 RBC Count White Count Chest X-Ray Blood Pressure In-Hospital Attributes Pre-Hospital Attributes ML: SARSRisk Prediction Statistical Learning and Inference

8. Steering Direction ML: Auto Vehicle Navigation Statistical Learning and Inference

9. Protein Folding Statistical Learning and Inference

10. The Scale of Biomedical Data Statistical Learning and Inference

11. Big Data and AI

12. EX. Pattern Classification • Objective: To recognize horse in images • Procedure: Feature => Classifier => Cross+Valivation Statistical Learning and Inference

13. Classifier Horse Non Horse Statistical Learning and Inference

14. x y S G ^ y LM Function Estimation Model • The Function Estimation Model of learning • Abstract Model: • Generator (G)generates observations x (typically in Rn), independently drawn from some fixed distribution F(x) • Supervisor (S)labels each input x with an output value y according to some fixed distribution F(y|x)

15. x y S G ^ y LM Function Estimation Model • Key concepts:F(x,y), an i.i.d. k-sample on F, functions f(x,) and the equivalent representation of each f using its index  • Learning Machine (LM)“learns” from an i.i.d. l-sample of (x,y)-pairs output from G and S, by choosing a function that best approximates S from a parameterised function class f(x,), where  is in  the parameter set Statistical Learning and Inference

16. The Problem of Risk Minimization • The loss functional (L, Q) • the error of a given function on a given example • Therisk functional (R) • the expected loss of a given function on an example drawn from F(x,y) • the (usual concept of) generalisation error of a given function Statistical Learning and Inference

17. The Problem of Risk Minimization • Three Main Learning Problems • Pattern Recognition: • Regression Estimation: • Density Estimation: Statistical Learning and Inference

18. General Formulation • The Goal of Learning • Given an i.i.d. k-sample z1,…, zk drawn from a fixed distribution F(z) • For a function class’ loss functionals Q (z ,), with  in • We wish to minimise the risk, finding a function * Statistical Learning and Inference

19. General Formulation • The Empirical Risk Minimization (ERM) Inductive Principle • Define the empirical risk (sample/training error): • Define the empirical risk minimiser: • ERM approximates Q (z ,*) with Q (z ,k) the Remp minimiser…that is ERM approximates * with k • Least-squares and Maximum-likelihood are realisations of ERM Statistical Learning and Inference

20. 4 Issues of Learning Theory • Theory of consistency of learning processes • What are (necessary and sufficient) conditions for consistency (convergence of Remp to R) of a learning process based on the ERM Principle? • Non-asymptotic theory of the rate of convergence of learning processes • How fast is the rate of convergence of a learning process? • Generalization ability of learning processes • How can one control the rate of convergence (the generalization ability) of a learning process? • Constructing learning algorithms (i.e. the SVM) • How can one construct algorithms that can control the generalization ability? Statistical Learning and Inference

21. TRADITIONAL Formulate hypothesis Design experiment Collect data Analyze results Review hypothesis Repeat/Publish NEW Design large experiments Collect large data Put data in large database Formulate hypothesis Evaluate hypothesis on database Run limited experiments Review hypothesis Repeat/Publish Change in Scientific Methodology Statistical Learning and Inference

22. Learning & Adaptation • Any method that incorporates information from training samples in the design of a classifier employs learning. • Due to complexity of classification problems, we cannot guess the best classification decision ahead of time, we need to learn it. • Creating classifiers then involves positing some general form of model, or form of the classifier, and using examples to learn the complete classifier. Statistical Learning and Inference

23. Supervised learning • In supervised learning, a teacher provides a category label for each pattern in a training set. These are then used to train a classifier which can thereafter solve similar classification problems by itself. • Such as Face Recognition, Text Classification, …… Statistical Learning and Inference

24. Unsupervised learning • In unsupervised learning, or clustering, there is no explicit teacher or training data. The system forms natural clusters of input patterns and classifiers them based on clusters they belong to . • Data Clustering, Data Quantization, Dimensional Reduction, …… Statistical Learning and Inference

25. Reinforcement learning • In reinforcement learning, a teacher only says to classifier whether it is right when suggesting a category for a pattern. The teacher does not tell what the correct category is. • Agent, Robot, …… Statistical Learning and Inference

26. Classification • The task of the classifier component is to use the feature vector provided by the feature extractor to assign the object to a category. • Classification is the main topic of this course. • The abstraction provided by the feature vector representation of the input data enables the development of a largely domain-independent theory of classification. • Essentially the classifier divides the feature space into regions corresponding to different categories. Statistical Learning and Inference

27. Classification • The degree of difficulty of the classification problem depends on the variability in the feature values for objects in the same category relative to the feature value variation between the categories. • Variability is natural or is due to noise. • Variability can be described through statistics leading to statistical pattern recognition. Statistical Learning and Inference

28. Object Representation in Feature Space S(x)>=0 Class A S(x)<0 Class B X2 (area) S(x)=0 Objects (perimeter) X1 Classification • Question: How to design a classifier that can cope with the variability in feature values? What is the best possible performance? Noise and Biological Variations Cause Class Spread Classification error due to class overlap Statistical Learning and Inference

29. Examples • User interfaces: modelling subjectivity and affect, intelligent agents, transduction (input from camera, microphone, or fish sensor) • Recovering visual models: face recognition, model-based video, avatars • Dynamical systems: speech recognition, visual tracking, gesture recognition, virtual instruments • Probabilistic modeling: image compression, low bandwidth teleconferencing, texture synthesis • …… Statistical Learning and Inference

30. Course Web • http://bcmi.sjtu.edu.cn/statLearnig/ • Teaching Assistant: 涂逸<tuyi1991@sjtu.edu.cn> Statistical Learning and Inference

31. Assignment • To write a report on the topic you are working on, including: • Problem definition • Model and method • Key issues to be solved • Outcome Statistical Learning and Inference