Statistical Learning & Inference

1. Statistical Learning & Inference Lecturer: Liqing Zhang Dept. Computer Science & Engineering, Shanghai Jiao Tong University

2. Books and References • Trevor HastieRobert TibshiraniJerome Friedman , The Elements of statistical Learning: Data Mining, Inference, and Prediction, 2008, Springer-Verlag • Vladimir N. Vapnik, The Nature of Statistical Learning Theory, 2nd ed., Springer, 2000 • M. Vidyasagar, Learning and generalization: with applications to neural networks, 2nd ed., Springer, 2003 • G. Casella & R. Berger, Statistical Inference, Thomson, 2002 • T. Cover & J. Thomas, Elements of Information Theory, Wiley Statistical Learning and Inference

3. Overview of the Course • Introduction • Overview of Supervised Learning • Linear Method for Regression and Classification • Basis Expansions and Regularization • Kernel Methods • Model Selections and Inference • Support Vector Machine • Latent Dirichlet Allocation • Unsupervised Learning Statistical Learning and Inference

4. Intelligent Systems

5. Why Statistical Learning? • The quiet statisticians have changed our world; not by discovering new facts or technical developments, but by changing the ways that we reason, experiment and form our opinions ....。 –Ian Hacking Statistical Learning and Inference

6. Problems in Statistical Learning • Predict whether a patient, hospitalized due to a heart attack, will have a second heart attack. • Predict the price of a stock in 6 months from now, on the basis of company performance measures and economic data. • Identify the numbers in a handwritten ZIP code, from a digitized image. • Identify the risk factors for prostate cancer, based on clinical and demographic variables.

7. Cloud Computing

8. SARS Risk Age Gender Albumin Blood pO2 RBC Count White Count Chest X-Ray Blood Pressure In-Hospital Attributes Pre-Hospital Attributes ML: SARSRisk Prediction Statistical Learning and Inference

9. Steering Direction ML: Auto Vehicle Navigation Statistical Learning and Inference

10. Protein Folding Statistical Learning and Inference

11. Big Data and AI

12. EX. Pattern Classification • Objective: To recognize horse in images • Procedure: Feature => Classifier => Cross+Valivation Statistical Learning and Inference

13. Classifier Horse Non Horse Statistical Learning and Inference

14. x y S G ^ y LM Function Estimation Model • The Function Estimation Model of learning • Abstract Model: • Generator (G)generates observations x (typically in Rn), independently drawn from some fixed distribution F(x) • Supervisor (S)labels each input x with an output value y according to some fixed distribution F(y|x)

15. x y S G ^ y LM Function Estimation Model • Key concepts:F(x,y), an i.i.d. k-sample on F, functions f(x,) and the equivalent representation of each f using its index  • Learning Machine (LM)“learns” from an i.i.d. l-sample of (x,y)-pairs output from G and S, by choosing a function that best approximates S from a parameterised function class f(x,), where  is in  the parameter set Statistical Learning and Inference

16. The Problem of Risk Minimization • The loss functional (L, Q) • the error of a given function on a given example • Therisk functional (R) • the expected loss of a given function on an example drawn from F(x,y) • the (usual concept of) generalisation error of a given function Statistical Learning and Inference

17. The Problem of Risk Minimization • Three Main Learning Problems • Pattern Recognition: • Regression Estimation: • Density Estimation: Statistical Learning and Inference

18. General Formulation • The Goal of Learning • Given an i.i.d. k-sample z1,…, zk drawn from a fixed distribution F(z) • For a function class’ loss functionals Q (z ,), with  in • We wish to minimise the risk, finding a function * Statistical Learning and Inference

19. General Formulation • The Empirical Risk Minimization (ERM) Inductive Principle • Define the empirical risk (sample/training error): • Define the empirical risk minimiser: • ERM approximates Q (z ,*) with Q (z ,k) the Remp minimiser…that is ERM approximates * with k • Least-squares and Maximum-likelihood are realisations of ERM Statistical Learning and Inference

20. 4 Issues of Learning Theory • Theory of consistency of learning processes • What are (necessary and sufficient) conditions for consistency (convergence of Remp to R) of a learning process based on the ERM Principle? • Non-asymptotic theory of the rate of convergence of learning processes • How fast is the rate of convergence of a learning process? • Generalization ability of learning processes • How can one control the rate of convergence (the generalization ability) of a learning process? • Constructing learning algorithms (i.e. the SVM) • How can one construct algorithms that can control the generalization ability? Statistical Learning and Inference

21. TRADITIONAL Formulate hypothesis Design experiment Collect data Analyze results Review hypothesis Repeat/Publish NEW Design large experiments Collect large data Put data in large database Formulate hypothesis Evaluate hypothesis on database Run limited experiments Review hypothesis Repeat/Publish Change in Scientific Methodology Statistical Learning and Inference

22. Learning & Adaptation • Any method that incorporates information from training samples in the design of a classifier employs learning. • Due to complexity of classification problems, we cannot guess the best classification decision ahead of time, we need to learn it. • Creating classifiers then involves positing some general form of model, or form of the classifier, and using examples to learn the complete classifier. Statistical Learning and Inference

23. Supervised learning • In supervised learning, a teacher provides a category label for each pattern in a training set. These are then used to train a classifier which can thereafter solve similar classification problems by itself. • Such as Face Recognition, Text Classification, …… Statistical Learning and Inference

24. Unsupervised learning • In unsupervised learning, or clustering, there is no explicit teacher or training data. The system forms natural clusters of input patterns and classifiers them based on clusters they belong to . • Data Clustering, Data Quantization, Dimensional Reduction, …… Statistical Learning and Inference

25. Reinforcement learning • In reinforcement learning, a teacher only says to classifier whether it is right when suggesting a category for a pattern. The teacher does not tell what the correct category is. • Agent, Robot, Game,…… Statistical Learning and Inference

26. SL Examples • User interfaces: modelling subjectivity and affect, intelligent agents, transduction (input from camera, microphone, or fish sensor) • Recovering visual models: face recognition, model-based video, avatars • Dynamical systems: speech recognition, visual tracking, gesture recognition, virtual instruments • Probabilistic modeling: image compression, low bandwidth teleconferencing, texture synthesis • …… Statistical Learning and Inference

27. Course Web • http://bcmi.sjtu.edu.cn/statLearning/ • Teaching Assistant: 顾章轩<zhangxgu@126com> Statistical Learning and Inference

28. Assignment • To write a short report on the topic you are working on, including: • Problem definition • Model and method • Key issues to be solved • Outcome Statistical Learning and Inference