Chapter 6 Machine Learning Algorithms and Prediction Model Fitting

1. Chapter 6Machine Learning Algorithms andPrediction Model Fitting

2. Taxonomy of Machine Learning Methods • The main idea of machine learning (ML) • To use computers to learn from massive amounts of data • ML forms the core of artificial intelligence (AI) • Especially in the era of big data • This field is highly relevant to statistical decision making and data mining • In building AI or expert systems • For tedious or unstructured data • Machines can often make better and more unbiased decisions than a human learner • Learning from given data objects • one can reveal the categorical class or experience affiliation of future data to be tested

3. Taxonomy of Machine Learning Methods (cont.) • This concept essentially defines ML as an operational term • To implement an ML task • Need to explore or construct computer algorithms to learn from data • Make predictions on data based on their specific features, similarities, or correlations • ML algorithms are operated by building a decision-making model from sample data inputs • The outputs of the ML model are data-driven predictions or decisions

4. Classification by Learning Paradigms • ML algorithms can be built with different styles in order to model a problem • The style is dictated by the interaction with the data environment • Expressed as the input to the model • The data interaction style decides the learning models that a ML algorithm can produce • The user must understand the roles of the input data and the model’s construction process • The goal is to select the ML model that can solve the problem with the best prediction result • ML sometime overlaps with the goal of data mining

5. Classification by Learning Paradigms • Three classes of ML algorithms based on different learning styles • Supervised, unsupervised, and semi-supervised • Three ML methods are viable in real-life applications • The style is hinged on how training data is used in the learning process

6. Classification by Learning Paradigms (cont.) • Supervised learning • The input data is called training data with a known label or result • A model is constructed through training by using the training data set • Improved by receiving feedback predictions • The learning process continues • Until the model achieves a desired level of accuracy on the training data • Future incoming data without known labels is tested on the model with an acceptable level of accuracy • Unsupervised learning • All input data are not labeled with a known result

7. Classification by Learning Paradigms (cont.) • A model is generated by exploring the hidden structures present in the input data • To extract general rules, go through a mathematical process to reduce redundancy, or organize data by similarity testing • Semi-supervised learning • The input data is a mixture of labeled and unlabeled examples • The model must learn the structures to organize the data in order to make predictions possible • Such problems and other ML algorithms will be treated under different assumptions on how to model the unlabeled data

8. Methodologies for Machine/Deep Learning • ML algorithms are distinguishable • By applying different similarity testing functions in the learning process • e.g., Tree-based methods apply decision trees • A neural network is inspired by artificial neurons in a connectionist brain model • One can handle the ML process subjectively • By finding the best fit to solve the decision problem based on the characteristics in data sets • Ensemble methods are composed of multiple weaker models • The prediction results of these models are combined

9. Methodologies for Machine/Deep Learning (cont.) • Makes the collective prediction more accurate • These models are independently trained • Much effort is put into what types of weak learners to combine • And the ways in which to combine them effectively • Consists of mixed learners applying supervised, unsupervised, or semi-supervised algorithms • Deep learning methods extend from Artificial neural networks (ANNs) • By building much deeper and complex neural networks • Built of multiple layers of interconnected artificial neurons

10. Methodologies for Machine/Deep Learning (cont.) • Often used to mimic the human brain process in response to light, sound, and visual signals • Often applied to semi-supervised learning problems • Large data sets contain very little labeled data

11. Supervised Machine Learning Algorithms • In a supervised ML system • The computer learns from a training data set of {input, output} pairs • The input comes from sample data given in a certain format • e.g., The credit reports of borrowers • The output may be discrete • e.g., yes or no to a loan application • The output can be also continuous • e.g., The probability distribution that the loan can be paid off in a timely manner • The goal is to work out a reliable ML model • Can map or produce the correct outputs from new inputs that were unseen before

12. Supervised Machine Learning Algorithms (cont.) • The ML system acts like a finely tuned predictor function g(x) • The learning system is built with a sophisticated algorithm to optimize this function • e.g., Given an input data x in a credit report of a borrower, the bank will make a loan decision based on the predicted outcome • Four families of important supervised ML algorithms • Including regression, decision trees, Bayesian networks, and support vector machines • In solving a classification problem, the inputs are divided into two or more classes

13. Supervised Machine Learning Algorithms (cont.) • The learner must produce a model that assigns unseen inputs to one or more of these classes • Typically tackled in a supervised way • Spam filtering is a good example of classification • The inputs are e-mails, blogs, or document files • The output classes are spam and non-spam

14. Supervised Machine Learning Algorithms (cont.) • Regression is also a supervised problem • The outputs are continuous in general but discrete in special cases • Uses statistical learning • Models the relationship between input and output data • The regression process is iteratively refined using an error criterion to make better predictions • Minimizes the error between predicted value and actual experience in input data • Decision trees offers a predictive model • Solve classification and regression problems • Map observations about an item to conclusions about the item’s target value

15. Supervised Machine Learning Algorithms (cont.) • Along various feature nodes in a tree-structured decision process • Various decision paths fork in the tree structure • Until a prediction decision is made hierarchically at the leaf node • Trained on given data for better accuracy • Bayesian methods are based on statistical decision theory • Often applied in pattern recognition, feature extraction, and regression applications • Offers a directed acyclic graph (DAG) model • Represented by a set of statistically independent random variables

16. Supervised Machine Learning Algorithms (cont.) • e.g., A Bayesian network can represent the probabilistic relationships between diseases and symptoms • Given symptoms, the system computes the probabilities of having various diseases • Many prediction algorithms are used in medical diagnosis to assist doctors, nurses, and patients in the healthcare industry • Both prior and posterior probabilities are applied in making predictions • Can also be improved with the provisioning of a better training data set • Support vector machines (SVMs) are often used in supervised learning methods

17. Supervised Machine Learning Algorithms (cont.) • For regression and classification applications • Decide how to generate a hyperplane • To separate the training sample data space into distinct subspaces • e.g., A surface in a 3D space • Builds a model to predict whether a new sample falls into one subspace or another

18. Unsupervised Machine Learning Algorithms • Unsupervised learning is typically used • In finding special relationships within the data set • No training examples used in this process • The system is given a set of data to find the patterns and correlations therein • Attempt to reveal hidden structures or properties in the entire input data set • Some reported ML algorithms that operate without supervision • Including clustering methods, association analysis, dimension reduction, and artificial neural networks

19. Unsupervised Machine Learning Algorithms (cont.) • Association rule learning generates inference rules • Used to discover useful associations in large multidimensional data sets • Best explain observed relationships between variables in the data

20. Unsupervised Machine Learning Algorithms (cont.) • These association patterns are often exploited by enterprises or large organizations • e.g., Association rules are generated from input data to identify close-knit groups of friends in a social network database • In clustering, a set of inputs is to be divided into groups • Grouping similar data objects as clusters • Modeled by using centroid-based clustering and/or hierarchal clustering • All clustering methods are based on similarity testing • Unlike supervised classification, the groups are not known in advance • Making this an unsupervised task

21. Unsupervised Machine Learning Algorithms (cont.) • Density estimation finds the distribution of inputs in some space • Dimensionality reduction exploits the inherent structure in the data • In an unsupervised manner • The purpose is to summarize or describe data using less information • Done by visualizing multidimensional data with principal components or dimensions • Then simplifies the inputs by mapping them into a lower-dimensional space • The simplified data can then be applied in a supervised learning method

22. Unsupervised Machine Learning Algorithms (cont.) • Artificial neural networks (ANNs) are cognitive models • Inspired by the structure and function of biological neurons • Tries to model the complex relationships between inputs and outputs • This forms a class of pattern matching algorithms • Used for solving regression and classification problems

23. Regression Analysis • Regression analysis is widely used in ML for prediction, classification, and forecasting • Essentially performs a sequence of parametric or nonparametric estimations • Finds the causal relationship between the input and output variables • Careful to make such predictions • Causality may lead to illusions or false relationships to mislead the users • The estimation function can be determined • By experience using a priori knowledge or visual observation of the data • Need to calculate the undetermined coefficients of the function by using some error criteria

24. Regression Analysis (cont.) • The regression method can be applied to classify data by predicting the category tag of data • Regression analysis tries to understand • How the value of the dependent variable changes • While the independent variables are held unchanged • The independent variables are the inputs of the regression process, aka the predictors • The dependent variable is the output of the process • Regression analysis estimates the average value of the dependent variable • When the independent variables are fixed • The estimated value is a function of the independent variables known as the regression function • Can be described by a probability distribution

25. Regression Analysis (cont.) • Most regression methods are parametric naturally • With a finite dimension in the analysis space • Nonparametric regression may be infinite-dimensional • Accuracy or performance depends on the quality of the data set used • Related to the data generation process and the underlying assumptions made • Regression offers estimation of continuous response variables • As opposed to the discrete decision values used in classification that demand higher accuracy

26. Regression Analysis (cont.) • In the formulation of a regression process • The unknown parameters are often denoted as β • May appear as a scalar or a vector • The independent variables are denoted by a vector X and a dependent variable as Y • When multiple dimensions are involved, these parameters are vectors in form • A regression model establishes the approximated relation between X, β, and Y • The function f(X, β) is approximated by the expected value E(Y|X)

27. Regression Analysis (cont.) • The regression function f is based on the knowledge of the relationship between a continuous variable Y and vector X • If no such knowledge is available, an approximated handy form is chosen for f • Measuring the Height after Tossing a Small Ball in the Air • Measure its height of ascent h at the various time instant t • The relationship is modeled as • β1 determines the initial velocity of the ball • β2 is proportional to standard gravity • ε is due to measurement errors

28. Regression Analysis (cont.) • Linear regression is used to estimate the values of β1 and β2 from the measured data • This model is nonlinear with respect to the time variable t • But it is linear with respect to parameters β1 and β2 • Consider k components in the vector of unknown parameters β • Three models to relate the inputs to the outputs • Depending on the relative magnitude between the number N of observed data points of the form (X, Y) and the dimension k of the sample space • When N < k, most classical regression analysis methods can be applied • Most classical regression analysis methods can be applied

29. Regression Analysis (cont.) • The defining equation is underdetermined • No enough data to recover the unknown parameters β • When N = k and the function f is linear • The equation Y = f (X, β) can be solved exactly without approximation • There are N equations to solve N components in β • The solution is unique as long as the X components are linearly independent • If f is nonlinear, many solutions may exist or no solution at all • In general, the situation that N > k data points • Enough information in the data that can estimate a unique value for β under an overdetermined situation • The measurement errors εi follows a normal distribution

30. Regression Analysis (cont.) • There exists an excess of information contained in (N - k) measurements • Known as the degrees of freedom of the regression • Basic assumptions on regression analysis under various error conditions • The sample is representative of the data space involved • The error is a random variable with a mean of zero conditioned over the input variables • The independent variables are measured with no error • The predictors are linearly independent • The errors are uncorrelated

31. Regression Analysis (cont.) • The variance of the error is a constant across observations • If not, a weighted least squares method is needed • Regression analysis is a statistical method • Determines the quantitative relation in a machine learning process • Two or more variables are dependent on each other • Including linear regression and nonlinear regression

32. Linear Regression • Unitary linear regression analysis • Only one independent variable and one dependent variable are included in the analysis • The approximate representation for the relation between the two can be conducted with a straight line • Multivariate linear regression analysis • Two or more independent variables are included in regression analysis • Linear relation between dependent variable and independent variables • The model of a linear regression y = f(X) • X = (x1, x2,⋯, xn) with n  1 is a multidimensional vector and y is scalar variable

33. Linear Regression (cont.) • A linear predictor function used to estimate the unknown parameters from data • Linear regression is applied mainly in the two areas • An approximation process for prediction, forecasting, or error reduction • Predictive linear regression models for an observed data set of y and X values • The fitted model makes a prediction of the value of y for future unknown input vector X • To quantify the strength of the relationship between output y and each input component Xj • Assess which Xj is irrelevant to y and which subsets of the Xj contain redundant information about y

34. Unitary Linear Regression • Fit linear regression models with a least squares approach • Consider a set of data points in a 2D sample space (x1, y1), (x2, y2), ..., (xn, yn) • Mapped into a scatter diagram • If they can be covered approximately by a straight line: y = ax + b + ε • x is an input variable, y is an output variable in the real number range, a and b are coefficients • ε is a random error, follows a normal distribution • Need to work out the expectation by using a linear regression expression: y = ax + b

35. Unitary Linear Regression (cont.) • The residual error of a unitary model • The approximation is shown by a linear line • Amid the middle or center of all data points in the data space • The main task for regression analysis

36. Unitary Linear Regression (cont.) • To conduct estimations for coefficient a and b via observation on n groups of input samples • The common method applies a least squares method • The objective function is given by • To minimize the sum of squares, need to calculate the partial derivative of Q • With respect to , and make them zero

37. Unitary Linear Regression (cont.) • are mean value for input variable and dependent variable, respectively • After working out the specific expression for the model • Need to know the fitting degree to the dataset • If the expression can express the relation between the two variables and can be used in actual predictions • To figure out the estimated value of the dependent variable with • For each sample in the training data set

38. Unitary Linear Regression (cont.) • The closer the coefficient of determination R2 is to 1, the better the fitting degree is • The further R2 is away from 1, the worse fitting degree is • Linear regression can also be used for classification • Only used in a binary classification problem • Decide between the two classes • For multivariate linear regression, this method is also applied to classify a dataset

39. Multiple Linear Regression • During solving actual problems • Often encounter many variables • e.g., The scores of a student may be influenced by factors like earnestness in class, preparation before class and review after class • e.g., The health of a man is not only influenced by the environment, but also related to the dietary habits • The model of unitary linear regression is not adapted to many conditions • Improve it with a model of multivariate linear regression analysis • Consider the case of m input variables • The output is expressed as a linear combination of the input variables

40. Multiple Linear Regression (cont.) • 𝛽0, 𝛽1,⋯, 𝛽m, 𝜎2 are unknown parameters • ε complies with normal distribution • The mean value is 0 and the variance is equal to 𝜎2 • By working out the expectation for the structure to get the multivariate linear regression equation • Substituted y for E(y) • Its matrix form is given as y = X𝛽 • X = [1, x1,⋯, xm], 𝛽 = [𝛽0, 𝛽1,⋯, 𝛽m]T • Our goal is to compute the coefficients • By minimizing the scalar objective function

41. Multiple Linear Regression (cont.) • Defined over n sample data points • To minimize Q, need to make the partial derivative of Q with respect to each βi zero • The multiple linear regression equation

42. Multiple Linear Regression (cont.) • Multivariate regression is an expansion and extension of unitary regression • Identical in nature • The range of applications is different • Unitary regression has limited applications • Multivariate regression is applicable to many real-life problems • Estimate the Density of Pollutant Nitric Oxide in a Spotted Location • Estimation of the density of nitric oxide (NO) gas, an air pollutant, in an urban location • Vehicles discharge NO gas during their movement

43. Multiple Linear Regression (cont.) • Creates a pollution problem proven harmful to human health • The NO density is attributed to four input variables • Vehicle traffic, temperature, air humidity, and wind velocity • 16 data points collected in various observed spotted locations in the city • Apply the multiple linear regression method to estimate the NO density • In testing a spotted location measured with a data vector of {1436, 28.0, 68, 2.00} for four features {x1, x2, x3, x4}, respectively • X = [1, xn1, xn2, xn3, xn4]T and the weight vector W = [b, β1, β2, β3, β4]T for n = 1,2,.…,16

44. Multiple Linear Regression (cont.)

45. Multiple Linear Regression (cont.) • e.g., for the first row of training data, [1300, 20, 80, 0.45, 0.066], X1 = [1, 1300, 20, 80, 0.45]T, which gives the output value y1 = 0.066 • Need to compute W = [b, β1, β2, β3, β4]T and minimize the mean square error • The 16 × 5 matrix directly obtained from the sample data table • y = [0.066, 0.005,.…, 0.039]Tis the given column vector of data labels

46. Multiple Linear Regression (cont.) • To make the prediction results on the testing sample vector x = [1, 1300, 20, 80, 0.45]T • By substituting the weight vector obtained • The final answer is {β1 = 0.029, β2 = 0.015, β3 = 0.002, β4 = −0.029, b = 0.070} • The NO gas density is predicted as = 0.065 or 6.5%

47. Logistic Regression Method • A linear regression analysis model can be extended to a broader application • For prediction and classification • Commonly used in fields like data mining, automatic diagnosis for diseases, and economic predictions • The logistic model may only be used to solve problems of dichotomy • The principle is to conduct classification to sample data with a logistic function • The expression for the logistic function • Known as a sigmoid function

48. Logistic Regression Method (cont.) • The input domain of the sigmoid function is (-∞, +∞) and the range is (0, 1) • The sigmoid function is a probability density function for sample data • The basic idea for logistic regression • Sample data may be concentrated at both ends of the by the use of intermediate feature z of the sample • Can be divided into two classes

49. Logistic Regression Method (cont.) • Consider vector X = (x1,⋯, xm) with m independent input variables • Each dimension of X stands for one attribute (feature) of the sample data (training data) • Multiple features of the sample data are combined into one feature by • Need to figure out the probability of the feature with designated data • And apply the sigmoid function to act on that feature

50. Logistic Regression Method (cont.) • During combining of multiple features into one feature • Make use of the linear function • The coefficient of the linear function, i.e., feature weight of sample data, needs to be determined • Maximum likelihood estimation is adopted to transform it into an optimization problem • The coefficient is determined through the optimization method