1 / 10

Logistic Regression

Logistic Regression. Geoff Hulten. Overview of Logistic Regression. A linear model for classification and probability estimation. Can be very effective when: The problem is linearly separable Or there are a lot of relevant features (10s - 100s of thousands can work)

minty
Download Presentation

Logistic Regression

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Logistic Regression Geoff Hulten

  2. Overview of Logistic Regression • A linear model for classification and probability estimation. • Can be very effective when: • The problem is linearly separable • Or there are a lot of relevant features (10s - 100s of thousands can work) • You need something simple and efficient as a baseline • Efficient runtime • Logistic regression will generally not be the most accurate option.

  3. Components of Learning Algorithm: Logistic Regression • Model Structure – Linear model with sigmoid activation • Loss Function – Log Loss • Optimization Method – Gradient Descent

  4. Structure of Logistic Regression Weight per Feature Bias Weight • Linear Model: Predict 1 Threshold = .9 Predict 0 Threshold = .5 Example:

  5. Intuition about additional dimensions Higher Threshold • 3 DimensionsDecision surface is plane • N-DimensionsDecision surface is n-dimensional hyper-plane • High-dimensions are weirdHigh-dimensional hyper-planes can represent quite a lot Predict 1 Predict 0 Lower Threshold

  6. Loss Function: Log Loss • -- The predicted (pre-threshold) • Log Loss: • If is 1: • If is 0: Examples Use Natural Log (base e)

  7. Logistic Regression Loss Function Summary Log Loss Same thing expressed in Sneaky Math Average across the data set is pre-thresholding Use natural log (base e)

  8. Logistic Regression Optimization: Gradient Descent Predict 1 Predict 0 Updated Model ‘Initial’ Model Training Set

  9. Finding the Gradient Derivative of Loss Function with respect to model weights Gradient for for training sample Partial Derivative per weight Calculus you don’t need to remember Model Parameters (all the w’s) Average across training data set Compute simultaneously for all with one pass over data Update each weight by stepping away from gradient Note: for all samples

  10. Logistic Regression Optimization Algorithm • Initialize model weights to 0 • Do ‘numIterations’ steps of gradient descent (thousands of steps) • Find the gradient for each weight by averaging across the training set • Update each weight by taking a step of size opposite the gradient • Parameters • – size of the step to take in each iteration • numIterations – number of iterations of gradient descent to perform • Or use a convergence criteria… • Threshold – value between 0-1 to convert into a classification

More Related