1 / 59

Chapter 6: Model Assessment

Chapter 6: Model Assessment. Chapter 6: Model Assessment. Summary Statistics Summary. Prediction Type. Statistic. Decisions. Accuracy/Misclassification Profit/Loss Inverse prior threshold. ROC Index (concordance) Gini coefficient. Rankings. Average squared error SBC/Likelihood.

martena-barker
Download Presentation

Chapter 6: Model Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 6: Model Assessment

  2. Chapter 6: Model Assessment

  3. Summary Statistics Summary Prediction Type Statistic Decisions Accuracy/Misclassification Profit/Loss Inverse prior threshold ROC Index (concordance) Gini coefficient Rankings Average squared error SBC/Likelihood Estimates ...

  4. Summary Statistics Summary Prediction Type Statistic Accuracy/Misclassification Profit/Loss Inverse prior threshold Decisions ROC Index (concordance) Gini coefficient Rankings Average squared error SBC/Likelihood Estimates ...

  5. Summary Statistics Summary Prediction Type Statistic Accuracy/Misclassification Profit/Loss Inverse prior threshold Decisions Average squared error SBC/Likelihood ROC Index (concordance) Gini coefficient Rankings Estimates

  6. Comparing Models with Summary Statistics This demonstration illustrates the use of the Model Comparison tool, which collects assessment information from attached modeling nodes and enables you to easily compare model performance measures.

  7. Chapter 6: Model Assessment

  8. Statistical Graphics – ROC Chart 1.0 captured response fraction (sensitivity) false positive fraction (1-specificity) 0.0 0.0 1.0 The ROC chart illustrates a tradeoffbetween a captured response fractionand a false positive fraction. ...

  9. Statistical Graphics – ROC Chart 1.0 captured response fraction (sensitivity) false positive fraction (1-specificity) 0.0 0.0 1.0 The ROC chart illustrates a tradeoffbetween a captured response fractionand a false positive fraction. ...

  10. Statistical Graphics – ROC Chart 1.0 Each point on the ROC chart corresponds to a specific fraction of cases, ordered by their predicted value. 0.0 0.0 1.0 ...

  11. Statistical Graphics – ROC Chart 1.0 Each point on the ROC chart corresponds to a specific fraction of cases, ordered by their predicted value. 0.0 0.0 1.0 ...

  12. Statistical Graphics – ROC Chart 1.0 top 40% For example, this point on the ROC chart corresponds to the 40% of cases with the highest predicted values. 0.0 0.0 1.0 ...

  13. Statistical Graphics – ROC Chart 1.0 top 40% For example, this point on the ROC chart corresponds to the 40% of cases with the highest predicted values. 0.0 0.0 1.0 ...

  14. Statistical Graphics – ROC Chart 1.0 top 40% The y-coordinate shows the fraction of primary outcomecases captured in the top 40% of all cases. 0.0 0.0 1.0 ...

  15. Statistical Graphics – ROC Chart 1.0 top 40% The y-coordinate shows the fraction of primary outcomecases captured in the top 40% of all cases. 0.0 0.0 1.0 ...

  16. Statistical Graphics – ROC Chart 1.0 top 40% The x-coordinate shows the fraction of secondary outcome cases captured in the top 40% of all cases. 0.0 0.0 1.0 ...

  17. Statistical Graphics – ROC Chart 1.0 top 40% The x-coordinate shows the fraction of secondary outcome cases captured in the top 40% of all cases. 0.0 0.0 1.0 ...

  18. Statistical Graphics – ROC Chart 1.0 top 40% Repeat for all selection fractions. 0.0 0.0 1.0 ...

  19. Statistical Graphics – ROC Chart 1.0 top 40% Repeat for all selection fractions. 0.0 0.0 1.0 ...

  20. Statistical Graphics – ROC Chart 1.0 weak model strong model 0.0 0.0 1.0 ...

  21. Statistical Graphics – ROC Index 1.0 weak model ROC Index < 0.6 strong model ROC Index > 0.7 0.0 0.0 1.0 ...

  22. Comparing Modelswith ROC Charts This demonstration illustrates the use of ROC charts to compare models.

  23. Statistical Graphics – Response Chart 100% cumulative percent response 50% 0% percent selected 100% The response chart shows the expectedresponse rate for various selection percentages. ...

  24. Statistical Graphics – Response Chart 100% cumulative percent response percent selected 50% 0% 100% The response chart shows the expectedresponse rate for various selection percentages. ...

  25. Statistical Graphics – Response Chart 100% Each point on the response chart corresponds to a specific fraction of cases, ordered by their predicted values. 50% 0% 100% ...

  26. Statistical Graphics – Response Chart 100% Each point on the response chart corresponds to a specific fraction of cases, ordered by their predicted values. 50% 0% 100% ...

  27. Statistical Graphics – Response Chart 100% top 40% For example, this point on the response chart corresponds to the 40% of cases with the highest predicted values. 50% 0% 100% ...

  28. Statistical Graphics – Response Chart 100% top 40% For example, this point on the response chart corresponds to the 40% of cases with the highest predicted values. 50% 0% 100% ...

  29. Statistical Graphics – Response Chart 100% top 40% 40% The x-coordinate shows the percentage of selected cases. 50% 0% 100% ...

  30. Statistical Graphics – Response Chart 100% top 40% 40% The x-coordinate shows the percentage of selected cases. 50% 0% 100% ...

  31. Statistical Graphics – Response Chart 100% top 40% 40% The y-coordinate shows the percentage of primary outcome cases found in the top 40%. 50% 0% 100% ...

  32. Statistical Graphics – Response Chart 100% top 40% 40% The y-coordinate shows the percentage of primary outcome cases found in the top 40%. 50% 0% 100% ...

  33. Statistical Graphics – Response Chart 100% top 40% 40% Repeat for all selection fractions. 50% 0% 100% ...

  34. 6.01 Poll In practice, modelers often use several tools, sometimes both graphical and numerical, to choose a best model.  True  False

  35. 6.01 Poll – Correct Answer In practice, modelers often use several tools, sometimes both graphical and numerical, to choose a best model.  True  False

  36. Comparing Modelswith Score Rankings Plots This demonstration illustrates comparing models with Score Rankings plots.

  37. Adjusting for Separate Sampling This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.

  38. Chapter 6: Model Assessment

  39. Outcome Overrepresentation A common predictive modeling practice is to build models from a sample with a primary outcome proportion different from the original population. ...

  40. Outcome Overrepresentation A common predictive modeling practice is to build models from a sample with a primary outcome proportion different from the original population. ...

  41. Separate Sampling secondary outcome primary outcome Target-based samples are created by considering the primary outcome cases separately from the secondary outcome cases. ...

  42. Separate Sampling secondary outcome primary outcome Target-based samples are created by considering the primary outcome cases separately from the secondary outcome cases. ...

  43. Separate Sampling secondary outcome primary outcome Select some cases. Select all cases. ...

  44. Separate Sampling secondary outcome primary outcome Select some cases. Select all cases. ...

  45. The Modeling Sample + Similar predictive power with smaller case count − Must adjust assessment statistics and graphics − Must adjust prediction estimates for bias ...

  46. Adjusting for Separate Sampling (continued) This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.

  47. Creating a Profit Matrix This demonstration illustrates how to create a profit matrix.

  48. Chapter 6: Model Assessment

  49. Profit Matrices solicit ignore primary outcome 15.14 0 0 -0.68 secondary outcome 0 profit distribution for solicit decision

More Related