1 / 29

Understanding the Human Estimator

Understanding the Human Estimator. Gary D. Boetticher Boetticher@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA. Nazim Lokhandwala Lokhandwala@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA. James C. Helm Helm@uhcl.edu

leonard
Download Presentation

Understanding the Human Estimator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Understanding the Human Estimator Gary D. Boetticher Boetticher@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA Nazim Lokhandwala Lokhandwala@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA James C. Helm Helm@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  2. Introduction • Chaos Chronicles [Standish03] • 300 billion dollars • 250,000 new projects • 1.2 million dollars per project http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  3. Boehm’s 4X http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  4. Types of Estimation [Jorgenson04] 7 - 16% Algorithmic and Machine Learners 63 - 86% Human-Based http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  5. Research Focus • Number of Papers On Software Estimation in IEEE [Jorgenson02] • Human-Based Estimation (17%) • Other (83%) http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  6. Statement of Problem How do human demographics affect human-based estimation? Can predictive models be constructed using human demographics? http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  7. Investigation Procedure • Collect demographics from participants • Request participants to estimate software components • Build models (Estimates vs. Actuals) Survey http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  8. Which Demographics? • Basic Demographics • Academic Background • Work Experience • Domain Experience http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  9. The Survey http://nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  10. Competitive Procurement Software Supplier Software Buyer Software Distribution Server Supplier1 Buyer Admin Supplier2 ... Buyer1 Buyern : Suppliern http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  11. Sample Estimation Screenshots http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  12. Survey Results Screenshots http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  13. Data Collection • Invitations • Filtered Incomplete Records • 122 Final Records http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  14. Participant Educational Background Most of the participants hold Bachelors or Masters Degrees http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  15. Participant Work Experience http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  16. Mean (Years) Maximum (Years) Standard Deviation Domain Experience Procurement and Billing 0.6209 10 1.3818 Process Industry 0.7274 20 2.2512 Participant Domain Experience http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  17. Data Preparation INPUT= • 69% zeros…Needs Consolidation Courses, Workshops, Conferences, Programming Exp. 45 attributed reduced to 14 attributes • Highest Degree Achieved…Need Transformation OUTPUT= MRE=Abs (Total Actual – Total Est.)/(Total Actual) http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  18. Build Models • Linear Regression (Excel) • Non-Linear Regression (DataFit) • Genetic Programming (GDB_GP) http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  19. GP Configuration 3 Settings • 1000 Chromosomes 50 Generations • 512 Chromosomes 128 Generations • 1000 Chromosomes 128 Generations 20 Trials each http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  20. Linear Regression Genetic Programming Non-Linear Regression R Squared 0.1550 0.9174 0.8847 Std. Error 4.4580 1.3875 1.6470 Linear Regression Genetic Programming Non-Linear Regression Mean 0.1550 0.5592 0.8847 T-test 3.45E-17 1.87E-15 Results: All Demographic Factors Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  21. Linear Regression Genetic Programming Non-Linear Regression R Squared 0.0373 0.2784 0.2136 Std. Error 4.6101 3.9738 4.1667 Linear Regression Genetic Programming Non-Linear Regression Mean 0.0373 0.1973 0.2136 T-test 2.74E-13 0.0486 Results: Educational Factors Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  22. Linear Regression Genetic Programming Non-Linear Regression R Squared 0.0596 0.7572 0.3698 Std. Error 4.5169 2.2855 4.0644 Linear Regression Genetic Programming Non-Linear Regression Mean 0.0596 0.5564 0.3698 T-test 2.73E-19 1.54E-11 Results: Work Experience Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  23. Linear Regression Genetic Programming Non-Linear Regression R Squared 0.0243 0.5911 0.3260 Std. Error 4.5425 2.9283 3.9091 Linear Regression Genetic Programming Non-Linear Regression Mean 0.0243 0.5405 0.3260 T-test 3.27E-23 4.55E-16 Results: Domain Experience Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  24. Summary of All Experiments R Square Values http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  25. Too Much of a Good Thing? Best Equation: All Factors. r2 = 0.9174 ((Log (TechGradCourses + (TechGradCourses ^ ((Log TotWShops)/(Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (((ProcIndExp + (Log (Sin MgmtGradCourses)))/(Sin SWPMExp)) + (Sin ((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Sin SWPMExp)))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((Log SWProjEstExp) / (((Log (ProcIndExp + (Log (TechGradCourses ^ ((Log SWProjEstExp) / (Log SWProjEstExp)))))) - 3) / (ProcIndExp + (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (TechGradCourses ^ (Log SWProjEstExp))))) / (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp)))))))))))))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) + ((Log SWProjEstExp) / (Log SWProjEstExp)))))) / (Log (Log (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp))))))))))))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Log ((((Log TotLangExp) / (Log SWProjEstExp)) / (Log SWProjEstExp)) / (Sin SWPMExp))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))))))) + (((((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + ((TechGradCourses ^ (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))) / (Sin SWPMExp))))))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (Sin SWPMExp))) http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  26. Conclusions • Viability of a human-based est. model • Model assessment • Non-linear  GP • Impact on Human Based Estimation 1) All Factors 2) Domain ExperienceWork Experience 3) Education http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  27. Future Directions • Equation Optimizer for GP • Collect More Data • Further analysis without consolidation • Detailed Effect of Educational Factors • Use other statistical indicators • Build other models • Hybrid (Non-linear and GP) • Classifiers • Impact of process on estimation http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  28. Questions? http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

  29. Thank You! http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

More Related