1 / 49

Shrinking the Cone of Uncertainty with Continuous Assessment

Shrinking the Cone of Uncertainty with Continuous Assessment. Pongtip Aroonvatanaporn CSCI 510 Fall 2011 October 5, 2011. Outline. Introduction Motivation Problems Related Works Proposed Methodologies Conclusion Tool Demo. Motivation. The Cone of Uncertainty

biana
Download Presentation

Shrinking the Cone of Uncertainty with Continuous Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Shrinking the Cone of Uncertainty with Continuous Assessment PongtipAroonvatanaporn CSCI 510 Fall 2011 October 5, 2011 (C) USC-CSSE

  2. Outline • Introduction • Motivation • Problems • Related Works • Proposed Methodologies • Conclusion • Tool Demo (C) USC-CSSE

  3. Motivation • The Cone of Uncertainty • Exists until the product is delivered, or even after • The wider, the more difficult to ensure accuracies and timely deliveries • Focus on uncertainties of team aspects from product design onwards • COCOMO II space • Many factors before that (requirements volatility, technology, etc.) (C) USC-CSSE

  4. Motivation • Key principles of ICSM • Stakeholder satisficing • Incremental and evolutionary growth of system definition and stakeholder commitment • Iterative system development and definition • Concurrent system definition and development • Risk management • COCOMO II • COCOMO II space is in the development cycle • Influences on estimations and schedules [Construx, 2006] • Human factors: 14x • Capability factors: 3.5x • Experience factors: 3.0x (C) USC-CSSE

  5. Motivation • Standish CHAOS Summary 2009 • Surveyed 9000 projects 68% project failure rate (C) USC-CSSE

  6. Terms and Definitions • Inexperienced • Inexperienced in general • Experienced, but in new domain • Experienced, but using new technology • Continuous Assessment • Assessments take place over periods of time • Done in parallel with process, instead of only at the end • Widely used in education • Used in software process measurement [Jarvinen, 2000] (C) USC-CSSE

  7. The Problem • Experienced teams can produce better estimates • Use “yesterday’s weather” • Past projects of comparable sizes • Past data of team’s productivity • Knowledge of accumulated problems and solutions • Inexperienced teams do not have this luxury No tools or data that monitors project’s progression within the cone of uncertainty (C) USC-CSSE

  8. Problems of Inexperience • Imprecise project scoping • Overestimation vs. underestimation • Project estimations often not revisited • Insufficient data to perform predictions • Project’s uncertainties not adjusted • Manual assessments are tedious • Complex and discouraging • Limitations in software cost estimation • Models cannot fully compensate for lack of knowledge and understanding of project • Overstating team’s capabilities • Unrealistic values that do not reflect project situation • Teams and projects misrepresented (business vs. technical) (C) USC-CSSE

  9. Imprecise Project Scoping • Based on CSCI577 data, projects either significantly overestimate or underestimate effort • Possibly due to: • Unfamiliarity with COCOMO • Inexperienced • Teams end up with inaccurate project scoping • Promise too much (C) USC-CSSE

  10. Overestimation • Estimate is too high to achieve within available resources • Need to reduce the scope of the project • Re-negotiate requirements with client • Throw away some critical core capabilities • Lose the expected benefits • Often do not meet client satisfactions/needs (C) USC-CSSE

  11. Underestimation • Estimate is lower than actual • Project appears that it can be done in with less resources • Clients may ask for more capabilities • Teams may end up promising more • As project progresses, team may realize that project is not achievable • If try to deliver what was promised, quality suffers • If deliver less that what was promised, clients suffer (C) USC-CSSE

  12. Project Estimations Not Revisited • At the beginning, teams do not always have the necessary data • No “yesterday’s weather” • High number of uncertainties • Initial estimates computed are not accurate • If estimates are readjusted, no problem • Reality is, estimates are left untouched (C) USC-CSSE

  13. Estimations in ICSM • Estimates are “supposedly” adjusted during each milestone reviews • Reviewed by team • Reviewed by stakeholders • Adjustments require necessary assessments to become more accurate • Without assessments, adjustments are made with no directions (C) USC-CSSE

  14. Manual Assessments are Tedious • Complex process • Time consuming • Require experienced facilitator/assessor to perform effectively • Often done by conducting various surveys, analyze the data, and determine weak/strong points • Repeated as necessary • Discouraging to the teams (C) USC-CSSE

  15. Size Reporting • How to accurately report progress • By developer’s status report? • By project manager’s take? • Report by size is most accurate • Counting logical lines of code is difficult • Even with tools support, a labor intensive task to report accurately (C) USC-CSSE

  16. Limitations in Software Cost Estimation • Little compensation for lack of information and understanding of software to be developed • The “Cone of Uncertainty” • There’s a wide range of products and costs that the project can result in • Not 100% sure until product is delivered • Designs and specifications are prone to changes • Especially in agile environment (C) USC-CSSE

  17. Overstating Team’s Capabilities • Unrealistic values • COCOMOII parameters • Do not reflect project’s situation • Business vs. Technical • Clients – want the highest value • Sales – want to sell products • ProjectManagers – want the best team • Programmers – want the least work • Is it really feasible? • Provide the evidence • From where? (C) USC-CSSE

  18. Competing Project Proposals • Write project proposals to win • Often overstate and overcommit yourselves • Put the best, highly-capabled people on the project • Have high experienced teams • Keep costs low • Any capabilities are possible • This may only be true at the time of writing • What about when project really begins? (C) USC-CSSE

  19. Ultimate Problem • Developers rather spend time to develop rather than • Documenting • Assessing • Adjusting • Not as valuable to developers as to other stakeholders • In the end, nothing is done to improve (C) USC-CSSE

  20. The Goal • Develop a framework to address mentioned issues • Help unprecedented projects track project progression • Reduce the uncertainties in estimation • Achieve eventual convergence of estimate and actual Must be quick and easy to use (C) USC-CSSE

  21. Benefits • Improve project planning and management • Resources and goals • Ensure the accuracy of estimation • Determine/confirm project scope • Improved product quality control • Certain about amount of work required • Better timeline • Allows for better work distribution • Actual project progress tracking • Better understanding of project status • Actual progress reports • Help manage realistic schedule and deliveries (C) USC-CSSE

  22. Outline • Introduction • Related Works • Assessment • Sizing • Management • Proposed Methodologies • Conclusion • Tool Demo (C) USC-CSSE

  23. IBM Self-Check [Kroll, 2008] • A survey-based assessment/retrospective • Method to overcome common assessment pitfalls • Bloated metrics, Evil scorecards, Lessons forgotten, Forcing process, Inconsistent sharing • Reflections by the team for the team • Team choose set of core practices to focus assessment on • Discussions triggered by inconsistent answers between team members • Develop actions to resolve issues (C) USC-CSSE

  24. Software Sizing and Estimation • Agile techniques • Story points and velocity [Cohn, 2006] • Planning Poker [Grenning, 2002] • Treatments for uncertainty • PERT Sizing [Putnam, 1979] • Wideband Delphi Technique [Boehm, 1981] • COCOMO-U [Yang 2006] Require high level of expertise and experience (C) USC-CSSE

  25. Project Tracking and Assessment PERT Network Chart [Wiest, 1977] • Identify critical paths • Nodes updated to show progress • Grows quickly • Becomes unusable when large, especially in smalleragile environments Goal Objective • Captures progress from conceptual, operational, and qualitative levels • Align with organization/team • Only useful when used correctly Question Answer GQM [Basili, 1995] Metric Measurement • Effective in tracking progress • Not good at responding to major changes Burn Charts [Cockburn, 2004] Architecture Review Board [Maranzano, 2005] • Reviews to validate feasibility of architecture and design • Increases the likelihood of project success • Adopted by software engineering course • Stabilize team, reduce knowledge gaps, evaluate risks (C) USC-CSSE

  26. Outline • Introduction • Related Works • Proposed Methodologies • Project Tracking Framework • Team Assessment Framework • Conclusion • Tool Demo (C) USC-CSSE

  27. Project Tracking Framework [Aroonvatanaporn, 2010] Integrating the Unified Code Count tool and COCOMO II model • Quickly determine effort based on actual progress • Extend to use Earned-Value for percent complete Adjusted with REVL Hypotheses: H1 (C) USC-CSSE

  28. Size Counting • COCOMO uses size to determine effort • Use of the Unified Code Count tool • Allows for quick collection of SLOC data • Then fed to the COCOMO model to calculate equivalent effort • Collected at every build • Depends on iteration length (C) USC-CSSE

  29. Project Tracking Results [Aroonvatanaporn, 2010] ~18% Initial estimate ~50% Initial estimate Adjusted estimate Adjusted estimate Accumulated effort Accumulated effort (C) USC-CSSE

  30. Team Assessment Framework • Similar to the approach of • IBM Self-Check • Use survey-based approach to identify inconsistencies and knowledge gaps among team members • Inconsistencies/uncertainties in answers • Use conflicting questions to validate consistencies • Two sources for question development (C) USC-CSSE

  31. Question Development • Team members assess and evaluate their own team • Operational concept engineering • Requirements gathering • Business case analysis • Architecture development • Planning and control • Personnel capability • Collaboration • These questions focus on resolving team issues and reducing knowledge gaps ICSM Team’s assessment performed by each team member Strengths, weaknesses, issues, etc. Survey questions (C) USC-CSSE

  32. Adjusting the COCOMOII Estimates • Answering series of questions is more effective than providing metrics [Krebs, 2008] • Framework to help adjust COCOMO II estimates to reflect reality • Questions developed to focus on • Team stabilization and reducing knowledge gaps • Each question relate to COCOMO II scale factors and cost drivers • Two approaches to determine relationship between question and COCOMO II parameters • Finding correlation • Expert advice (C) USC-CSSE

  33. Example Scenario COCOMO II ACAP: HI PCAP: LO APEX: NOM PLEX: HI LTEX: NOM HI NOM NOM + 50% HI NOM + 50% 10 Survey Have sufficiently talented and experienced programmers and systems engineering managers been identified? Average: 7.7 Deviation: 3.2 4 9 Discussion • Where do we lack in experience? • How can we improve? (C) USC-CSSE

  34. Outline • Introduction • Related Works • Proposed Methodologies • Conclusion • Past 577 data • Summary • Tool Demo (C) USC-CSSE

  35. COCOMO II Estimation Range Team provided range vs. COCOMO II built-in calculation • Data from Team 1 of Fall 2010 – Spring 2011 semesters Team’s pessimistic Most likely COCOMO II pessimistic Team’s optimistic COCOMO II optimistic (C) USC-CSSE

  36. CSCI577 Estimation Errors • Data from Fall 2009 – Spring 2010 semesters • Fall 2010 – Spring 2011 will be collected after this semester ends (C) USC-CSSE

  37. Conclusion • This research focuses on improving team performance and project outcomes • Tracking project progress • Synchronization and stabilization of team • Improving project estimations • Framework to shrink the cone of uncertainty • Less uncertainties in estimations • Less uncertainties within team • Better project scoping • The tool support for the framework will be used to validate and refine the assessment framework (C) USC-CSSE

  38. Outline • Introduction • Related Works • Proposed Methodologies • Conclusion • Tool Demo • What is the tool? • What does it support? (C) USC-CSSE

  39. Tool Support for Framework • Develop with IBM Jazz • Provides team management • Provides user management • Support for high collaborative environment • Potentials • Extensions to Rational Team Concert • Support for other project management tools (C) USC-CSSE

  40. Tool Support • Tool will be used throughout the project life cycle • Used for: • Tracking project progress • Project estimation • Team assessment • Frequency • Start after prototyping begins • Done every two weeks? (C) USC-CSSE

  41. Different Project Types • Architected Agile • Track through development of source code • NDI/NCS • Utilize Application Points (C) USC-CSSE

  42. Tool (C) USC-CSSE

  43. Tool (C) USC-CSSE

  44. Tool (C) USC-CSSE

  45. Tool (C) USC-CSSE

  46. Publications Aroonvatanaporn, P., Sinthop, C., and Boehm, B. “Reducing Estimation Uncertainty with Continuous Assessment: Tracking the ‘Cone of Uncertainty’.” In Proceedings of the IEEE/ACM International conference on Automated Software Engineering, pp. 337-340. New York, NY, 2010. (C) USC-CSSE

  47. References Basili, Victor R. “Applying the goal/question/metric paradigm in the experience factory”. In Software Quality Assurance and Measurement: Worldwide Perspective, pp. 21-44. International Thomson Computer Press, 1955. Boehm, B. Software Engineering Economics. Prentice-Hall, 1981. Boehm, B. and Lane, J. “Using the incremental commitment model to integrate systems acquisition, systems engineering, and software engineering.” CrossTalk, pp. 4-9, October 2007. Boehm, B., Abts, C., Brown, A.W., Chulani, S., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. Software Cost Estimation with COCOMO II. Prentice-Hall, 2000. Boehm, B., Port, D., Huang, L., and Brown, W. “Using the Spiral Model and MBASE to Generate New Acquisition Process Models: SAIV, CAIV, and SCQAIV.” CrossTalk, pp. 20-25, January 2002 Boehm, B. et al. “Early Identification of SE-Related Program Risks.” Technical Task Order TO001, September 2009. Cockburn, A. “Earned-value and Burn Charts (Burn Up and Burn Down). Crystal Clear, Addison-Wesley, 2004. Cohn, M. Agile Estimating and Planning. Prentice-Hall, 2006. Construx Software Builders, Inc. “10 Most Important Ideas in Software Development”. http://www.scribd.com/doc/2385168/10-Most-Important-Ideas-in-Software-Development Grenning, J. Planning Poker, 2002. http://www.objectmentor.com/resources/article/PlanningPoker.zip (C) USC-CSSE

  48. References IBM Rational Jazz. http://www.jazz.net Jarvinen, J. Measurement based continuous assessment of software engineering process. PhD thesis, University of Oulu, 2000 Koolmanojwong, S. The Incremental Commitment Spiral Model Process Patterns for Rapid-Fielding Projects. PhD thesis, University of Southern California, 2010 Krebs, W., Kroll, P., and Richard, E. “Un-assessment – reflections by the team, for the team.” Agile 2008 Conference, 2008. Kroll, P. and Krebs, W. “Introducing IBM Rational Self Check for Software Teams, 2008”. http://www.ibm.com/developerworks/rational/library/edge/08/may08/kroll_krebs Maranzano, J.F., Rozsypal, S.A., Zimmerman, G.H., Warnken, P.E., and Weiss, D.M. “Architecture Reviews: Practice and Experience.” Software, IEEE, 22: 34-43, March-April, 2005. Putnam, L. and Fitzsimmons, A. “Estimating Software Costs.” Datamation, 1979. Standish Group. CHAOS Summary 2009. http://standishgroup.com Unified Code Count. http://sunset.usc.edu/research/CODECOUNT/ USC Software Engineering I Class Website. http://greenbay.usc.edu/ (C) USC-CSSE

  49. References Wiest, J.D. and Levy, F.K. A Management Guide to PERT/CPM. Prentice-Hall, Englewood Press, 1977. Yang, D., Wan, Y., Tang, Z., Wu, S., He, M., and Li, M. “COCOMO-U: An Extension of COCOMO II for Cost Estimation with Uncertainty.” Software Process Change, 2006, pp.132-141 (C) USC-CSSE

More Related