COSYSMO Usage Experience Panel: What is Happening at Lockheed Martin

1. COSYSMO Usage Experience Panel: What is Happening at Lockheed Martin Garry Roedler, Lockheed Martin Engineering Process Improvement Center garry.j.roedler@lmco.com Pete McLoone, Lockheed Martin Integrated Systems and Solutions peter.j.mcloone@lmco.com

2. Talking Points • Collecting data points for the industry COSYSMO Effort across Lockheed Martin Business Areas • Electronics Systems has completed a usage pilot • Integrated Systems & Solutions has done some preliminary evaluation of the model • Lockheed Martin is deeply involved in extending COSYSMO to Cover Risk and Schedule, distributing effort across the life cycle, and addressing Reuse • Intending to do more local calibration and evaluation • Promoting COSYSMO internally via Lockheed Martin Measurement Workshop and other communications mechanisms • Contributing to the review of the COSYSMO User Manual and other aspects of the industry effort

3. Collecting Data • Reasonable success obtaining data points; need to capture many more • Starting to obtain data relevant to the COSYSMO Risk Module • Difficult to validate effort and size data without a lot of effort • Suspect very uneven methods for counting interfaces • “Engineered” system requirements seem to work better than customer system requirements • Labor hours are probably somewhat uneven in terms of “rules” utilized; cannot easily supply a EIA 632 breakdown • Cost drivers are sometimes misinterpreted but not a big problem; helps if a team of systems engineers is used and a COSYSMO expert can facilitate their determination

4. Electronic Systems Pilot • Had to adjust inputs because of reuse; did not calibrate to local data; default parameters generated high estimates compared to actual effort • May be useful as a sanity check for SE cost estimates • Reuse and local calibration need to be added • Academic COSYSMO is bare bones spreadsheet • Does not have features of other parametric models • Reuse, local calibration capability, Class of cost, Schedule, Risk, ranges for parameters, Cost particular phase(s), labor distributions, summary reports, graphical outputs • Commercial version SystemStar has some of these features • Need a comprehensive User’s Guide (in development) • Better guidance on how to combine multiple questionnaire attributes into a single model parameter • Better definitions of attributes, with examples • Definitions of phases covered • Next steps: more data points, local calibration, use reuse and risk profile and evaluate

5. Integrated Systems and Solutions • Did not do local calibration • Results with four data points showed reasonable results • But data collection was “quick and dirty” so not much confidence in the results • Nonetheless, it appears with local calibration the model should be usable • The road ahead involves more careful data collection to obtain sufficient data points to do local calibration properly

6. Final Thoughts • Local calibration is only as good as the data that goes into it • Don’t underestimate the effort needed to obtain good data • Experiment! • Is accuracy related to size driver weights? • Is accuracy related to cost driver weights? • Does accuracy get better if I throw away a size parameter? • Provide quality data points to Ricardo