A Case Study in Acquiring Robotic Capabilities Inherently a System of Systems Activity?

1. A Case Study in Acquiring Robotic CapabilitiesInherently a System of Systems Activity? DeWitt T. Latimer IV, dlatimer@usc.edu USC Center for Systems & Software Engineering http://robotics.usc.edu/~dlatimer http://csse.usc.edu USC-CSSE Convocation 25 Oct 2006 The views expressed in this presentation are those of the author and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the U.S. Government.

2. Outline • Overview • Case Study Environment • Acquisition Program Goals • Surface Assessment Robot • Root Cause of Acquisition Failure • Summary

3. Overview Overview • Describe the acquisition environment • Examine the acquisition goals for a straight-forward robotic system • Describe the system as it was developed • Identify a root cause of the acquisition failure at the transition to operations stage • Discuss if there are inherent system of systems aspects of straight-forward robotic systems and need to generate case studies for application of knowledge

4. Case Study Environment Case Study Environment • Based on completed robot built by Carnegie Mellon University graduate students and staff • Acquiring commercial company was very familiar with subcontracting, sponsoring research projects, and a had long history of successful acquisitions with CMU

5. Case Study Environment Diffuse Management Quality Assurance Office (TQM/6-Sigma Champion) Individual Project General Managers The business units involved in client organization No senior level involvement, business units essentially autonomous QA office provides mechanism for handling deviations GMs manage delivery of roadways and utilize inspectors VP R&D (Acquirer)

6. Acquisition Program Goals Acquisition Program Goals • Proposed system to: • Assess suitability of roadway subsystem by identifying deviations from required profile • Automate measurement function performed by aging senior field engineer • Ensure Return on Investment (ROI) achieved in single roadway construction project • Ensure system can be delivered in time to utilize on upcoming project • Provide system that increases the technical prestige of the company in delivery of systems that include roadway surfaces • Continue successful relationship between company and CMU

7. Surface Assessment Robot System History • Developed by CMU team (I was a member) • Met all technical requirements • Autonomously detected and marked high/low variations of a road exceeding 1/8” in 10 feet (~1 part per 1000) • Met all business objectives • ROI: 100 to 1 • Paid for R&D investment in demo project • Currently sits on a shelf (in a bay actually)

8. Surface Assessment Robot Development Activities • Requirement to mark all deviations against profile specification handled by 2-pass autonomous detector/marking system • Driven by human operator for safety concerns

9. Surface Assessment Robot Development Activities • Developer engaged significantly with the client's six-sigma organization to ensure appropriate deviation handling and recording • Recast field engineer's role to be to review collected data to disposition deviations

10. Surface Assessment Robot Development Activities • Developers noted system requirement for roadway was in terms of ride quality, not road roughness • Proposed quarter-car model to characterize deviations before marking as either impacting system requirements or being insignificant • All deviations to roughness specification would be automatically catalogued and those not impacting ride quality would be classified as no rework needed • All deviations could be audited and reviewed

11. Surface Assessment Robot Development Activities • Client's staff engineers rejected, indicating that all engineering decisions must be made by humans – field engineer to classify all deviations • Development continued and delivered system on-time • Robotic system demonstrated on actual program track with client and roadway subcontractor representatives

12. Surface Assessment Robot Final Roll-Out • System marked several orders of magnitude more deviations than expected, surprising as subcontractor was the best-qualified and had a track record of low/zero defect roadways • All deviations were verified by senior engineer, however the engineer indicated that most deviations were so minor as to have no system impacts and would have been ignored – counter to the QA culture!

13. Surface Assessment Robot End Game • After final system demonstration, system was transitioned to customer • No maintenance role for CMU was desired by either party (this was an upfront requirement) • We later learned that the system was never transitioned to regular usage, indeed was not used on any future project

14. Root Cause of Acquisition Failure Candidate Root Causes(but not the actual ones) Acquisition Requirements Development (RD) • All top level requirements were present, roughness was the right technical requirement for contract enforcement reasons. Developer Error • System met and exceeded all technical performance measures, cost targets, and return on investment parameters Human-Robot Interaction Design (HRI) factors • Operators in various interface walk-through events and field tests were very satisfied Acquisition Verification or Validation (VER/VAL) • How to perform VER/VAL on requirements at an acquisition level if no engineer has experience to indicate what analysis would be needed? See next slide...

15. Root Cause of Acquisition Failure Proposed Root Cause Constraints did not conceive of the robot acting as a senior engineer agent (Acquisition TS SG1) • Acquisition constraints levelled on the project, while valid, implied that no automated analysis tools could classify defects in this project. • Discovered real deviation rate by roadway roughness being larger than previously believed would have required the client to entirely rethink both their roadway standard and their quality management practice – both of which were outside the scope of the project's management. This is the essence of a system of systems issue.

16. Summary Summary This provides a compelling story for the study of acquisition and acquisition engineering and provides a concrete example of how an acquisition organization can fail to acquire “the right system” without appropriate attention paid to system of systems issues. Therefore, observing an organization's acquisition and employment of robotics provides a microcosm to explore system of systems acquisition issues.

17. Summary Summary • Overview - Environment • Acquisition Program Goals • Surface Assessment Robot • Root Cause of Acquisition Failure • Summary

18. Backup

19. Acronyms • CMU – Carnegie Mellon University • GM – General Manager (construction project) • HRI – Human-Robot Interaction • QA – Quality Assurance • RD – Requirements Development Process Area • SG – Specific Goal (of a CMMI process area) • TS – Technical Solution Process Area • USC – University of Southern California • VAL – Validation Process Area • VER – Verification Process Area

20. References • “Adapting CMMI for Acquisition Organizations: A Preliminary Report”, Dodson, et. al., CMU/SEI-2006-SR-005 • "Running Surface Assessment Technology Review", Latimer, et. al., CMU/ICES-04-03-02 • Research project notes, meeting minutes, and project plan documents from CMU for Roadway Assessment System

21. Bio 1Lt DeWitt Latimer IV, USAF is currently assigned as a PhD Student to the Air Force Institute of Technology and working towards a PhD in Computer Science at the University of Southern California. He is advised by Prof Barry Boehm and Prof Gaurav Sukhatme. His research focuses on investigating the nature of acquiring autonomous robotic systems. He earned his MS degrees in Robotics (2001) and Civil Engineering (2002) at Carnegie Mellon University. He is a senior member of the IEEE and a member of ACM, ASCE, and AFCEA and was awarded the CSDP credentials from the Computer Society.