Assuming Software Maintenance of a Large, Embedded Legacy System from the Original Developer

1. Assuming Software Maintenance of a Large, Embedded Legacy System from the Original Developer by William L. Miller Lawerence B. Compton Bruce L. Woodmansee

2. The Mission • What: Transfer maitenance of an embedded application consisting of ~200,000 executable lines of Ada code, running on three processors from the OEM to an organization that was in the process of being created to support mission critical systems’ software maintenance • Why: An econimic study indicated that organic maintenance, even including start-up costs, would save quite a bit of money. • How: The United States Government enlisted the support of Georgia Tech Research Institute to guide the transfer from the OEM.

3. Project Phases • Optimistic anticipation • Fear and regret • Relief, feeling of acomplishment

4. The Problems • There was no organic team in place • There were no organic software processes in place (i.e., development, configuration management or quality assurance) • All of the OEM’s development systems and tools were past end-of-life and no longer supported by vendors (some vendors had ceased to exist) • Custom test equipment could not be repaired due to obsolesence of many components • Documentation was incomplete and in some cases, inaccurate. • There was a challenge in the relationship between the Government organization assuming the maintenance and the OEM

5. Overall Transition Strategy • Develop processes that enabled the software engineering team to use the current, OEM supplied source code, to build a binary image that compared bit-for-bit to the binary image that was generated by the OEM for the supplied source code • Validate the processes at Georgia Tech Research Institute • Build a United States Government team • Transfer the processes from Georgia Tech Research Institute to the United States Government team and train the team • Validate that the Government team can successfully execute all the the processes

6. Specific Issues to Overcomein Building the System • A compiler that was no longer supported – who was the vendor? • A host software development system that was no longer in production • Incomplete software build instructions by the OEM • Obsolete test equipment

7. Overcoming Issuesin Building the System • Located the vendor for the compiler, obtained a licence • Purchased the host software development system on eBay • Reverse engineered the build process by incrementally matching compiler outputs for files to received binary load modules • Reverse engineer the existing test equipment, creating a new design with available components • Define a formal software development process

8. The Software Architecture Perfect Storm • Static Analysis of the code: • 10% of the packages had afferent and efferent coupling indices > 50, with some as high as 200 • 15% of the packages had cyclomatic complexities > 100 • Many procedures / functions consisted of more than 1000 executable lines of code

9. Specific Issues to Overcomein Maintaining the System • The system was poorly architected • Incomplete software documentation • Delivered documentation files could not be modified (PDF vs. Word)

10. Overcoming Issuesin Maintaining the System • To compensate for the poor architecture, the Georgia Tech Research Institute (GTRI) technical team is larger and more senior than an applicaiton this size would justify. • The GTRI team spent ~ 1 year reviewing code and documentation, reverse engineering missing or inaccurate documentation • The GTRI team spent ~ 1 year implementing and testing notional changes to the software • During year 2, the United States government team integrated with the GTRI team and jointly implemented changes • Manually convert 18,000 pages of software documentation from PDF to Word – documents were heavily table-oriented • A full release cycle is currently in process under GTRI technical guidance

11. Validating the Software Development Process • Follow the formal process to build a system based upon unmodified source code • Confirm that a bit-for-bit compare between the OEM delivered load module matches the newly generated load module • Audit all deliverable documents Word vs. PDF • Follow formal test procedures (~7,000 pages) using legacy test equipment to establish a performance baseline • Follow formal test procedures using newly developed test equipment • Confirm that test results are identical

12. Is this Transition Scenario Unique • Not really, many issues identified could exist for any long-lived software product • Internal teams and key team members can depart • A company and it’s products may be purchased by another company

13. Lessons Learned • The most important lesson learned for a mission critical system is that transition of software maintenance begins during intial development and deployment • Plan for system updates and component obsolesence • Do not assume that the business relationships will always be cooperative • Documentation • Insure that every document and artifact at every release is correct and modifiable • Component obsolesence – this is not necessarily an organizational transition issue for software maintenance • Frequently assess the availability / supportability of key components • Software build auditing • Frequently the software should be completely compiled and built without the development engineers, using release documentation