Computing and SE II Chapter 2: A Brief History of SE

1. Computing and SE II Chapter 2: A Brief History of SE Er-Yu Ding Software Institute, NJU

2. A View of 20th and 21st Century Software Engineering Barry Boehm ICSE 2006 Keynote Address May 25, 2006 boehm@usc.edu http://sunset.usc.edu Modified by Er-Yu Ding

3. Outline Motivation and Context A 20th Century View A 21st Century View Conclusions Timeless principles and aging practices

4. Motivation and Context Importance of historical perspective We are losing our history

5. Why Software Projects Fail- Average overrun: 89.9% on cost, 121% on schedule, with 61% of content

6. Importance of Historical Perspective Santayana half-truth: “Those who cannot remember the past are condemned to repeat it” Don’t remember failures? Likely to repeat them Don’t remember successes? Not likely to repeat them Our objectives Understanding SE more deeply Organizing important techniques/methods/methodologiesof SE systematically

7. A Hegelian View of Software Engineering Evolution

8. Outline Motivation and Context A 20th Century View A 21st Century View Conclusions Timeless principles and aging practices

9. 1950’s Thesis: Engineer Software Like Hardware Hardware-oriented: Software applications: airplanes, bridges, circuits Economics: Boehm supervisor, 1955 “We’re paying $600/hour for that computer, and $2/hour for you, and I want you to act accordingly.” Software Processes: SAGE (Semi-Automated Ground Environment) 1 MLOC air defense system, real-time, user-intensive Successful development of highly unprecedented system Hardware-oriented waterfall-type process

10. The SAGE Software Development Process - (Benington, 1956)“We were successful because we were all engineers”.

11. 1950’s Thesis: Engineer Software Like Hardware Software Techniques: Machine-Centric Application For special usage, research mainframes 1GL, 2GL Organization software with statements Coding tips for a optimized program are important

12. 1960’s, techniques progress Complier can optimizing statements well Organizing program with function Function was derived from λ calculus software get more complex

13. 1960’s Antithesis: Software Is Not Like Hardware- Four Brooks factors plus two Invisibility: like the Emperor’s Magic Cloth Visibility and controllability of project is always a topic of SE Complexity: we have see this last lecture Conformity: executed by computers, not people Changeability: up to a point, then becomes difficult Doesn’t wear out: different reliability, maintenance phenomena Unconstrained: can program antigravity, time travel, interpenetration, …

14. 1960’s Antithesis: Software Crafting Flexible materials, frequent changes as above SW demand exceeded supply of engineers Cowboy programmers as heroes Code-and-fix process Hacker culture (Levy, 1984) Judge programmers by the elegance of their code

15. 1960’s Progress and Problems Better infrastructure: OS, compilers, utilities Computer Science Departments Product Families: OS-360, CAD/CAM, math/statistics libraries Some large successes: Apollo, ESS, BofA check processing Problems: 1968, 1969 NATO Reports Failure of most large systems Unmaintainable spaghetti code Unreliable, undiagnosable systems

16. 1970’s Antithesis: Formal and Waterfall Approaches Structured Methods Structured programming (Bohm-Jacopini: GO TO unnecessary) Formal programming calculus: Dijkstra, Hoare, Floyd Formalized Top-Down SP: Mills, Baker Waterfall Methods Code and fix too expensive (100:1 for large systems) Precede code by design (De Marco SD, Jackson JSD/JSP) Precede design by requirements (PSL/PSA, SA, SREM)

17. 1970’s Antithesis: Formal Methods “formal methods” branch that focused on program correctness, either by mathematical proof, or by construction via a “programming calculus” Over the decade of the '60s theoretical computer science achieved standing as a discipline recognized by both the mathematical and the computing communities, and it could point to both applications and mathematical elegance

18. 1970’s: Formal Methods Organized programs with Function Algorithm Three control logic Type and type check When compiler deal with functions well, organized programs with Modularity This things gets importance with modularity Information hiding Data abstraction

19. 1970’s: Formalized Top-Down SP Function decomposition divide and rule Process modeling DFD Data modeling ERD

20. Increase in Software Cost-to-fix vs. Phase (1976) • • • • 1000 1000 Larger Software Projects 500 500 IBM-SSD GTE 200 200 • 100 100 80% Relative cost to fix defect Relative cost to fix defect • • • • Median (TRW Survey) 50 50 20% SAFEGUARD • • 20 20 • • Smaller Software Projects 10 10 • • 5 5 • • 2 2 1 1 Requirements Design Code Development Requirements Design Code Development Acceptance Operation Acceptance Operation test test test test Phase in Which defect was fixed Phase in Which defect was fixed

21. The Royce Waterfall Model (1970)- Explicit feedback- “Do it twice”

22. 1970’s: Problems with Formal Methods Successful for small, critical programs Largest proven programs around 10 KSLOC Proofs show presence of defects, not absence Defects in specification, proofs happen Scalability of programmer community Techniques require math expertise, $500/SLOC Average coder in 1975 survey: 2 years of college, SW experience Familiar with 2 languages, applications Sloppy, inflexible, in over his head, and undermanaged

23. Large-Organization HW/SW Cost Trends (1973)

24. 1970’s: Problems with Waterfall Model Overly literal interpretation of sequential milestones Prototyping is coding before Critical Design Review Mismatch to user-intensive systems Heavyweight documentation hard to review, maintain 7 year project with 318% requirements change Milestones passed but not validated Mismatch to COTS, reuse, legacy SW Bottom-up vs. top-down processes

25. A Hegelian View of Software Engineering Evolution

26. 1980’s Synthesis: Productivity, Reuse, Objects Worldwide concern with productivity, competitiveness Japanese example: autos, electronics, Toshiba SW reuse Major SW productivity enhancers Working faster: tools and environments Working smarter: processes and methods Work avoidance: reuse, simplicity; objects Technology silver bullets: AI, transformations, DWIM, PBE Do what I mean; programming by example

28. Tools, Environments, and Process 1970s, some requirements and design tools developed, significant tool progress had been mode in such areas as test tools and SCM tools Major emphasis in 1980s was on integrating tools into support environments: IPSEs to CASEs Integrated Project Support Environment, Computer Aided Software Engineering Requirements, design, planning and control, office support Formatted vs. formal specifications 1980s: Resource estimations and middleware is underway Process-driven CASEs Use process knowledge as tool integration framework Some overkill in locking people into roles

29. Process: Industrial Engineering Frederick W. Taylor : scientific management First. They develop a science for each element of a man's work Second. They scientifically select and then train, teach, and develop the workman Third. They heartily cooperate with the men so as to insure all of the work is being done in accordance with the principle of the science which has been developed. Fourth. There is an almost equal division of the work and the responsibility between the management and the workmen.

30. Process Process execution support: “SW processes are SW too” What’s good for products is good for processes Reuse, prototyping, architecture, programming Process compliance support: Standards and CMMs

31. Reuse: Mechanical Engineering In modular software, clearly the "standardized units or dimensions" should be standards such that software modules meeting the standards may be conveniently fitted together (without "trimming") to realize large software systems. The reference to "variety of use" should mean that the range of module types available should be sufficient for the construction of a usefully large class of programs.

32. Reuse and Object Orientation 1950’s: Math routines, utilities 1960’s: McIlroy component marketplace, Simula – 67 1970’s: Abstract data types, Parnas program families 1980’s: Smalltalk, Eiffel, C++, OO methods, reuse libraries 1990’s: Domain engineering, product lines, UML, components, patterns, pub-sub architectures, etc. 2000’s: Model driven development, service oriented architectures

33. No Silver Bullet: Brooks Automated solutions are good for “accidental” software problems Simple inconsistencies, noncompliance, inferences They do not do well on “essential” software problems Changeability: adapting themselves to unanticipated changes Conformity: working out everything the computer needs to “know” Devoid of intuition, commonsense reasoning Complexity: integrating multiple already- complex programs Invisibility: communicating their likely behavior to humans Closest thing to silver bullet: great designers

34. People: The Most Important Factor- SW engineering is of the people, by the people, and for the people 1970’s: Weinberg Psychology of Computer Programming 1980’s: DeMarco-Lister Peopleware 1990’s – 2000’s: Importance emphasized in both Agile and CMM cultures Individuals and interactions over process and tools People CMM, Personal Software Process, Team Software Process

35. Dual 1990’s – Now: software progress 1950s-1960s: machine-centric software 1960s-1980s: application-centric software 1990s-now: enterprise-centric software Distributed, large-scale Design patterns, Software architectures, UML Requirement engineering Middleware, Components, CDE, Product Line, COTS, etc. Collaborative development environment, Commercial off the shelf Future? domain-centric software Components, Product Line, COTS

36. Dual 1990’s – Early 2000’s: software techniques OO methods were strengthened through OOP ( Java )OO Design (1995)OO Analysis UML OO design principles Design patterns

37. Dual 1990’s – Early 2000’s Antithesis:- Maturity Models and Agile Methods Predictability and Control: Maturity Models——CMMI Reliance on explicit documented knowledge Heavyweight but verifiable, scalable Time to Market and Rapid Change: Agile Methods Reliance on interpersonal tacit knowledge Lightweight, adaptable, not very scalable

38. Other 1990’s – Early 2000’s Developments Y2K and reverse engineering Mature risk-driven software model: RUP Rational unified process Nature and importance of software architecture Usability and Human-Computer Interaction COTS, open source, and legacy software Software as the primary competitive discriminator

39. Open source and legacy Open source From its roots in the hacker culture of the 1960’s Collective code ownership Free software, data, computing access Free Software Linux Raymond’s “The Cathedral and the Bazaar” Legacy EAI (enterprise application integration), BPR (business process reengineering) Components and COTS are their future

40. COTS: The Future Is Here Escalate COTS priorities for research, staffing, education Software is not “all about programming” anymore New processes required * • CBA: COTS-Based Application • * Standish Group CHAOS 2000 (54%)

41. Mid-2000’s Synthesis: Risk-Driven Hybrid Products and Process Increasing integration of systems engineering and SW engineering Increasing trend toward “soft systems engineering” Increasing focus on usability and value Fit the software and hardware to the people, not vice versa Model-driven development and service-oriented architectures

43. Outline Motivation and Context A 20th Century View A 21st Century View Conclusions Timeless principles and aging practices

44. The Future of Systems and Software Eight surprise-free trends Increasing integration of SysE and SwE User/Value focus Software Criticality and Dependability Rapid, Accelerating Change Distribution, Mobility, Interoperability, Globalization Complex Systems of Systems COTS, Open Source, Reuse, Legacy Integration Computational Plenty Two wild-card trends Autonomy Software Combinations of Biology and Computing

45. Outline Motivation and Context A 20th Century View A 21st Century View Conclusions Timeless principles and aging practices

46. Timeless Principles (+) and Aging Practices (-) From the 1950’s + Don’t neglect the sciences + Look before you leap (avoid premature commitments) Avoid inflexible sequential processes From the 1960’s + Think outside the box + Respect software’s differences - Avoid cowboy programming

47. Timeless Principles (+) and Aging Practices (-) From the 1970’s + Eliminate errors early + Determine the system’s purpose Avoid top-down development and reductionism From the 1980’s + These are many roads to increased productivity + What’s good for products is good for processes - Be skeptical about silver bullets

48. Timeless Principles (+) and Aging Practices (-) From the 1990’s + Time is money and value to people + Make software useful to people Be quick, but don’t hurry From the 2000’s + If change is rapid, adaptability trumps repeatability + Consider and satisfice all of your stakeholders’ value propositions - Avoid falling in love with your slogans

49. Timeless Principles (+) and Aging Practices (-) For the 2010’s + Keep your reach within your grasp + Have an exit strategy Don’t believe everything you read “It’s true because I read it on the Internet”

50. The END Recommend papers A View of 20th and 21st Century Software Engineering Fifty Years of Progress in Software Engineering Finding a History for Software Engineering Next Lecture Software developments activities