CSCE 431: Software Lifecycle

CSCE 431: Software Lifecycle Some material from B. Meyer, M. Oriol, B. Schoeller, ETH Zurich

Outline • Early ideas about improving software quality • Software development process • Process models • Process maturity • Organizational structure that supports a process • Summary CSCE 431 Software Lifecycle

Idea: Better Languages • Working with machine languages was tedious • Especially tedious to write efficient programs • High accidental complexity • No portability, each machine programmed differently • As a consequence, low productivity • Extreme example: • Microcode developers typically wrote 2 lines of debugged code per day – 1 before lunch, 1 after lunch • Idea: translate programs from a higher-level description • Backus, 1954: FORTRAN (FORmulaTRANslator) • First HLL implementation • 20x reduction in statements CSCE 431 Software Lifecycle

Idea: Better Languages • Idea of a better language applied many times, still being applied • FORTRAN as an example: • FORTRAN I (1957) • 32 instructions • FORTRAN II (1958) • Added procedures, COMMON, arithmetic IF – my first language, on punched cards! • FORTRAN IV (1962) • Added logical expressions, logical IF – my first college language, still punched cards! • FORTRAN 66 • ANSI standard • FORTRAN 77 • ELSE, DO WHILE, (previously had to use GOTO) • Taught in freshman engineering until 1999 • FORTRAN 90, 95 • Modules, abstract data types, dynamic memory allocation, FORALL vectorization • FORTRAN 2003, 2008 • Objects, procedure pointers • FORTRAN 2015– C interoperability, new parallel features • Other successful languages have had similar progress over the decades CSCE 431 Software Lifecycle

Idea: Structure Programs Better • Böhm, Jacopini, 1966 • Any program can be rewritten as a structured program • Using only sequence, iteration, selection control constructs • One entry point • Dijkstra, 1966: Argued against the use of GOTO in his Notes on Structured Programming • Structured programming makes it more feasible to develop convincing arguments of a program’s correctness (proofs) • Adaptability and manageability of a program improves • E. W. Dijkstra, Go To Statement Considered Harmful, CACM, Vol. 11, No. 3, March 1968, pp. 147-148 http://www.cs.texas.edu/~EWD/ewd02xx/EWD215.pdf CSCE 431 Software Lifecycle

Managing Complexity: First Ideas of Modularity • Top-down design • N. Wirth’s Stepwise Refinement • Decompose problem into high-level units • Decompose each unit recursively to smaller units • When at code-level for each unit, stop • Exemplified in • N. Wirth, Program Development by Stepwise Refinement, CACM, Vol. 14, No. 4, April 1971, pp. 221-227, http://sunnyday.mit.edu/16.355/wirth-refinement.html CSCE 431 Software Lifecycle

Stepwise Refinement Example • A program that removes the “single line” comments from C++ source code • Read the file • Open the file • Read line-by-line and put in an array of strings • Remove the comments • Iterate through the array • In each line, look for first occurrence of “//” • Remove the rest of the line • Write the modified file back • Iterate through the array, store each line • Code CSCE 431 Software Lifecycle

Pros and Cons + Modularity + Big picture visible + Systematic, nothing missed – Testable only at the end – Code reuse unlikely – Simplistic • Not a realistic approach for today’s SW scale CSCE 431 Software Lifecycle

Bottom-Up Design • Lowest-level subsystems specified and implemented first • Gradually composing and building larger subsystems • System emerges from smaller subsystems + Reuse possible – “Big picture” may get lost • In practice, both approaches in use • Design, mostly top-down • Component/algorithm use and composition from libraries, bottom-up • Highly-productive programmers tend to mix top-down and bottom-up methods all the time, working wherever most progress can be made • I use the term “top-down design with bottom-up feedback” • Used in chip design too CSCE 431 Software Lifecycle

Object-Oriented Design • Simula 67, 72, Dahl and Nygaard • My first OO programming language, 1978 • Point of OO: objects modularize data and code by co-locating code and data relevant to each other • Reuse (hopefully) • Modularity • Encapsulation – Dave Parnas’ information hiding • Visual notation • Ideally modularization of design matches modularization of code • Eiffel: specification and code within the same formalism CSCE 431 Software Lifecycle

About the Early Ideas • Languages have surely improved, but a language alone is not a full answer to SE problems • No silver bullet • Simula was REALLY SLOW • Process ideas in small scale, possibly useful, but certainly not enough • Design ideas helpful, but not sufficient • More complete picture of the activities of software construction and the artifacts produced, and of organizing them, is necessary CSCE 431 Software Lifecycle

Process • Loose definition: a series of activities, involving constraints and resources, that produce an intended output • Prescribes activities • Uses resources subject to constraints to produce (intermediate or final) products • May be composed of subprocesses • Each activity has entry and exit criteria, so that it is known when the activity begins and ends • Activities organized (in a sequence, usually) so that it is clear when one activity is performed in relation to the other activities • Guiding principles that explain the goals of each activity • Constraints or controls may apply to activities (time limits, budget limits) CSCE 431 Software Lifecycle

Process • When a process is for building a product, it is often called a lifecycle • A software lifecycle describes the life of software from an idea to a deployed and maintained application CSCE 431 Software Lifecycle

What is Your Personal Process? • Code and run • Code, run, fix defects and try again • Design, code, run, fix defects and try again • Design, code, test, fix, run • Design, code, test, fix, document, run • Specify, design, code, test, fix, document, run • Requirements, specify, design, code, test, fix, document, run • Contains ~all components of established SW processes • Different component organizations lead to different processes • When not organized (with minimal specify, design, document phases), we have the “no-method” process CSCE 431 Software Lifecycle

No-Method • Sketch an idea • Code immediately • Get feedback from self or customer • Modify • Repeat ad infinitum CSCE 431 Software Lifecycle

No-Method • Does not scale • Other processes are reactions against no-method • Still often used • Especially in crises when the chosen process breaks down and panic and desperation take over • In projects where developers have domain knowledge (experts of user domain as well) • Not good to follow, but good to recognize, and to know predicted outcome • No-method process is often practiced in CSE education CSCE 431 Software Lifecycle

Best Software Process? • Proponents of a particular SW process believe that it reduces the discrepancies of what a program is supposed to do and what it does (at an acceptable cost) • Success in one setting may convert user to “believer”, but application in another environment/project/team may disappoint • A really good hammer does not make everything a nail CSCE 431 Software Lifecycle

Best Software Process? • Conjecture: there is no universal best SW process • Approach and tools depend on problem, team, quality requirements,… • An SE skill is ability to select appropriate approach and tools CSCE 431 Software Lifecycle

Process Models • Next: brief introduction to some well-known processmodels that have had an impact on SE • Practical SW processes for particular teams and applications can be tailored from process models CSCE 431 Software Lifecycle

Waterfall Model Requirements • Originally suggested by Royce, 1970 • Proposed as an initial (and flawed) concept, argued that feedback from each phase to its predecessor needed • Popular in “the beginning”, was sometimes required, e.g. DoD contracts • Archetype of all inflexible sequential models • But still used in many organizations Design Implementation Verification Maintenance CSCE 431 Software Lifecycle

Waterfall Model with Feedback Requirements • Feedback may go to any predecessor phase • E.g. maintenance problem may lead to new requirements Design Implementation Verification Maintenance CSCE 431 Software Lifecycle

About the Waterfall Model • Context: computer time was scarce, attempt to minimize coding time • Focus on documents, carefully reviewed after every phase • Every phase has completion criteria, document “sign off” • Too document-intensive • Documentation update effort >> code update effort • See: The Mythical Man-Month, Fred Brooks • Long development cycles • Models green field SW development • Reality: SW developed with a lot of iteration/versions • Shortcoming: treats SW development as manufacturing process, not problem-solving process • Curtis et al, On Building SW Process Models Under the Lamppost, ICSE, 1987, pp. 96-103. CSCE 431 Software Lifecycle

Waterfall Model and Requirements • Key assumption: possible to define/control requirements fully and precisely • Reality: requirements evolve and emerge as understanding of the system increases and environment evolves • Estimate (Bernstein & Yuhas): 40-60% of requirements can be known during requirements elicitation phase • Rest are emergent requirements (Barry Boehm) CSCE 431 Software Lifecycle

Risk in Waterfall Model CSCE 431 Software Lifecycle

More Phases in More Recent Waterfall Model CSCE 431 Software Lifecycle

Variation in Waterfall: V-Model • Development phases → code → test phases • Each development phase has corresponding test phase • Test planning a parallel activity • Intent: mitigate risk, improve & guarantee quality, reduce total project/lifecycle cost, improve communications between all stakeholders CSCE 431 Software Lifecycle

Variant of V-Model • Implementation and design guided by testing CSCE 431 Software Lifecycle

Sequential vs. Iterative Models • Difficulties understanding a SW system before it has been built lead to iterative models • Many variations, basic idea is that certain process phases are repeated, until convergence CSCE 431 Software Lifecycle

Evolutionary Model + Prototype can be evaluated and design changed + User can evaluate prototype and give feedback – Code twice may be too much to ask – Still a sequential model CSCE 431 Software Lifecycle

Prototype Model • Iterate until convergence CSCE 431 Software Lifecycle

Spiral Model CSCE 431 Software Lifecycle

Spiral Model • Iterations typically 6 months to 2 years • Generates a succession of prototypes (or versions) + Focus on continuous assessment of risk + Estimates become increasingly accurate + Manages change + SEs start their work earlier – Not a “lean” process - ~overkill for small projects CSCE 431 Software Lifecycle

Processes Abound • Boehm’s WinWin Spiral • Development Plan approach (L. Bernstein) • RUP (Rational Software Corp. – now IBM) • … • Processes arise from • Organization’s need to have control over SW system development, accountability for delays • Desire to guarantee SW quality • Desire to rely on codified process to help organization plan/manage resources CSCE 431 Software Lifecycle

Agile Processes • Current fashion • Probably for good reason – success • Misconception: agile process == no-method process • “Those who brand proponents of … agile methodologies as hackers are ignorant of both the methodologies and the original definition of the term hacker” (Bernstein & Yuhas) CSCE 431 Software Lifecycle

Agile Processes • Everything in moderation • Embraces modeling, but not to be filed in a dusty corporate repository • Embraces documentation, but not to waste paper in never-maintained/used tomes • Insists on planning, but recognizes its limits in changing environments • Core emphasis: schedule • Driver organizations to schedule containment CSCE 431 Software Lifecycle

Examples of Agile Processes • Extreme Programming (XP) • Scrum • Crystal • Dynamic Systems Development Method (DSDM) • Lean Development • Feature-Driven Development (FDD) CSCE 431 Software Lifecycle

Example: Scrum CSCE 431 Software Lifecycle

Some Typical Practices • Test-first programming • Regular refactoring • Continuous integration • Pair programming • Shared code (no-one claims ownership) • Coding standard to which everyone adheres • A common work area • Customer on site • User stories • “Manager-servant” • Timeboxing CSCE 431 Software Lifecycle

Iterative Development • Each iteration is a mini-project • Iteration goal: a release • stable, integrated, tested, partially-complete system • All produced code from all teams integrated into a release • Most iteration releases internal (to the team) • During an iteration, no changes from external stakeholders • Can make changes between iterations CSCE 431 Software Lifecycle

Some Typical “No-Nos” • Big up-front requirements specifications • Binding documents • More of a collaboration agreement with the customer than a legally binding contract of a feature list and specs • Diagrams (e.g. UML) • Non-deliverable products • Iterations should be deliverable, even if only partially-complete CSCE 431 Software Lifecycle

Why Minimize Up-Front Requirement Specification Effort? • Details too complex for people to grasp • Many details only revealed during development • Stakeholders not sure what they want • They have difficulty stating it • As product develops, stakeholders will change their minds • External forces (environment) cause changes and extensions CSCE 431 Software Lifecycle

How Many Requested Features Are Used? • Jim Johnson, 2002: Never 45% Seldom 19% Occasionally 16% Often 13% Always 7% • Reinertson, Thomke, 1998: • Requirements always change • Complete requirements to developers in 5% of cases • On average, design starts with 58% of requirements specified in detail CSCE 431 Software Lifecycle

What is Needed for Architecture Design? • Agile answer: • Architect needs: • Most nonfunctional or quality requirements (internationalization, load, response time, etc.) • Subset of functional requirements • to design a good core architecture CSCE 431 Software Lifecycle

Agile Manifesto • Individuals and interactions >> processes and tools • Working SW >> comprehensive documentation • Customer collaboration >> contract negotiations • Responding to change >> following a plan • http://agilemanifesto.org CSCE 431 Software Lifecycle

Principles Behind the Agile Manifesto • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage. • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. • Business people and developers must work together daily throughout the project. • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. • Working software is the primary measure of progress. • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. • Continuous attention to technical excellence and good design enhances agility. • Simplicity–the art of maximizing the amount of work not done–is essential. • The best architectures, requirements, and designs emerge from self-organizing teams. • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. CSCE 431 Software Lifecycle

Intermediate Summary • Evolution of languages has helped to reduce number of programming defects • Implementation effort has been reduced thanks to abstractions, libraries, better languages • Result: projects have grown to tackle more complex tasks • Processes have emerged as an attempt to make SW projects more predictable • Brief look at early process ideas • Brief overview of some processes in use today CSCE 431 Software Lifecycle

CSCE 431: Software Lifecycle