There is no magic Bob Martin bobmartin08008@mac

1. There is no magic Bob Martin bobmartin08008@mac.com

2. There is no magic Bob Martin bobmartin08008@mac.com • “I assume you will show the alternative ways to use a cage, a whip, and a chair to manage software teams?” - Stu Feldman, VP Engineering Google

3. About Me • Built a lot of software ~100M lines of code • System software • Operating systems • Programming languages • Transaction control • Network management • Service control points • Embedded systems • Packet switches • Network equipment • Testing equipment • Love technology • EE and CS trained • Life long passion – now neural science, molecular & evolutionary biology • Help me convince Al to work in computational biology! • Not expert in contemporary software technology/tools • Conned into this talk by a great friend of 43+ years

4. Facts of Life • Failure is the norm • But it needn’t be • Error correction costs increase exponentially during the lifecycle • Programmers are a special breed* • They like machines not people • They have high need for approval/recognition • They must respect the leader • Beware, complicators and prima donnas lurk in the weeds • Software engineering is complexity management • Race with technology/tools • Complexity is still ahead • Good software engineering is old hat • But is not always used - why???? “Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe” - Einstein * “Psychology of Computer Programmer” - Gerald Weinberg

5. Why Software Projects Fail * Average overrun: 89.9% on cost, 121% on schedule, with 61% of content * Barry Boehm - A View of 20th and 21st Century Software Engineering

6. Increase in Software Cost-to-fix vs. Phase (1976) * 1000 1000 500 500 200 200 100 100 Relative cost to fix defect 50 50 20 20 • • Smaller Software Projects 10 10 5 5 2 2 1 1 Requirements Design Code Development Requirements Design Code Development Acceptance Operation Acceptance Operation test test test test Phase in which defect was fixed * Barry Boehm - A View of 20th and 21st Century Software Engineering

7. Design Guidelines External function, e.g., user language Internal function, e.g., parsers Techniques Deductive or inductive Modularity and interfaces Complexity control Proscriptions Completeness, modularity, efficiency Self-monitoring and performance improvement Incremental systems Balance Security, control, … vs. cost Limited goals to attain excellent performance Design problems Data structures on system design Fixed resources Cooperating processes Documentation Production Organization for producing software Number and quality of people Structure of large groups Control and measurement Internal communication Production techniques Monitoring Service Distribution Media Acceptance criteria Documentation Adaptation Maintenance Error detection & reporting Response and distribution Documentation Performance Feedback into design What Year Was This Conference?

8. What Year Was This Conference?NATO Software Engineering Conference - 1968 Design • Guidelines • External Function, e.g., user language • Internal Function, e.g., parsers • Techniques • Deductive or inductive • Modularity and interfaces • Complexity control • Proscriptions • Completeness, modularity, efficiency • Self-monitoring and performance improvement • Incremental systems • Balance • Security, control, … vs. cost • Limited goals to attain excellent performance • Design problems • Data structures on system design • Fixed resources • Cooperating processes • Documentation Production • Organization for producing software • Number and quality of people • Structure of large groups • Control and measurement • Internal communication • Production techniques • Monitoring Service • Distribution • Media • Acceptance criteria • Documentation • Adaptation • Maintenance • Error detection & reporting • Response and distribution • Documentation • Performance • Feedback into design

9. Learning • Disasters • Mine • Others • Really good companies • IBM - Discipline & inspections • Sun - Tempo • Japan - Quality circles & reuse • Microsoft - Reuse • Apple - User centered design & tool kits, “Design of Everyday Things” - Don Norman • Really smart professionals • Al Aho – Swamp drainer • Fred Brooks –“Mythical Man Month,”“No Silver Bullet” • Barry Boehm – Estimation and cost of errors • Michael Cusumano – Software factories • Edsger Dijkstra – Importance of time in systems • Watt Humphrey - Encapsulated team and personal discipline - “TSP, PSP” • Martyn Thomas - Formal methods • Ken Thompson - The programming craft

10. Two Troubled Projects • Loop Maintenance System • IBM MVS system • PDP 11/20 backup • 8 Months to go No schedule No performance plan No requirements Weak management

11. Two Troubled Projects • Loop Maintenance System • IBM MVS system • PDP 11/20 backup • 8 Months to go No schedule No performance plan No requirements Weak management Switch Test Head PDP 11/20 Repair Service Bureaus PDP 11/70 IBM 370

12. Two Troubled Projects • Loop Maintenance System • IBM MVS system • PDP 11/20 backup • 8 Months to go No schedule No performance plan No requirements Weak management Switch Test Head PDP 11/20 Repair Service Bureaus PDP 11/70 IBM 370 • Loop Assignment System • Univac system • Failed trial • $250 million down the drain • Demoralized, weak team • Senior 18-month commitment No architecture or requirements No technology infrastructure • Working “temporary” subsystem • Unix based

13. Two Troubled Projects • Loop Maintenance System • IBM MVS system • PDP 11/20 backup • 8 Months to go No schedule No performance plan No requirements Weak management Switch Test Head PDP 11/20 Repair Service Bureaus PDP 11/70 IBM 370 • Loop Assignment System • Univac system • Failed trial • $250 million down the drain • Demoralized, weak team • Senior 18-month commitment No architecture or requirements No technology infrastructure • Working “temporary” subsystem • Unix based “Temporary” Subsystem “Robust Pipe” Univac 1100 with C

14. Building The Team • All - domain expertise, discipline & simplifiers • Team leads – programming, lateral & motivational skills • Architect – deep system software & computer science • Great ones are really rare “Everything should be made as simple as possible ... but not simpler” - Einstein • Specification - user-centered design & customer skills • Design/programming – proud craft persons • Read great books to learn to write, read great programs to learn to program “Lion’s Commentary on Unix” - John Lions • Testing – sadists & analysts • Performance – back of envelopers • Quality - superb communicators • Tools & environment – geeks emeritus • Product manger coordinated extended team • Sales, marketing, pricing, legal, service, … “Executives owe it to the organization and to their fellow workers not to tolerate nonperforming individuals in important jobs” - Drucker

15. Process is not a four-letter word • Iterative by many names • Build a little, test a little, refine requirements a little • Entrance/exit criteria • Templates & best cases • Lightweight • Web • Inspections & reviews • Requirements, design, code, test, etc • Lightweight to heavyweight • Especially on tough stuff and with new people • Lead customers • User-centered design • Screw-up early detection & action • Jeopardies and speaking up • Audits with smart friends “I had the world’s greatest group, and I should have made the world’s two greatest groups. I didn’t realize there are benefits to having real implementers and real users” - Alan Kay

16. Process is not a four-letter word R1 R2 R3 • Iterative by many names • Build a little, test a little, refine requirements a little • Entrance/exit criteria • Templates & best cases • Lightweight • Web • Inspections & reviews • Requirements, design, code, test, etc • Lightweight to heavyweight • Especially on tough stuff and with new people • Lead customers • User-centered design • Screw-up early detection & action • Jeopardies and speaking up • Audits with smart friends Iterative Definition/Development b1 b2 b3 b4 b5 Integration, Test, & Trial Phased Deployment • Early on • Architecture • New technology/risk • Performance • Factory operations Handoffs Automation Configuration control • Later with increasing process discipline • Features • Quality “I had the world’s greatest group, and I should have made the world’s two greatest groups. I didn’t realize there are benefits to having real implementers and real users” - Alan Kay

17. Impact of Increased Discipline 0.8 Industry-wide 0.7 Company X 0.6 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 • Low software engineering maturity results in: • Reduced productivity (35%)* • Late detection of defects (22%)* • Longer time to market (19%)* • Higher post-release defect rates (39%)* • * median annual improvement CMM** process maturity • Takes several years of focused improvement to become a high-performance software team • Focus on the programmer • Just management: well-controlled junk • Delicate touch to avoid process imprisonment CMM Level ** Software Engineering Institute’s Capability Maturity Model (CMM)

18. What’s your favorite bug?

19. Architecture “Controlling complexity is the essence of computer programming” - Kernighan • Top-down decomposition of a complex problem to allow independent, bottom-up development and change • Goals • Changeability/customizability • Chunk independence • Reduce n2 effects • Inject huge-payoff CS technology • Tiny languages, protocols, finite state machines, search, algorithms & data structures, formal methods, etc • Performance • Reuse • Check it • Prototype • Audit with smart friends • Crumble with time • “An Architecture History of the Unix System” - Feldman Usenix ‘84

20. Calling the shot • The true art • Best to wait until specification/design done • Domain expertise crucial • Sizing models help, but not magic (still +/- 30%) “Let the numbers do the talking” - Don Reifer • Aggressive but doable • Starvation not gluttony • Reduce communications, complexity & feature creep “Whilst it is true a large programming project might require a large programming team, a large programming team will always build a large project” - J. K. Buckle “Managing Software Projects” • Slips • If you must, do it once • Iterative development provides an out “A customer will always forgive you a slipped schedule or missed feature, they will never forgive you for bad quality or performance” - Buckle

21. Which Project Used 15x Staff of the Other? • Loop Maintenance System • IBM MVS system • PDP 11/20 backup • 8 Months to go No schedule No performance plan No requirements Weak management Switch Test Head PDP 11/20 Repair Service Bureaus PDP 11/70 IBM 370 • Loop Assignment System • Univac system • Failed trial • $250 million down the drain • Demoralized, weak team • Senior 18-month commitment No architecture or requirements No technology infrastructure • Working “temporary” subsystem • Unix based Temporary Subsystem “Robust Pipe” Univac 1100 with C

22. Tempo “.. the disaster is due to termites, not tornadoes; and the schedule has slipped imperceptibly but inexorably. Indeed, major calamites are easier to handle; one responds with major force…” - Brooks • The plan • Delicate balance of top/down and bottom/up • PERT to plan, GANTT to manage • Critical path control • Don’t multitask • Buffer schedule • 50% estimates • Railroad train feature loading • Leave the station on time • Features left behind • The weekly/biweekly project meeting • Milestones • All milestones are important, particularly early ones and few big ones • Build team morale, tempo, customer confidence “No plan survives contact with the enemy” - Helmuth von Moltke the Elder

23. The Project Meeting “The Screaming”

24. Whipping the rowers • Project manager style • Most decisions are 50/50 • Avoid acting like “A glacier with dignity” “An officer in charge on an Indian agency made a requisition in the autumn for a stove costing seven dollars, certifying at the same time that it was needed to keep the infirmary warm during the winter, because the old stove wore out.” Then, after glacial dignity “The stove is here. So is spring” * * “An Autobiography” - Theodore Roosevelt

25. Testing “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” - Dijkstra • Full lifecycle activity • One day build and test (tempo!) • Multi-level • Automated • Domain specific tools • Field-driven test libraries • What to test • Requirements & design • Coverage • Usability • Performance • Measure with S curves

26. Testing “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” - Dijkstra Full lifecycle activity One day build and test (tempo!) Multi-level Automated Domain specific tools Field driven test libraries What to test Requirements & design Coverage Usability Performance Measure with S curves Number Time Test Cases Run Bugs Found Bugs Closed

27. Testing “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” - Dijkstra Full lifecycle activity One day build and test (tempo!) Multi-level Automated Domain specific tools Field driven test libraries What to test Requirements & design Coverage Usability Performance Measure with S curves Buy death march whip Number Time Test Cases Run Bugs Found Bugs Closed

28. Testing “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” - Dijkstra Full lifecycle activity One day build and test (tempo!) Multi-level Automated Domain specific tools Field driven test libraries What to test Requirements & design Coverage Usability Performance Measure with S curves Buy death march whip } Number Announce new features Time Test cases run Bugs found Bugs closed

29. Get your S-curve

30. Get your S-curve You can’t make this stuff up

31. Communications & Documentation “Schedule disaster, functional misfits, and systems bugs all arise because the left hand doesn’t know what the right hand is doing” - Brooks • Project meetings • Hierarchical • Weekly or biweekly • Jeopardies are good • Documents • “The act of writing forces hundreds of tiny decisions” - Brooks • Few key ones, all template driven, all reviewed/inspected • Requirements, architecture, design, test & project plan • Strong version and configuration control • Code • Structure - aim for one pagers • Comments • “This section of code is cursed” - Stroustrup C++ • “You are not expected to understand this” - Unix

32. Metric Driven Improvement • Quality • Measures • Fault density • Service responsiveness • Improvement • Root cause analysis • Team & individual • Productivity Number Severity 4 3 2 1 4 3 2 1 4 3 2 1 …. >6 2 1 Age

33. Metric Driven Improvement • Quality • Measures • Fault density • Service responsiveness • Improvement • Root cause analysis • Team & individual • Productivity • Customer satisfaction • Huge fault density correlation Number Severity 4 3 2 1 4 3 2 1 4 3 2 1 …. >6 2 1 Age . . . Good . . Installability Usability Service Performance Functionality Quality Organization Project

34. Metric Driven Improvement • Quality • Measures • Fault density • Service responsiveness • Improvement • Root cause analysis • Team & individual • Productivity • Customer satisfaction • Huge fault density correlation Number Severity 4 3 2 1 4 3 2 1 4 3 2 1 …. >6 2 1 . Age . Project Teaches . . Good Organization Learns . . Project Learns Installability Usability Service Performance Functionality Quality Organization Project

35. A Project-Specific Metric • Project characteristics • Gluttonously staffed ~ 5,000 people • Long crumbled architecture ~ 25 years • Kryptonite-weight processes • Humongously successful • 99.9999 uptime • Down 30 seconds a year for all causes • Drug dealer cash flow

36. A Project-Specific Metric • Project characteristics • Gluttonously staffed ~ 5,000 people • Long crumbled architecture - 20+ years • Kryptonite-weight processes • Humongously successful • 99.9999 uptime • Down 30 seconds a year for all causes • Drug dealer cash flow • Proposed programmer metric Lines of code Meetings attended Improvement objective: exceed 1!

37. Contrarian Views • Design methodologies • Great designers make great designs • Methodologies make lousy designs look good • Fancy measures • Confuse precision with accuracy

38. Making It Stick • People support what they invent • Thick dictates are good only for cellophane • Bellcore’s was ten one-pagers (what not how) • Participatory multi-day course • Crusty expert led • Project team takes course • Time it with a new release schedule • Output is their plan & their process • Team mentors • Senior support

39. My Most Notable Disasters • “Hi ho silver” • Loop maintenance system • Domain ignorance • Customer ordering system “Organizations can only do damage to themselves and to society if they tackle tasks that are beyond their specialized competence, their specialized values, their specialized functions” - Drucker • Second system effect • Switch assignment system “This second is the most dangerous system a man ever designs. …The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.” - Brooks

40. Maybe a little magic • High quality, on time useful software can be built • Bellcore after six-years of continuous improvement (1994) • 95% on time/content • 3-4x competitive productivity • 10x competitive quality • It’s all about people • Picking and developing the best and getting rid of the worst • Helping them select and use good software engineering practices • Motivating and rewarding them for • Using and improving these practices • Behaving as teams • Giving them the time and resources to do their jobs • Aggressive but realistic goals • Making and holding them accountable • Crowd control can be fun

41. Things you might consider trying(Other than shooting Alfie “the shot-caller”) • Development process • Iterative development • Entrance/exit • Design template • User-centered design (pair with another team) • Inspections • Heavy and lightweight • Performance model • Defect metrics & root cause analysis • Tempo control • Project meeting • Milestones • Railroad • Jeopardy • Outside audit (pair with another team) • Changeability/customization • Automated testing • System & module • S curves • Lots of extra-bold Starbucks • Two shots

42. Final That's All Folks!!!