Overview of the Software Process

1. Overview of the Software Process Bill Lorensen GE Research (retired) bill.lorensen@gmail.com Open Source Software Practice

2. Course Wiki http://public.kitware.com/OpenSourceSoftwarePractice Open Source Software Practice

3. Outline • Motivation • The Process • Case Study – Insight Toolkit • Lessons Learned Open Source Software Practice

4. A Personal Look Back • 1976- NASA’s Computer Software Management and Information System (COSMIC) • 1978 - MOVIE.BYU • 1984 - GE Research Workstation • 1984 - LYMB • 1994 - Visualization Toolkit • 2000 - Insight Toolkit Open Source Software Practice

5. Flattener #4 Open Sourcing Self-organizing collaborative communities Open Source Software Practice

6. Open Source Anecdote * IBM Manager: “So, walk me through the development process for e-commerce. What’s the underlying web server?” Developer: “It’s built on top of Apache.” Manager: “Apache?” Developer: “A shareware program for web server technology, produced for free by a bunch of geeks working online in some open-source chat room.” Manager: “How do you buy it?” Developer: “You download it off a Web site for free.” Manager: “Who supports it if something goes wrong?” Developer: “I don’t know, it just works!” *from The World is Flat Open Source Software Practice

7. Successful Open Source Projects Open Source Software Practice

8. Visualization Toolkit - vtk Open source toolkit for scientific visualization, computer graphics, and image processing www.vtk.org Open Source Software Practice

9. Computer Vision Library - vxl • Based on Target jr and Image Understanding Environment • Numerics • Imaging • Geometry • Camera models vxl.sourceforge.net Open Source Software Practice

10. Slicer • Developed by Brigham and Womens Surgical Planning Lab • User interface plus plug-ins for applications • Segmentation • Registration • Image Guidance http://www.slicer.org Open Source Software Practice

11. SCIRun • Developed by U Utah SCI Institute • Computational Workbench • Visual Programming • Modeling, Simulation and Visualization http://www.sci.utah.edu Open Source Software Practice

12. National Library of MedicineSegmentation and Registration Toolkit $14 million over 6 years Leading edge algorithms Open Source Software www.itk.org Open Source Software Practice

13. Open Source Menu for Success • A Community with a common vision • A pool of talented and motivated developers/scientists • A mix of academic and commercial • An organized, light weight approach to software development • A leadership structure • Communication • A business model Open Source Software Practice

14. Project Stages • Startup • Early • Middle • Mature Open Source Software Practice

15. Project Stages - Startup • Define goals and/or requirements • Select developer community • Define developer roles • Select host server • Roll your own • SourceForge, CollabNet, GForge • Select software development tools • Revision control system • Compilers • Code coverage • Dynamic analysis • Build system • Testing system • Select an open source license Open Source Software Practice

16. Project Stages - Early • Refine tools • Change ASAP if necessary • Establish coding guidelines • Setup communication channels • Mailing lists • Developers • Users • Establish web presence • Project home pages • Wiki for developers • Establish nightly build/test Open Source Software Practice

17. Project Stages - Middle • Establish release mechanisms • Establish policy to add additional developers • Establish backward compatibility policy • Establish procedure to add new features • Marketing • Papers • Tutorials • Web buzz Open Source Software Practice

18. Project Stages - Mature • Project has an installed base • Everything should be hard to change • Code • Tools • Process Open Source Software Practice

19. Inside Insight (itk) Open Source Software Practice

20. What is itk? • A common Application Programmers Interface (API). • A framework for software development • A toolkit for registration and segmentation • An Open Source resource for future research • A validation model for segmentation and registration. • A framework for validation development • Assistance for algorithm designers • A seed repository for validation case studies Open Source Software Practice

21. Insight - Open Source Products Open Source Software Practice

22. itk by the Numbers • 400K • # of lines of code • 100K • # of lines of test code • 45K • # of lines of examples • 160K • # of lines of Applications • > 300 • # weekly t-cons • 85 • # unique developers • March 2000 • First code checkin • 1600 • # of nightly builds • 1200 • # tests run nightly • 50 • # of platforms • 730 • # of classes • 2000 • # of files with code Open Source Software Practice

23. itk by the Numbers • 186 • # of subscribers to the developers mailing list • 900 • # of subscribers to the users mailing list • 350 • # of monthly posts to users-list Open Source Software Practice

24. http://ohloh.net Open Source Software Practice

25. http://ohloh.net Open Source Software Practice

26. A Common Vision Create a dynamic, self-sustaining, public domain and extensible toolkit that will empower researchers throughout the world to develop new segmentation and registration algorithms and create new applications that leverage the NLM’s investment in the Visible Human Male and Female data sets Open Source Software Practice

27. The Original Team • Six prime contractors • Industrial • Kitware (Schroeder) • Insightful (Ng) • UPenn (Gee) • GE Research (Lorensen) • Brigham and Womens (Kikinis) • Academic • UNC (Aylward) • Pitt (Stetton) • Utah (Whitaker) • Rutgers (Metaxas) • Columbia (Imielenski) • UPenn (Udupa) Open Source Software Practice

28. Insight Consortium • A competitive process selected the development team • Each group was evaluated individually, without regard to how they might integrate with each other • Each team had strengths in software development, validation or algorithm development, but no one group had the necessary skills to do the entire project • Distributed software developers with varying software engineering experience Open Source Software Practice

29. Insight Software Architecture • Object-oriented design • Generic Programming • Design Patterns • Frameworks • Separation of Algorithms from Interfaces Open Source Software Practice

30. Object-Oriented Design • Dominated software systems throughout the 1990’s • Continues to be the accepted software design technique • Particularly useful for dealing with complexity • Provides programmatic abstractions to deal with generalization and encapsulation • C++ and Java have mechanisms to support OOD Open Source Software Practice

31. Generic Programming • Organize libraries consisting of generic—or reusable—software components. • The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms. • ITK uses generic programming to process n-dimensional “images”. Open Source Software Practice

32. Design Patterns • Good object-oriented software systems have recurring designs (patterns) that occur frequently • ITK employs a number of powerful design patterns • object factories • command/observer • smart pointer memory management Open Source Software Practice

33. Frameworks • Define how a group of participants can be put together to solve a particular task. • Particularly suitable for describing complex flows or algorithms that have a number of steps that can be varied • ITK Frameworks • A demand-driven data processing pipeline that connects algorithms to process n-dimensional image data • Registration framework • Level-set framework Open Source Software Practice

34. Registration Framework Open Source Software Practice

35. Level-Set Framework Open Source Software Practice

36. Separation of Algorithms from Interfaces • Implement the algorithms with a clear separation from the applications and especially the user interfaces. • Uses the Command/Observer design pattern that permits applications to watch for significant events during the execution of an algorithm • ITK has no built-in visualization, but has been interfaced to several systems including 3D Slicer, Analyze, SciRun and Volview. Open Source Software Practice

37. In search of a new software development process Open Source Software Practice

38. Google Hits “Extreme Programming”: 1,860,000

39. Extreme Programming Open Source Software Practice

40. A Light Weight Software Engineering Process • Based on the new Extreme Programming process • High intensity design, test, implement cycle • Supported with web-enabled tools • Automated testing integrated with the software development Open Source Software Practice

41. Extreme Programming Compresses the standard analyze, design, implement, test cycle into a continuous process Open Source Software Practice

42. A process supported by a suite of portable, open source tools • Apache, perl, php • Web services • cvs, subversion • Revision control • MediaWiki • Collaborative content capture • Doxygen • Automated documentation • CMake • Cross-platform program build • Dart • Continuous and distributed test reporting Open Source Software Practice

43. Extreme Programming The community owns the code Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code In the end, all of the software should appear as though one author wrote it Open Source Software Practice

44. Extreme Programming Release early, release often Although developers are tempted to keep their code under wraps until it is perfect, the process encourages them to release their code as soon as it passes some minimum tests The longer the code is visible to the community, the better integrated it will be Open Source Software Practice

45. Extreme Programming Continuous integration There is no scheduled porting to computer platforms All new software builds supported platforms every evening Open Source Software Practice

46. Extreme Programming All developers agree to keep the software defect free Although everyone is encouraged to submit their code early, the code must compile and pass tests nightly A continuous build process sends e-mails to developers who check in code that does not compile More effectively, the community enforces the commitment though peer pressure Open Source Software Practice

47. Insight - Development Cycles • Daily – dashboard • Weekly – telephone conferences • Periodic – architecture reviews • Quarterly – developer meetings • Yearly – work assignments Open Source Software Practice

48. Extreme ProgrammingDaily Testing Is The Key • Testing anchors and drives the development process (Dart) • Opens up the development process to everyone • Developers monitor the testing dashboard constantly • Problems are identified and fixed immediately • Developers receive e-mail if they“Break the Build” Open Source Software Practice

49. Results posted on web(the dashboard) How DART Enables Collaboration CVS maintainssource code revisions DART compilessource code, runs tests CVS Developers check-in code Developers review results Open Source Software Practice

50. Dart Dart Open Source Software Practice