1 / 15

Rapid Software Development for CLEO III

Rapid Software Development for CLEO III. Martin Lohner Cornell University CHEP 2000. Overview. Switch from stable CLEO II software new software framework and new coding languages: A Daunting Task! History of Project Rapid Development Approach: Module Plug & Play

linkerj
Download Presentation

Rapid Software Development for CLEO III

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Rapid Software Development for CLEO III Martin Lohner Cornell University CHEP 2000

  2. Overview Switch from stable CLEO II software new software framework and new coding languages: A Daunting Task! • History of Project • Rapid Development Approach: • Module Plug & Play • Embedded Scripting Language(s) • Programming Languages, Compilers, Platform Support • Dynamic Loading vs. Static Linking • Library Submission and Build Procedure • Web Documentation • Workshops • The importance of Intuitive Design • Summary Martin Lohner, Cornell U A264 CHEP 2000

  3. History of Project • Mid-1996: 3 CLEO physicists decide to break w/ CLEO-II data access model and develop new framework • see A216: “CLEO’s User-Centric Data Access System” (poster) • Important Factors in Early Stage: • No Users! • Radical changes and redesigns possible • knowledge from CLEO-II background helped: • we knew what had worked and what hadn’t • Fall 1997: First Workshop • now had users to worry about • first “application” code written Martin Lohner, Cornell U A264 CHEP 2000

  4. More History • 1998-1999 Code development started ramping up quickly • Today: • 1.2 Million lines of code • 500+ libraries • daily release schedule Martin Lohner, Cornell U A264 CHEP 2000

  5. Module Plug & Play • One skeleton executable “Suez”: • allows plug & play of software modules • All modules share a common type-safe “data-bus”: • either requesting data from the bus • and/or supplying data to the bus • supplying data is done actually in terms of algorithms • Easy run-time configuration: • any number of algorithms can supply the same data • a fancy tracker, or a simple one for debugging • no need to recompile any code Great for development, great for debugging, great for any use. Martin Lohner, Cornell U A264 CHEP 2000

  6. Embedded Scripting Languages • Suez uses Tcl/Tk as its command language • great for interactive use • great for batch jobs • Internals only know about an abstract Interpreter! • Other interpreters, e.g. perl or python possible • Some higher-level functionality implemented in Tcl • faster to develop than in C++ • no difference to the user Martin Lohner, Cornell U A264 CHEP 2000

  7. Programming Langs, Compilers, Platforms • C++, Fortran on OSF1 4.x, Solaris 2.x, soon Linux/Intel • Cmpl.-/platform-spec. problems handled w/ CPP bug flags • name the bug, NOT the platform • CPP macros for STL containers and iterators • Explicit Template Instantiation: • more reliable, less error-prone • more efficient for building code • more usable for dynamic linking • Wrapped Legacy Fortran code is part of C++ framework • would be foolish to rewrite in C++ • data required is translated from C++ objects to Fortran • common blocks only allowed for in-algorithm, not for cross-algorithm communication Martin Lohner, Cornell U A264 CHEP 2000

  8. Dynamic Loading vs. Static Linking • Both equally well supported, can mix. • Static linking required for reconstruction jobs • need stable environment for long periods of time • Dynamic Linking/Loading for rapid code development • Fast turn-around time needed • Cutting link times from hours/minutes to minutes/seconds • Limit the number of libraries to link to: • Proper Layering of code • Separation of data types from the algorithms that supply them • why would I have to link to a tracker to access tracks??? • No direct links between objects reduces # of libs to link to • instead we use index-list objects (“Lattice”) • Run-time cost of resolving symbols is low! Martin Lohner, Cornell U A264 CHEP 2000

  9. Dynamic Linking/Loading (cont.) • Special care required: • Symbols have to be unique • if load two modules linked with some of the same symbols, the load order shouldn’t matter! • Beware of interface changes (added member data, virtual methods) • changed memory layout • don’t mix! • Solution: • Shared libraries versioned via cvs tag • e.g. “libMyTracker.so libMyTracker.so.v01_03_04” Runtime loader complains if module linked with different version! Martin Lohner, Cornell U A264 CHEP 2000

  10. Library Submission and Build Procedure • Ideal System: • allow users to submit changes at any time • system always stable and usable • always up-to-date Conflicting Goals! • Standard tools: • gmake, cvs • BaBar/Fermilab’s SoftRelTools package + our own scripts on top • package area separate from release area • Submission Procedure: • tag all files in a library with same new cvs tag • submit changes to cvs at any time, but only tagged changes enter the release system Martin Lohner, Cornell U A264 CHEP 2000

  11. Library Release Builds • Two daily release builds: • “test” release test-builds all the latest tags • “current” release with all the latest tags that • passed “test” compile/link/test cycle successfully • and no “important” packages failed normal lag time of 1 day for rotation from “test” to “current” • in fact two (!) current releases: • if new build is finished by 7am, switch • Strong reliance on extensive testing procedures • individual feature testing • mock reconstruction, MC, etc. jobs • Dated releases: • available and stable for months • required when we enter the reconstruction era Martin Lohner, Cornell U A264 CHEP 2000

  12. Documentation • Web-based documentation: • design principles • hands-on “howtos” • detailed descriptions of libraries • document designs • People have followed positive examples of framework developers -- Surprise! • Auto-generate documentation from code: • doxygen • doc++ • mozilla tools (lxr etc.) Viewable via web browser or ala “man” pages with script using lynx! Martin Lohner, Cornell U A264 CHEP 2000

  13. Workshops • Several focused 1-day workshops: • talks mixed with • hands-on experience at terminals Enthusiastic Response! • Goals: • introduce people to new framework • gain feedback from people about framework • New users strongly encouraged to do workshops • available online • limit interruptions of experts Martin Lohner, Cornell U A264 CHEP 2000

  14. The Importance of Intuitive Designs • Encourage people to give feed-back on ease-of-use • Have accommodated many requests for simplification! • Keep it consistent! • Be willing to back out of changes! “If it’s too hard to use, let’s change it”! Martin Lohner, Cornell U A264 CHEP 2000

  15. Summary: Successful Break with the Past • Rapid development approach • Module Plug & Play! • Dynamic loading! • Core group of developers needed early on • no users! • Advantage of a lean-and-mean development group: • few meetings • easy decision-making process • few experts to have to consult with What would have happened with 10 times the people? Martin Lohner, Cornell U A264 CHEP 2000

More Related