Towards an Open Source Project for Online Software

1. Towards anOpen Source Project forOnline Software Bob Jones CERN EP/atd Robert.Jones@cern.ch

2. Brief overview of ATLAS DAQ/EF Prototype -1 Project Structure and organisation of the Back-end DAQ sub-system Lessons learnt Proposal to the HENP community: An Open Source project for online software Issues involved How to proceed Summary Contents Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

3. ATLAS DAQ/EF Prototype -1 • Goal • to produce a prototype DAQ system for evaluating candidate technologies and architectures. 1996-2000 • Organisation • functionality divided into sub-systems • Information • overview, tech. notes, conf. papers etc. • http://atddoc.cern.ch/Atlas/ Detector Interface Data Flow Event Filter Back-end Software Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

4. Back-end Sub-system • Excludes the processing and transportation of physics data • Interfaces to: • Data-flow • event builder, ROB and • detector ROD crates • Triggers • Lvl1, Lvl2, event filter • DCS • SCADA Software for configuring, controlling & monitoring the online system It is the software glue of the experiment Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

5. Back-end Software Issues • How to... • Start, stop & synchronize 100s programs on 100s processors? • Configure the programs for many data taking modes? • Identify and report problems when running? • Integrate software from many different sub-systems • While... • Minimizing the time taken to start/stop a run • Simplifying the information presented to the operator • Given that... • Rapidly evolving environment (e.g. UNIX/WNT, Java/C++) • Not all software can be produced at CERN Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

6. Back-end Organisation Collaboration of some institutes in ATLAS Trigger-DAQ system • Univ. of Innsbruck, Austria • IN2P3 Marseille, France • NIKHEF, Netherlands • LIP, Portugal • IAP Bucharest, Romania • JINR Dubna, Russia • PNPI St. Petersburg, Russia • CERN, Switzerland • Univ. of Geneva, Switzerland Functionality divided into a number of components Institutes responsible for the development and maintenance of components within a well-defined framework Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

7. Back-end Components Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

8. Back-end Design Approach • Activities • Concentrate development effort on domain specific items • Use external software when possible • Use common solutions across all components • Adhere to relevant standards (e.g. OMG, ANSI etc.) • Design for portability from the start • Use Software engineering principles, OO techniques and follow a well-defined software process Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

9. SW Development Process • Not formally defined in a document but web pages • - more dynamic • - less imposing • Basic but practical • Used for all the components Phases Reviewed Deliverables Collect Requirements Requirements Evaluation report(s) Pre-design Investigations High-level Design High-level Design Code Users Guide Test Plan Impl. Note Detailed DesignImplementation Testing &Integration Test programs Test report Software Development Environment Training, Programming and Testing Tools, FrameMaker, html, SRT Configuration Management, StP/OMT CASE TOOLS Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

10. Back-end Status • All components developed and deployed (except Evt Dump) • 8 releases • 0.0.0 - Jan’99 … 0.0.7 - Dec’99 • Integrated with data-flow • use in test-beam summer 2000 • Evaluated and accepted by Level-1 & 2 triggers • 0.0.7 (James Bond) • 800K lines of code • 600K external & 200K developed in-house • C++, Java, shells, IDL, CHSM, CLIPS, QRL Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

11. Lessons Learnt: Organisation • Component model is effective • Clear interfaces allows parallel development by multiple groups • Small group dedicated to each component (max. 4 developers) • One coordinator for each component • One institute per component works best • Component development based on agreed priority (core components first) • Look for commonality between components - avoid duplication • Centralised integration is best • Need people and tests dedicated to this task • Must have good collaborative tools • Code repository, release management, web site, mailing lists etc. • External software (Corba/ILU, Tools.h++, CmdLine, CHSM etc.) • One developer responsible for each external package Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

12. Lessons Learnt: Software Process • Software process is important • do start gently (can’t go from chaos to Nirvana in one step) • do keep it simple and stick with it (don’t abandon ½ way through) • do review/inspect every deliverable (requirements, design, code etc.) • do provide templates, checklists and examples for all deliverables • do get a non-author to perform component testing • do encourage developers to share their work and ideas • Things to avoid • don’t ask developers for a deliverable unless it will be used • don’t underestimate time & effort for sw mgmt, integration & test. Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

13. Lessons Learnt: Software Releases • Each one must be better than the last • Iterative development model reduces errors and risks • Use feedback to drive/focus work • Nightly on all supported platforms - build & test • Official - coincide with Back-end meetings: discuss status of last release and contents of the next • More platforms == more work • Implies every developer has access to every platform • Try to keep the list short - agree within experiment • Software librarian != developer • Librarian is not there to fix faults in the software • Web page gives access to build log for each component A release is an important milestone and a lot of work for everybody Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

14. Towards an Open Source Project • Motivation • Similar solutions to the same problems in many groups & experiments • Online groups are getting bigger (ATLAS: > 50 institutes from 17 countries) • Commodity hardware (PCs) and software (Linux) now being used in online systems • Open Source projects have proved they can deliver high-quality reliable software Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

15. Proposal to HENP Community:Use Back-end as basis of Open Source Project • Justification • Software is general - can be use in other projects & experiments • Open Source project-like culture and organisation - would not require many changes • Participating Institutes want to use the software they have developed on other projects • Modular architecture and copious doc. means developers can learn the internals quickly • Developed using sw. eng. principles, OO techniques and a defined sw. process - a sound base for extensions Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

16. Issues: Selection of a license • Why do we need a license? • Guarantees • developers’ contributions are acknowledged • software remains open • no warranty - can’t sue us if it does not work • Which one to choose? • GNU GPL & Lesser GPL, BSD, NPL etc. • LGPL would be my personal choice: • Can be mixed with non-free software - e.g. SCADA • Code that begins free remains free - can’t make mods. private • Cannot be re-licensed by anyone • Must have a legal basis • Need help from the men in suits Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

17. Issues: Use of Third Party Software • Each external package has its own license policy • ILU: “unlimited reproduction modification, distribution…” • ACE: open source - being phased out • CHSM: GNU GPL • CLIPS: “freely used and distributed without restrictions…” • CmdLine: “freely copy and redistribute…” • SRT: based on CVS and GNU make (GPL) • Motif: Open Group Master Software License - use LessTif (LGPL) • Tools.h++:cannot distribute source - replace with STL • Objectivity/DB: not in SRT & not distributed - only needed by OBK • Java: Get Internal Noncommercial Use Src. License from Sun if needed • Licenses must be respected and compatible Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

18. Issues: Support • Continue to distribute official releases on CD-ROM • Fully tested and documented- for short list of platforms • Support level to be defined by participating experimnts. & institutes • Source code and tools available to build own version • No direct support but participants will assist on a best effort basis • Experiments still need online groups! • Back-end software will not cover all the needs of each experiment • Treat Back-end as a third-party software suite to be integrated with experiment specific software • Online groups remain responsible for ensuring the Back-end software works for them on their experiment Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

19. Issues: Organisation • Similar to existing Back-end organisation • Components built from packages • Prefer one institute per component/package • One coordinator per component/package • Handles bug reports, change requests, patch submissions etc. • Only coordinator and trusted developers have write access to the code repository • External package - coordinator tracks and selects best version • Distributed development & centralised integration • All major modifications and additions must: • Follow the software process • Meet the quality criteria • Adopt the same software license Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

20. How Can You Participate? • Testing and Problem Reporting • You have access to full doc., test programs and source code • pin-point faults so they get fixed quicker • suggest fixes and improvements • Reading doc. & source code • you are helping to inspect/review the software • Code Development • Extend functionality of existing components • Add new components • Port to new platforms Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

21. Why Should You Participate? • For the Boss - i.e. person responsible for online sw. in an experiment • Quick start-up - Working software already available • Long-term survival • Safety-net - source code available • Software license - means it will always be available to you • Less software to be developed by you • Contributing extensions, patches etc. means everyone benefits & improves quality • For the Developer - i.e. person willing to take part in the development • Chance to work on a global project - doing what you enjoy/know best • Merit based organisation - your work is more visible • Easier to get jobs - same sw. used in many institutes/experiments Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

22. How to Proceed • Try the James Bond (0.0.7) release - to see how much interest • CD-ROM contains header files, libraries, binaries (Linux, Solaris & LynxOS), full documentation • Source code (via LXR tool) & class diagrams visible on the web where permitted by license • No commitment from you or us • Give feedback - used to improve subsequent official releases • Next Step - if sufficient interest • Seek agreement from relevant bodies • Work on turning the proposal into reality • Training session (mixed hands-on and presentations) • To be held at CERN: 2 days during week of April 10th • Will be put on the web (Web Archive Project) & distributed (CD-ROM) Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

23. Summary • Back-end software successfully developed in a distributed, collaborative manner • Open Source Project Proposal to HENP community • Given sufficient interest we are prepared to make the Back-end software the basis of an Open Source project for online software • Issues will be addressed in forthcoming official releases • Steps are incremental improvements to the existing software and organisation - not fundamental changes • Based on level of interest we will seek approval from appropriate bodies • More info: http://atddoc.cern.ch/Atlas/ • Live demo: this evening 18:30 - 19:30 after session F in location P5 • Back-end scalability and performance: Igor Soloviev B329 Thurs. 17:00 • Back-end inspections and reviews: Doris Burckhart F331 Thurs. 17:50 Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones