1 / 16

CHEP 2003 General Summary

CHEP 2003 General Summary. Torre Wenaus, BNL/CERN CHEP 2003, UC San Diego, La Jolla March 28, 2003. Themes and observations. Lesson from the past: Make it simple (R. Brun) No more complex than necessary Users want consolidation, ease of use, and stability

ketan
Download Presentation

CHEP 2003 General Summary

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CHEP 2003 General Summary Torre Wenaus, BNL/CERN CHEP 2003, UC San Diego, La Jolla March 28, 2003

  2. Themes and observations • Lesson from the past: Make it simple (R. Brun) • No more complex than necessary • Users want consolidation, ease of use, and stability • Must consider also needs of the future; longer view of maintainability and evolution • In the interests of long term stability • OO and C++ is the accepted paradigm • No major OO/C++ migration or usage angst at this conference, it is done and accepted • Offline and online: “Triumph of C++ for HEP DAQ confirmed” – DAQ summary • Now we are hearing reports on Nth generation C++ software • L. Sexton Kennedy, CDF: Every component has been rewritten at least once. Implementations have now stabilized such that every new arrival doesn’t start by discarding and rewriting software • “Many more talks about redesign than about design” – Data management summary • And on the maturation and emergence of tools as broad standards, after years of development and refinement • e.g. Geant4, ROOT I/O

  3. Themes and observations • The tyranny of Moore’s Law • Wolbers: it is not a substitute for more efficient & faster code, smaller data size • it works against thinking before doing • Optimize wherever possible • Addressing the digital divide in networking (H.Newman) • HEP is obligated as a community to work on this • A world problem in which our field can have visible impact • Farm challenges • Don’t underestimate farm installation and operations (R.Divia) • Big issues are power, cooling, space! (S.Wolbers) • Watts/$ steadily rising (R.Mount) • Tape-disk random access performance gap in analysis is receding as an issue, but disk-memory gap is hardly being addressed (R.Mount)

  4. Rising trends • ROOT • For analysis, I/O, and much else • Now fully supported at CERN: EP/SFT section • Close interaction with experiments on new developments • Run II, RHIC, ALICE, LCG, BaBar, … • Foreign classes, PROOF, geometry, grid integration, … • Mentioned in 47+ talks at this conference • Open source databases (MySQL, Postgres, …) • Metadata, distributed computing, conditions, … • Empowering software: easy and potent • MySQL mentioned in 37 talks! Postgres in 8, Oracle in 27 • Online – offline continuum • Similar Linux farm environments, attainable time budgets • Same framework, maybe same algorithms, in HLT as in offline (V.Boisvert, ATLAS) • Stringent performance/robustness requirements on software

  5. Rising trends • Common projects • Joint projects one of the CDF/D0 successes (Wolbers) • But hard to align running experiments with LHC • LHC Computing Grid project • Grid projects in general • Laudable but difficult; increasingly forced by the circumstances • Resource constraints and increasing scale and complexity makes go-it-alone N times too costly • cf. comments in online/DAQ context by G. Dubois-Feldmann today: somewhat less success in online where it is even harder than offline, but possible LHC inroads • Related is software reuse… • Respect what we know about long software development timescales

  6. Rene’s time to develop plot LCG?

  7. LCG must effectively re-use and leverage existing software, or fail This is the approach taken: cf. POOL, SEAL talks. Time will tell! cf. next CHEP LCG?

  8. Rising trends – The Grid • The central importance of distributed computing to future (increasingly, present) HENP is long known • ‘The Grid’ as the means to that is now established • Major, broad successes in funding and in attracting collaboration with CS • F.Berman, Grid 2003: “HEP has set a model for integration, focus, coordination” • Progress in applying Grid software and infrastructure to real problems • Batch production • Clearly the chosen path; success to be proven, but has promise and broad commitment

  9. The Grid • F.Berman, Grids on the horizon: • Must be useful, usable, stable; supported • More cooperative than competitive • [Not always the case today!] • Applications are key to success • Not a “Field of Dreams” “build it and they will come” R&D field any more • Grid killer app: a focus on data. Good match to us • Still a long way to go

  10. The Grid • Miron Livny: • Benefit to science: democratization of computing • Still very manpower intensive: when the support team goes on holiday, so does the Grid (CMS testbed in Dec) • Best practice middleware requires • True collaboration, “open minds” (cf. Berman) • Testing, deployment/adoption, evaluation metrics, robustness, professional support, longevity, responsiveness to show stoppers, … • Much to do and improve but important progress • E.g. VDT as standard middleware suite

  11. Receding trends • Objectivity and ODBMS in general • “Jury still out” at CHEP 2000 (P.Sphicas), but now clear • Objectivity dropped or being phased out by LHC experiments, COMPASS, BaBar event store • In PHENIX “becoming a liability” (compiler issues); augmented with RDBMSs • Not due to technical failure but a mix of technical problems, commercial concerns, manpower costs, availability of an alternative • Its replacements are not other ODBMSes but files (often ROOT) + RDBMS (mySQL, Oracle, Postgres…) for metadata • Magnetic tape (apart from archival) • PASTA: “unlimited” multi-PB disk caches technically possible but true cost is unclear (reliability, manageability) • File system access under urgent investigation • “tapes as random access device no longer a viable option” – large disk caches needed for LHC analysis

  12. Receding trends • Commercial software? No… • Some in decline (Objy, LHC++), but new prospects opening (IBM, Sun, MS, …) in Grid • Open source now has an important commercial element we derive great benefit from (even post-.com crash) • Red Hat, MySQL, Qt, …

  13. Underrepresented • Collaborative tools • Was represented this week, but only lightly • Vital for distributed collaboration on software development and physics analysis • H. Newman: need culture of collaboration • Distributed and remote collaboration should be the norm • Not solely, or even predominantly?, a matter of tool development in the community • How is the exponential commercial side evolving and how can we leverage it • What is the evolutionary path, strategy, role for community-developed tools such as VRVS • Why is the user experience often poor • Poor physical facilities/configurations, instabilities, heterogeneous tools/protocols, support issues, … • Current experience sometimes competes unsuccessfully with the telephone, despite all the shortcomings

  14. Concerns • Data analysis as “the last wheel of the car” (R. Brun) • Clear message from current generation (e.g. Run 2, BaBar): don’t leave data analysis systems and infrastructure too late, it will lead to problems • Vastly more true when we are talking about doing globally distributed analysis, for the first time • with unprecedented volume and complexity, e.g. Terabyte scale at the LHC • Making dist analysis both very difficult and mandatory • We cannot bootstrap ourselves into a global analysis system, it will take long incremental work, so we better be working in a coordinated & effective way now • R. Brun: Will not converge on one system; will be multiple competing systems, and that will not be bad [hopefully a small number]

  15. Concerns • Are we doing enough to ensure senior people can contribute directly to physics analysis? • How do we interpret the fact (R. Brun) that PAW usage is still rising? • Has everyone bought the C++/OO paradigm shift? • Are we developing and/or providing the right tools? • Is there enough engagement of senior physicists in the (limited) exploratory work being done on future physics analysis environments? • Almost certainly no, and may be difficult to attract their attention unless/until attractive prototypes can be turned loose on them

  16. Conclusions (2) • Grids and networking are making great strides • HENP is a successful and valued partner with CS • We provide a community focused on challenging large-scale deployments in real research settings • But Murphy’s Law is a potent adversary today; far from robust transparency, and much much more to do • Global collaborative computing must become a successful norm for us • Down to the global researcher at the home institute • Rich leadership potential for our field • Important new common endeavours like the Grid and LCG have much invested in their success… will be interesting to measure the degree of success at next CHEP

More Related