GridPP Collaboration Meeting 14 Feb 2002

1. Tony Doyle GridPP Collaboration Meeting 14 Feb 2002

2. Starting Point Project Management Structures Experiments Board (EB) CERN and the LHC Computing Grid (LCG) Technical Board (TB) Project Management Board High Level Deliverables Staff Effort and Recruitment DataGrid “Inter Grids” 2002 Challenges UK Testbed = “The Grid” Summary Agenda GridPP Status Report Tony Doyle - University of Glasgow

3. GridPP Management Who’s who?.. c.f. Steve’s web list of contacts Tony Doyle - University of Glasgow

4. Collaboration Board Chair Steve Lloyd Birmingham Peter Watkins Liverpool Themis Bowcock Bristol Greg Heath Manchester Robin Marshall Brunel Peter Hobson Oxford Susan Cooper CERN Les Robertson Queen Mary Tony Carter Cambridge Janet Carter Royal Holloway John Strong Durham Mike Whalley RAL Ken Peach Edinburgh Steve Playfer Sheffield Neil Spooner Glasgow David Saxon Sussex David Wark IC Peter Dornan Swansea Chris Alton Lancaster Peter Ratoff UCL Peter Clarke Project Leader (in attendance) Tony Doyle e-Science Director (in attendance) Neil Geddes • Members are 'Group Leaders' of all GridPP Institutes • Chair elected by Board Tony Doyle - University of Glasgow

5. Experiments Board • ATLAS Roger Jones, Andy Parker • CMS Dave Britton, Dave Newbold • LHCb Nick Brook (Chair), Glenn Patrick • ALICE Ian Bloodworth • BaBar Roger Barlow (Deputy Chair) • D0 Iain Bertram • CDF Todd Huffman • Others Dan Tovey • Theory Alan Irving • CERN Jamie Shiers • TB Cross Members: Pete Clarke, tbd • Members chosen by Experiments • Chair and Deputy elected by Board Tony Doyle - University of Glasgow

6. Recruitment Timetable • 8th October - EB input deadline • Inputs: • 1. TECHNICAL CASE Complete • 2. ASSESMENT OF HOW MUCH EFFORT IS REQUIRED Complete • 3. WHO WILL CONTRIBUTE TO/LEAD THESE DEVELOPMENTS Complete • 4. WHAT IS REMAINING TO BE DONE - I.E. A POSTS REQUIREMENT Complete • 5. A JOB SPECIFICATION OF THE OUTSTANDING POST(S) INCLUDING DELIVERABLES AND MILESTONES Complete • 19th October - TAG assessment document • Next step - technical deployment issues • 26th October - PMB definition of tasks • Next step - deployment/resource issues • first review of CERN posts • 2nd November - Tender document • 5-6th November - Collaboration Meeting • 30th November- Deadline for submission by Institutes • 7th December - PRSC mtg. • 14th December- Oversight Committee mtg. Ambitious timescales. Experiments Input? “good” to “very good” Tony Doyle - University of Glasgow

7. Experiments Board Nick Brook • New phase for EB – moving from focussing on post “creation” to the need for exploitation and utilisation of GridPP resources • Remit: • To ensure that the GridPP project is ‘experimentally-driven’ • ‘Long term’ planning – assessment of needs and reqts from GridPP • Dissemination of experiences and information between UK collaborations • To facilitate close working relationship between expts. • Overall liaison between the expts & GridPP Tony Doyle - University of Glasgow

8. EB Meetings Nick Brook • Next meeting • Late March, after LCG startup workshop • Areas of discussion: • status of recruitment • status & experience of Grid work • long term plans (changes to original submissions) • reporting procedures to GridPP • testbed experiences • management & "booking" of  UK Grid resources • quarterly reporting to TB Tony Doyle - University of Glasgow

9. CERN Status Tony Cass • The LCG Project is up and running • PEB, SC2 and POB have all met. • Launch Workshop, 11th-15th March • Hardware for 2002 Prototype is available • Scheduling ongoing. • RTAGs launched for Persistency, Managing Software, Maths Libraries and Grid Requirements. • Initial report from Persistency and Managing Software RTAGs at launch workshop. • RTAG reports will determine assignments for GridPP staff in relevant areas. • Fabric Management Activity Plan being prepared • Similarly, this will determine assignments for GridPP staff working on Fabric Management. Tony Doyle - University of Glasgow

10. CERN concerns Tony Cass • Ensuring compatibility between EU and US testbeds • Understanding impact if first physics moves to 2007 • On Production system • On LCG Phase I • Scheduling of the Prototype • Understand needs and types of challenge - pushing the envelope or mass production for LHCC milestones • Fitting these in resources available • Tight schedule for new tape drives for ALICE • And we need to support running experiments in a reduced budget • Scheduling to allow test of Grid tools across multiple sites. Tony Doyle - University of Glasgow

11. Technical Board Pete Clarke • Composition to date (reflects dynamic nature): • Chair Pete Clarke • Deputy Robin Middleton • WP1 Dave Colling • WP2 Gavin McCance • WP3 Steve Fisher • WP4 Lex Holt/Paul Anderson • WP5 John Gordon • WP6 Andrew McNab • WP7 Richard Hughes-Jones • WP8 Frank Harris • Middleware Alex Martin • Experiments Steve Burke • Security Dave Kelsey • EB Nick Brook Tony Doyle - University of Glasgow

12. TB Meetings Pete Clarke • December • adoption of DataGRID reporting mechanisms for WP1-7 (3-monthly reports) • Jan 9th • EDG: Post quarterly submission review of delivered resources • EDG: Review of UK deliverable responsibilities • Gantt chart : testbed functionality • Testbed rollout planning: first discussion • Application builders workshop planning • adoption of TB-SUPPORT@JISCMAIL.AC.UK • Feb 8th • Detailed discussion of Testbed rollout planning => SEE TOMORROW • First discussion of non-LHC exp support • GridPP DEVELOPMENT Grid Tony Doyle - University of Glasgow

13. Input = 3-monthly report A work-segment is a structure something like: {Quarter, Person, Institute, Experiment, Task, StaffMonths} for example: {2002Q1, P.Mealor, UCL, NONE, T7.2, 3.00} Extract information such as: 1) Total delivered effort as a function of time. 2) Effort as a function of task or workgroup. 3) Effort as a function of Institute or Experiment (as a function of time). 4) Histograms of any of these quanitites (e.g. effort vs workgroup or Institute or experiment). Resource Monitoring Dave Britton Next three-monthly reports in March (~18th).. • Technically - Input into Excel (visual basic macros) • Need to integrate into project management tools • Need to match to high level goals Tony Doyle - University of Glasgow

14. TB Documentation Pete Clarke • EDG resource management document produced and circulated • WP1-7 mechanisms now in place and being used usefully • Applications posts monitoring mechanisms now being reviewed • Testbed rollout planning and support document in preparation • iterated within TB • next step is circulation to EB and WPs affected (WP6) for comment • will then be circulated to all for comment • to be outlined by PC tomorrow [Note: TB-WWW pages being fixed now] Tony Doyle - University of Glasgow

15. TB Meetings Pete Clarke • Application Builders meeting 31st Jan @ RAL • aim was “how to use middleware” talks where appropriate (e.g. WP1) • in addition overview of what other middleware does • All future meetings will be as requested by GridPP (i.e. the audience) • Will probably only cover one theme • Next planned is half day (or more) on use of Job submission system • We will attempt to make the format more “training oriented”(perhaps use RAL or IC training rooms) • Will circulate request for interest (6 weeks from now) • Use other meetings, where appropriate e.g. RAL (SRB) and NeSC e-Science Institute (Globus).. • But try to minimise.. Tony Doyle - University of Glasgow

16. Tony Doyle - University of Glasgow

17. Project Management Board • Project Leader Tony Doyle Appointed by CB • Deputy Project Leader John Gordon Appointed by Project Leader • Chair of Collaboration Board Steve Lloyd Ex officio • Chair of Technical Board Peter Clarke Appointed by Project Leader • Chair of Experiments Board Nick Brook Ex officio • Deputy Chair of EB Roger Barlow Ex officio • Resource Manager Dave Britton Appointed by Project Leader • DataGrid UK Project Leader Robin Middleton Ex officio • CERN Liaison Tony Cass Appointed by CERN • e-Science Director Neil Geddes Ex officio • As proposal plus Resource Manager + DataGrid UK Project Leader + Part time Secretary at Glasgow • Three year posts (except EB 1+2) • Meets weekly via VC and monthly face to face Tony Doyle - University of Glasgow

18. High Level Deliverables To be re-defined to extend to “The Grid” i.e. testbed Aspiration: all that you expect from a central service? Probably not.. Tomorrow’s Discussion.. Tony Doyle - University of Glasgow

19. High Level Deliverables Requires input from WP contacts Tony Doyle - University of Glasgow

20. High Level Deliverables Pre-defined at high level, but requires further iteration Tony Doyle - University of Glasgow

21. Projected Staff Effort [SY] AreaGridPP @CERN CS WP1 Workload Management 0.5 [IC] 2.0 [IC] WP2 Data Management 1.5 [Ggo] 1.0 [Oxf] WP12 WP3 Monitoring Services 5.0++ [RAL, QMW] 1.0 [HW] Security ++ [RAL] 1.0 [Oxf] WP4 Fabric Management 1.5 [Edin., L’pool] WP5 Mass Storage 3.5++ [RAL, L’pool] WP6 Integration Testbed 5.0++ [RAL/M’cr/IC/Bristol] WP7 Network Services 2.0 [UCL/M’cr] 1.0 [UCL] WP8 Applications 17.0 ATLAS/LHCb (Gaudi/Athena) 6.5 [Oxf, Cam, RHUL, B’ham, RAL] CMS 3.0 [IC, Bristol, Brunel] CDF/D0 (SAM) 4.0 [IC, Ggo, Oxf, Lanc] BaBar 2.5 [IC, M’cr, Bristol] UKQCD 1.0 [Edin.] Total 36.0++10.0 ->20.0?6.0 Resource Management Tony Doyle - University of Glasgow

22. DataGrid Recruitment Status Robin Middleton Dec 01 Tools being developed Tony Doyle - University of Glasgow

23. DataGrid-UK Status Robin Middleton • Recruitment • EU funded posts filled & in post (1 additional from 1/2/02) • All but 2 of 15 “unfunded” posts active • Reporting • UK templates defined, with TB meetings before & after quarterly submissions • 81% of effort contracted delivered (but not uniform across WPs) for WP1-7 (WP8 hard to quantify) • UK Progress (in 2001) • Architectures, Testbed-1, network monitoring, certificates & security, R-GMA, LCFG, MDS deployment, Spitfire… … Tony Doyle - University of Glasgow

24. DataGrid-UK Effort Robin Middleton Tony Doyle - University of Glasgow

25. DataGrid-UK Status Robin Middleton • EU Review • All UK deliverables passed EDG internal review and now submitted to EU • Review at CERN on 1st March • 3 external reviewers • ~40 deliverables (inc. quarterly reports) • Summary presentations only • Can expect detailed feedback of areas to address • 4th Project Conference – Paris 4th-8th March • Plans for 2002 – TB2 • Absorb initial feedback from EU review Tony Doyle - University of Glasgow

26. DataGrid-UK News Robin Middleton • Licensing is still an issue • ATF reconstituted with more Applications involvement • New project web site at original address • Based at CERN • Uses EDMS for document handling • Testbed (see elsewhere at meeting) • Minor releases every 2 months • V2 in September • Transatlantic testbeds coming closer to reality • Scope being clarified • Interoperability a major issue  architectures Tony Doyle - University of Glasgow

27. InterGrid etc Robin Middleton • Strictly HENP-InterGrid • HICB meeting on 17th Feb Sunday (prior to GGF) • HIJTB meet monthly by phone • Projects • BaBar testbed • GRIT – Grid Reference International Testbed • ATLAS & CMS • DataTAG started • DataTAG : WP2=QoS, etc; WP4=m/w interoperability • Collaborates with iVDGL • CrossGrid, GridStart Tony Doyle - University of Glasgow

28. GRIT Ruth Pordes Tony Doyle - University of Glasgow

29. “The Grid” is an evolving concept • enabler for transient ‘virtual organisations’ • anatomy: a software infrastructure that enables flexible, secure, co-ordinated resource sharing among dynamic collections of individuals, institutions and resources • Foster, Kesselman & Tuecke (2001) • evolution of and integration with web services • physiology: everything is a Grid service i.e. a service that conforms to a set of conventions for management and exchanging messages • Foster, Kesselman, Nick & Tuecke (2002) • http://www.globus.org/research/papers/ogsa.pdf • Global Grid Forum: define a standard Grid architecture • big business and big science starting to work together • GGF4 (Toronto) next week Tony Doyle - University of Glasgow

30. GGF Architecture • 1) scope the problem - what are we trying to produce • 2) decide on the require functionality • 3) modularize the functionality • 4) decide on the relationships among the modules • 5) identify the data that must flow among the modules • 6) decide what state has to be maintained, and where • 7) design the protocols for communicating among and with the modules • 8) APIs • The challenge in the Grid forum is • - picked a large and amorphous problem (wide area distributed computing) • - for which there is not agreement (yet) as to what all of the functionality should be • - and even where we think that we understand the functionality requirements (e.g. data and resource management) there is not yet consensus on the modules • ANSWERS?? • DataGrid++ is “The Architecture” we have adopted - look here first Tony Doyle - University of Glasgow

31. e-Science and Grid • We are asked to map onto this plane… • Recognisable words, focussing on “middleware” • DataGrid++=middleware • (where ++=SAM, SLAC-based developments) • Experiments=Application Area • Middleware=Application Area?? • Draft 2.0 status • Response should be built up from technical definitions Tony Doyle - University of Glasgow

32. 2002 Challenges • Complete rollout of TB-1 and plan future upgrades • Reconvened ATF to work closely with applications • Make TB-2 a success • Deploy and exploit Tier-1/A • Applications to make good use of testbeds • Solve interoperability issues • We are part of many larger collaborations/structures/groupings - we need to collaborate/discuss engage here, and • Focus on implementation in the UK… this will tell us what works (and what doesn’t) at any given point. Tony Doyle - University of Glasgow

33. Status 13/2/02: Requires upgrades to functionality (extract hosts from info. services automatically) and maintainability (integrate into testbed workplan) Technical Management: See tomorrow’s discussion Tony Doyle - University of Glasgow

34. Status 13/2/02: Expect that Experiments will deliver production monitoring systems e.g. AliEn for ALICE (integrate into testbed workplan) Tony Doyle - University of Glasgow

35. Guidelines and Policies Its all in the small print. • Disk/CPU resources allocated to each “group” • Grid is based on distributed resources - a “group” is an experiment • An institute is typically involved in many experiments • Institutes define priorities on computing resources via OPEN policy statements • e.g. Edinburgh/Glasgow RAL ++ • ScotGrid - Priority 1: ATLAS, LHCb, Computing Science Data Mining BaBar (2/3-1/2-1/3) LHC (1/3-1/2-2/3) tomorrow’s • - Priority 2: BaBar, CDF, UKQCD e-Science discussions • BioInformatics • All jobs submitted via Globus authentication - Certificates identified by user and experiment • Need to implement “priority” • Minimum amount of data to deliver at a time for a job? • Where to store files? • Which data access/storing activities have the highest priority? • Sharing of the resources among groups? • Users belong to multiple groups? • How many jobs per group are allowed? • What processing activities are allowed at each site? • To which sites should data access and processing activities be sent? • How should the resources of a local cluster of PCs be shared among groups? • REQUIRES Discussion at the Collaboration Meeting… issues will arise which require ALL Institutes to publish a policy • Longer term goals… Grid Credits issued? * Implemented by site administrators, with OPEN policies defined at each site based on e.g. case to funding authority What’s new? Ability to monitor/allocate unused resources We will be judged by how well we work as a Virtual Organisation Tony Doyle - University of Glasgow

36. Summary • Most effort so far directed towards DataGrid=middleware • Significant UK contributions • Startup difficulties – largely overcome • Recruitment, late filling of posts – funded & unfunded • Lost ground being recovered.. • All WP with UK involvement active & contributing • Testbed-1 : so far modest, predictable teething problems (many !) • Next major challenges focus on INTEGRATION • US experiments programme - planning for today/tomorrow.. • LCG programme - planning for 2007... • Testbed (focus for tomorrow’s discussions) - requires input from system managers as well as software developers. • Positioned reasonably well ~6 months into GridPP.. • with many challenges ahead... Tony Doyle - University of Glasgow

37. Closing thought… 14/2/02 With many hidden words, meanings, nuances… “communication is the answer to the problem” (apparently) Grids are everywhere.. Tony Doyle - University of Glasgow

38. 3rd Collab Meeting Agenda • Feb 14th • Opening Session (Chair - Andy Parker): • 11:00-11:30 Welcome and Background - Steve Lloyd • 11:30-12:00 GridPP Project Status - Tony Doyle • 12:00-12:30 LHC Computing Grid Project Status - Tony Cass • Middleware Status Reports and Development (Chair - Tony Doyle): • 13:30-13:50 Information Services (FTREE and MDS) - Alex Martin • 13:50-14:10 Information Services (RGMA) - Manfred Oevers • 14:10-14:30 Mass Storage - Jens Jensen • 14:30-14:50 Fabric management - Michael George • 14:50-15:10 Grid Data Management - Will Bell • 15:10-15:30 Networking - Paul Mealor • Experiment Developments (Chair - John Gordon): • 16:00-17:00 ATLAS, CMS, LHCb and theory - Nick Brook • 17:00-17:30 WP8 status - Frank Harris • 17:30-18:25 BaBar, CDF, D0, SAM and others - Roger Barlow • 18:25-18:30 Logistics - Andy Parker • 18:30-19:30 Project Management Board Meeting • Feb 15th • Cambridge Perspective and TestBed Introduction (Chair - Robin Middleton): • 9:00-9:30 Cambridge e-Science Centre I - Andy Parker • 9:30-10:00 Cambridge e-Science Centre II - Mark Hayes • 10:00-10:30 TestBed Technical Development - Pete Clarke • Grid Deployment I (Chair - Pete Clarke): • 11:00-11:30 Testbed Tools and Release in UK - Andrew McNab • 11:30-11:40 Security issues - Linda Cornwall • 11:40-11:55 Testbed status at RAL - Steve Traylen • 11:55-12:10 Testbed status at M'cr - Andrew McNab • 12:10-12:25 Testbed status at Bristol - Marc Kelly • 12:25-12:40 Testbed status at IC - Dave Colling • Grid Deployment II (Chair - Steve Lloyd): • 13:30-13:40 Testbed status at Edinburgh/Glasgow - David Martin • 13:40-13:50 Testbed status at Liveropool - Girish Patel • 13:50-14:00 Testbed status at Oxford - Pete Gronbech • 14:00-14:10 Testbed status at UCL - Paul Mealor • 14:10-14:20 Testbed status at Cambridge - Mark Hayes • 14:20-14:30 Testbed status at Brunel - Peter van Santen • 14:30-14:40 Testbed status at Lancaster - Roger Jones • 14:40-15:00 Discussion • 15:00-15:30 TestBed UK Organization - John Gordon • WELCOME! Tony Doyle - University of Glasgow