Experiences of the Grid…

Experiences of the Grid… Gavin McCance University of Glasgow NeSC Meeting, 24 October 2001

Background • Experimental Particle Physics background • Analysing the structure of matter • …Fortran (19)77 ! • Working in ‘Grid’-like areas since January this year Gavin McCance, University of Glasgow

GridPP 20+ institutes… Gavin McCance, University of Glasgow

…GridPP • £17M 3-year project • Working in collaboration with EU DataGrid project • Middleware production • Integration of middleware technologies into HEP experiments • Validation of Grid Software Gavin McCance, University of Glasgow

…GridPP • Initial GridPP testbed underway • A personal snapshot of activities on the grid… • Middleware activities we’re involved in • Some examples • Technologies we’re using • Issues with integration of ‘Grid’ with particle physics experiments Gavin McCance, University of Glasgow

Middleware • What is middleware…??? Application programs – local gridopen() Layered API’s. Transparent security. Transparent data access. Intelligent use of distributed resources. Grid middleware Data access specifics – HPSS, Castor Job submission specifics – PBS, LSF Specific security procedures Gavin McCance, University of Glasgow

Middleware Activities • GridPP ~mirrors EU DataGrid: • Workload Management • What jobs go where? • Data Management (*) • Where’s the (best) data? • Information Services (*) • What’s the state of everything? Gavin McCance, University of Glasgow

…Middleware Activities • Fabric Management • Interfaces to underlying systems • Mass Storage Management • How to get the data to/from the fabric e.g. Implementing ‘file-save()’ APIs for different mass storage systems • Security • Crops up everywhere … transparent to applications Gavin McCance, University of Glasgow

Data Management • Data Replication • Transparent and Secure Data Access • Meta Data Storage • Query Optimisation Gavin McCance, University of Glasgow

Example problem: Data Replication • Problems if data exist only in one place • Multiple accesses to the same data overload network! Petabytes! • Funding constraints! e.g. CERN can’t store all of the data required • Make Replica! But need to keep track of all the files and their various replica! • Need replica catalogue! Gavin McCance, University of Glasgow

…Catalogues • Examples solutions: • Have a globally unique Logical File Name (LFN) mapping to multiple physical instances of the file (PFNs). • Replica selection required • Choose the ‘best’ / ‘nearest’ / ‘fastest’ • Cost modelling… how time expensive to transfer files X’ from A to B File-1 LFN Paris File-1 Glasgow File-1 Chicago File-1 Gavin McCance, University of Glasgow

…Data Replication • Grid Data Mirroring Package • C, C++, JAVA, command-line APIs • Replication issues: • File transfer… • Synchronisation / consistency models • Basic middleware doesn’t enforce any policy • Scalable architectures Gavin McCance, University of Glasgow

…GDMP • File transfer uses GridFTP • Existing IETF-approved (?RFC?) ftp additions • + the standard grid security (GSI) • Registers new files in replica catalogue • E.g. interfaced to the existing Globus Replica Catalogue • Basic replica manager functionality to maintain consistency of replica sets Gavin McCance, University of Glasgow

CERN Root RC INFN RC UK RC CERN RC …Implementation issues • Structure not imposed by the middleware software itself… • But … must think about scalable implementations • E.g. a RC may exist on each storage element  responsible for its own files Queries will propagate down until replica information is found… Gavin McCance, University of Glasgow

…Longer term problems • Query / Replica Optimisation • Grid can make / delete replica • Eg. Many people in Glasgow & Edinburgh access the ATLAS Higgs dataset ‘A1’… • Autonomously make new replica in / near Scotland based on historical information • Grid might re-cluster data Paris Glasgow Paris Glasgow A1 B3 A1 B3 A2 B1 A2 B2 A3 A3 B2 B1 Gavin McCance, University of Glasgow

MONARC simulation tool real Grid... …simulated Grid provides testing arena for more adventurous ideas! …longer term Gavin McCance, University of Glasgow

…Integration of middleware • Many iterations of requirements and use-cases with end-users… meetings… • Middleware solutions must be scalable and useable by a variety of end users • HEP, Biological, Earth sciences, Astro • Always looking for common elements • E.g. replica / meta-data catalogues… data transport… security… Gavin McCance, University of Glasgow

…examples of common interfaces:generic meta-data catalogue tools • SQL Database Service: • Problem: many relational databases, diverse security, diverse wire protocols • …Solution: • Build on existing wire protocols: XML transported over HTTP(S) • Grid standard security framework (GSI) Gavin McCance, University of Glasgow

..examples • Leverage open-source technology • JAVA servlet based (Apache Tomcat engine) • JDBC drivers • Utilises Oracle’s XSQL servlet (open source) • Security over HTTPS with Grid-standard GSI mechanism Gavin McCance, University of Glasgow

…examples • Allows any HTTP compliant system e.g. Web-browsers / standard C++ HTTP libraries to access any relational database… + + Oracle PostgreSQL PKI Security Standard communication protocols (XML over HTTPS) = SQL Database Service (Spitfire) Gavin McCance, University of Glasgow

Global Grid Forum • Global Grid Forum meetings • GGF1: Amsterdam meeting in April 2001 • Helps define aspects common to all Grid-like projects. • E.g. architectures, ‘grid’ protocols • As example… Grid Monitoring Architecture (GMA) Gavin McCance, University of Glasgow

Information Services - GMA • One Implementation of the GMA •  Globus MDS, currently based on (Open)LDAP • Hierarchical directory like structure • Very fast for information retrieval if you already know the query  designed into structure. • Bad for complex or ranged queries Gavin McCance, University of Glasgow

..complementary implementation Register, re-register, publish Producer Servlet Registry Servlet Producer Producer API Schema Servlet subscribe Query stream Consumer Querying API Implementation of GMA Relational queries in SQL format Relational Database Gavin McCance, University of Glasgow

…relational GMA • Information is transferred in generic SQL format… • ‘Producers’ of information register themselves… • ‘Consumers’ construct (possibly complex) SQL query and are streamed query results directly from Producers. Gavin McCance, University of Glasgow

…implementation • Again, uses JAVA servlets • Tomcat servlet engine • Again, communication with servlet is over standard HTTP. • All the internal parts communicate via HTTP and XML modular design, easily replaceable… Gavin McCance, University of Glasgow

Useful Tools… • JAVA… nicely platform independent • UML Universal(?) Modelling Language • Architecture and API’s ‘should be’ defined in this…! • CASE tools • Together Control Centre Gavin McCance, University of Glasgow

…useful tools • Globus toolkit • Both the original and its java implementation (CoG) • My experience of CoG so far is generally good…! • Easy GSI authentication, Globus file transfer, Globus job submission, MDS interface Gavin McCance, University of Glasgow

Testbeds • For GridPP, primary testbeds are the HEP experiment ones • CERN LHC (EU DataGrid WP8) • US experiments, e.g. Fermilab, SLAC • First software release now!! • Integration team ‘show-and-tell’ at CERN end of this month… Gavin McCance, University of Glasgow

...testbed work Grid software packaged for release to experiments! Primarily packaged using RPM For end of October release, supported platforms are: Linux (and Solaris on a best effort basis) Gavin McCance, University of Glasgow

..Globus installation • Generally found the Globus software installation OK! • Successfully deployed on a number of batch systems in UK • Experience fed back into eScience Centres Difficulties were setting up and recognising each countries’ Certificate Authorities (CAs)  Tricky legal implications to resolve! Gavin McCance, University of Glasgow

Testbed work so far… • UK Certificate Authority set-up… • Many institutes already on testbed Grid Status and Network monitoring demonstrator available soon Networking status information provided by GridPP and DataGrid networking groups! Gavin McCance, University of Glasgow

…testbed work so far • Successful tests within ATLAS (and others) of some middleware products • E.g. Large file transfers between UK, Italy, US and CERN • Further tests planned with new release! Gavin McCance, University of Glasgow

…experimental integration • Work to do… • Taking the kit and trying to integrate it into the experiments’ software frameworks ATLAS/LHCb software framework (GAUDI) Make Grid Services transparently available to ATLAS and LHCb programs GANGA framework Grid middleware Gavin McCance, University of Glasgow

Grid validation • Preliminary tests of basic middleware has been successful • Now we have opportunity to see how it performs and scales with real datasets and real experimental users Gavin McCance, University of Glasgow

Overall experience • Middleware development is fun! • Several good products have already been delivered • Re-using industry standard components and protocols where they exist • LDAP, SQL, HTTP(S), XML, SOAP • PKI security • Open Source…! Gavin McCance, University of Glasgow

…overall • Middleware being built using a variety of languages… JAVA, C++, C, Python • APIs should be available for all JAVA, C++, C and command line… web access(?) Gavin McCance, University of Glasgow

…overall • Coordination very important • Forums for discussion: • Vital to ensure middleware is useful to a wide range of applications • Prevent divergent technology Gavin McCance, University of Glasgow

…finally • Preliminary grid software architectures have been defined • Basic middleware has been delivered • Large scale validation underway NOW!! • A excellent base to build on! • Plenty still to do! Gavin McCance, University of Glasgow

Experiences of the Grid…