190 likes | 331 Views
APAN Workshop Jan 27, 2005 Bangkok. Belle/Gfarm Grid Experiment at SC04. Osamu Tatebe Grid Technology Research Center, AIST. Goal and feature of Grid Datafarm. Goal Dependable data sharing among multiple organizations High-speed data access, High-performance data computing Grid Datafarm
E N D
APAN Workshop Jan 27, 2005 Bangkok Belle/Gfarm Grid Experiment at SC04 Osamu Tatebe Grid Technology Research Center, AIST
Goal and feature of Grid Datafarm • Goal • Dependable data sharing among multiple organizations • High-speed data access, High-performance data computing • Grid Datafarm • Gfarm File System– Global dependable virtual file system • Federates scratch disks in PCs • Parallel & distributed data computing • Associates Computational Grid with Data Grid • Features • Secured based on Grid Security Infrastructure • Scalable depending on data size and usage scenarios • Data location transparent data access • Automatic and transparent replica selection for fault tolerance • High-performance data access and computing by accessing multiple dispersed storages in parallel (file affinity scheduling)
/gfarm ggf jp file1 file2 aist gtrc file2 file1 file3 file4 Grid Datafarm (1): Gfarm file system - World-wide virtual file system [CCGrid 2002] • Transparent access to dispersed file data via global namespace • Files can be stored somewhere in a Grid • Applications can access Gfarm file system without any modificationas if it were mounted at /gfarm • Automatic and transparent replica selection for fault tolerance and access-concentration avoidance Globalnamespace mapping Gfarm File System File replica creation
Grid Datafarm (2): High-performance data access and computing support [CCGrid 2002] User’s view Physical execution view User A submits that accesses is allocated on a node that has File A File A Job A Job A User B submits that accesses is allocated on a node that has File B File B Job B Job B network Computational Grid CPU CPU CPU CPU Gfarm File System Compute and file system nodes Shared network file system Do not separate Storage and CPU Scalable file I/O by exploiting local I/O
GfarmTM Data Grid middleware • Open source development • GfarmTM version 1.0.4-4 released on Jan 11th, 2005 (http://datafarm.apgrid.org/) • Read-write mode support, more support for existing binary applications • A shared file system in a cluster or a grid • Accessibility from legacy applications without any modification • Standard protocol support by scp, GridFTP server, samba server, . . . Metadata server application gfmd slapd * Existing applications can access Gfarm file system without any modification using LD_PRELOAD Gfarm client library CPU CPU CPU CPU gfsd gfsd gfsd gfsd . . . Compute and file system nodes
Demonstration • File manipulation • cd, ls, cp, mv, cat, . . . • grep • Gfarm command • File replica creation, node & process information • Remote (parallel) program execution • gfrun prog args . . . • gfrun -N #procs prog args . . . • gfrun -G filename prog args . . .
Belle/Gfarm Grid experimentat SC2004 1. Online KEKB/Belle distributed data analysis 2. KEKB/Belle large-scale data analysis (terabyte-scale US-Japan file replication)
1. Online KEKB/Belle distributed data analysis (1) • Online distributed and parallel data analysis of raw data using AIST and SDSC clusters • Realtime histogram and event data display at SC2004 conference hall Raw data 10 MB/sec SC2004 Gfarm file system KEK • realtime histogram display • realtime event data display 192 nodes 53.75 TB SDSC AIST 128 nodes 3.75 TB 64 nodes50 TB • on demand data replication • distributed & parallel data analysis
1. Online KEKB/Belle distributed data analysis (2) • Construct a shared network file system between Japan and US • Store KEKB/Belle raw data to the Gfarm file system • Physically, it is divided into N fragments, and stored on N different node • Every compute node can access it as if it were mounted at /gfarm Raw data 10 MB/sec SC2004 KEK • realtime histogram display • realtime event data display Gfarm File System SDSC AIST 192 nodes 53.75 TB 64 nodes50 TB 128 nodes 3.75 TB
1. Online KEKB/Belle distributed data analysis (3) • Basf is installed at /gfarm/~/belle • Install once, run everywhere • The raw data will be analyzed at AIST or SDSC just after it is stored • Analyzed data can be viewed at SC2004 in realtime • Histogram snapshot is generated every 5 minutes Raw data 10 MB/sec Computational Grid SC2004 KEK • realtime histogram display • realtime event data display Gfarm File System SDSC AIST 192 nodes 53.75 TB 64 nodes50 TB 128 nodes 3.75 TB
2. KEKB/Belle large-scale data analysis in a Grid • Gfarm file system using SC conference hall and AIST F cluster • Assume data is stored at SC conference hall • Terabyte-scale mock data • Submit data analysis job at AIST F cluster • Required data is automatically transferred from SC to AIST on demand • Users just see a shared file system • Network transfer rate is measured • Conf 1: 8 parallel processes (2GBx8 data) • Conf 2: 16 parallel processes (8GBx16 data)
PC AIST F cluster PC PC PC PC 10G (OC192) PC PC PC FCx16 1Gx8 PCx256 2TBx16 PCx8 2. Network & machine configuration JGN2 Tsukuba WAN SC2004 StorCloud AIST F cluster JGN2 Japan-US 10Gbps 10Gbps 10Gbps Chicago Tokyo Pittsburgh
SC→AIST (Iperf x 8) 7,925,607,155 bps (Wed Nov 10 17:13:22 JST 2004) (5-sec average bandwidth, 991 Mbps / TCP flow)
Iperf measurement • Standard TCP (Linux 2.4) • Socket buffer size and txqueuelen • No kernel patch, no TCP modification • No traffic shaping • No bottleneck, no problem
Conf 1: 8 processes (2GBx8) 2,084,209,307 bps (Fri Nov 12 03:41:54 JST 2004) (5-sec average, 261 Mbps / TCP flow, ~disk performance of F cluster)
Conf 2: 16 processes (8GBx16) 738,920,649 bps (Fri Nov 12 05:30:35 JST 2004) (5-sec average, 46 Mbps!! / TCP flow, ?????)
Conf 2: network traffic of JGN2 int’l link Heavy traffic when application started Heavy packet loss→ssthresh decreases
Summary • Belle/Gfarm Grid experiment at SC2004 • 1. Online KEKB/Belle distributed data analysis • 2. KEKB/Belle large-scale data analysis • We succeeded distributed & parallel data analysis of KEKB/Belle data and realtime display at SC conference hall
Development Status and Future Plan • Gfarm – Grid file system • Global virtual file system • A dependable network shared file system in a cluster or a grid • High performance data computing support • Associates Computational Grid with Data Grid • Gfarm v1 Data Grid middleware • Version 1.0.4-4 released on Jan 11, 2005 (http://datafarm.apgrid.org/) • Existing programs can access Gfarm file system as if it were mounted at /gfarm • Gfarm v2 – towards *true* global virtual file system • POSIX compliant - supports read-write mode, advisory file locking, . . . • Performance and Robustness improved, Security enhanced. • Can be substituted for NFS, AFS, . . . • Application area • Scientific application (High energy physics, Astronomic data analysis, Bioinformatics, Computational Chemistry, Computational Physics, . . .) • Business application (Dependable data computing in eGovernment and eCommerce, . . .) • Other applications that needs dependable file sharing among several organizations • Standardization effort with GGF Grid File System WG (GFS-WG) https://datafarm.apgrid.org/