50 likes | 145 Views
( non-archive) Storage Futures. Wahid Bhimji SRM; FTS3; xrootd ; DPM collaborations; cluster filesystems. SRM ; FTS3; xrootd. SRM is currently required on all WLCG storage It has limitations; not much of the spec is used Some ( eg . CERN!) are talking about not using it
E N D
(non-archive)Storage Futures Wahid Bhimji SRM; FTS3; xrootd; DPM collaborations; cluster filesystems
SRM ; FTS3; xrootd • SRM is currently required on all WLCG storage • It has limitations; not much of the spec is used • Some (eg. CERN!) are talking about not using it • There is a WLCG WG to monitor alternatives (ensure interoperation; limit proliferation; etc.) • BUT ATLAS and LHCb require developmentto get away from SRM and some issues are not solved So.. • Storage for coming years needs a stable SRM interface. • In future it may not – there will be an interface of some sort – but it will be lighter (I hope). • FTS already supports gridftp-only endpoints and FTS3 will also offer http and xrootd. • Xrootd use is expanding • Big interest is “federated” storage –failover and “anydata anywhere” • (Other solutions e.g http can offer this and are not hep specific) • CMS is asking all sites to have xrootd interface by end of year • ATLAS is also pushing deployment – but use cases not clear…
DPM collaboration • DPM support at CERN decreasing from current (v. good) level • CERN asking for collaborators to continue to maintain DPM • They say they will provide “minimal” support even without collaboration (bug fixes etc.) • Collaboration also has advantages in terms of getting needed developments • On the other hand – landscapes change: • dCache is maybe easier to use than before; StoRM maybe more stable; Lustre and HDFS are well established • Next years shutdown _may_ also be an opportunity to try something different • Though DPM also offering “DMLite” ontop of Lustre/HDFS
Options and issues:my twist on a doc from Jens • Join collaboration (~1FTE or 2 x 0.5) • Do we have the skills for core development? • Does DPM have a long term (support) future? • Is the shutdown a chance to move to something “better” (e.g. for “hotfiles”) • Move to something else • dCache; Storm/Lustre; DMLite/Lustre; DMLite/HDFS • Migrating data (for ATLAS a recopy is fine but there is bound to be some hassle) • Migrating storage (onto something new / unfamiliar) lot of work – especially for smaller sites. • Have a lot of DPM experience (e.g. tuning) so alternative may not work out “better” for us
Evaluation • Need to try dpm development to see how easy it is (e.g. with DMLite) • Need criteria if comparing alternatives. E.g: • Transition effort • Maintenance effort For our use cases: • Stability • Functionality • Performance (inc. ease of tuning) • Both of these take time (i.e. six months evaluating could be spent training in DPM) • Site-admin view should have high weight...