1 / 24

Data Transfer Efficiency - leave no byte unchurned

Data Transfer Efficiency - leave no byte unchurned. Jens Jensen Rutherford Appleton Laboratory GridPP26, U Sussex, March 2011. Background. GridPP’s data grid Distributed Storage Elements Data movers (FTS, PhEDEx et al ) Catalogues (usu. replica)

aminia
Download Presentation

Data Transfer Efficiency - leave no byte unchurned

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Transfer Efficiency- leave no byte unchurned Jens Jensen Rutherford Appleton Laboratory GridPP26, U Sussex, March 2011

  2. Background • GridPP’sdata grid • Distributed Storage Elements • Data movers (FTS, PhEDExet al) • Catalogues (usu. replica) • e-Infrastructure (aka cyberinfrastructure) • (Presentation at ISGC)

  3. The Data Grid • WLCG is primarily a data grid • Computation can (in principle) be redone • Jobs go to where data is • Moving a job is quicker than moving data

  4. Premature Optimisation is the Root of All Evil

  5. Postmature non-optimisation is the root of some evil • The role of infrastructure code • Scientist as a programmer • “Bad” code moves up the stack? • “Bad” code improves over time? • Doofers stay in prod’n

  6. Efficiencaciousness Goals Service • Availability • Performance • Grows as needed • Robust (no SPoF?) People • (Effective) support • Training • Expertise • Availability of…

  7. Approaches • Philosophy • Get it done – WLCG • Get it done right – EGI? • Do It Perfectly The First Time… • Evolutionary (control system) vs revolutionary • Proactive vs reactive

  8. Efficiencaciousness Issues • Failures • Sites – BDII, network • Elements – storage • Components – disk servers • Timeouts • DDoS

  9. Efficiencaciousness Issues • Overall effort • Funded, contributed, external • Availability of expertise • Single Point of Knowledge • Decoherence • 2nd Law of Thermodynamics • Learning from incidents

  10. Efficiencaciousness Issues • Primary communication • Sites • Users: large VOs, small VOs, single users • PMB • Secondary • WLCG • NGS

  11. Efficiencaciousness Issues • Sites • There Is Always A Bottleneck Somewhere • Site dependent • Usage dependent • Information • Freshness • Accuracy (“spped is substutefoaccurcy”)

  12. Efficiencaciousness Issues • Usage patterns • C.f. Wahid’s talk yesterday • WAN vs LAN (WN) traffic • Technology • In the narrow sense (drives, controllers) • And the wider sense: dist’dfilesystems • Support: Upstream (EGI), Fabric

  13. Efficiencaciousness Issues • Overheads • Complexity of use of stack (see next) • Infrastructure is complex • But Complexity Has To Go Somewhere • Time-to-production • Testing, troubleshooting, monitoring, tweaking, tuning

  14. With apologies to the OSI stack

  15. Particular Pain Point Principle Progress

  16. Progressing Forward • What is progress • How to measure progress

  17. The Good News • We’ve come a long way • Don’t think there is a skills gap • But some SPoKs

  18. Graeme’s talk • “Get the best out of what we can afford to buy” • Proactive sites better • Standards are good

  19. E[GM]I involvement • EMI data roadmap • Support for dCache, DPM, StoRM • Support for standards (NFS4, CDMI) • But then • StoRM=INFN, dCache=DESY, DPM=CERN

  20. The Cloud View • Supplement resources with on-demand • Agile • CDMI is superset of SRM • But using ReST+JSON, not SOAP

  21. (Open) Standards • Standards promote interoperation and stability • Interoperation • Multiple (independent) implementations • Both Java and (C or C++)

  22. The Case for Non-HEP Data • Benefit from non-HEP data • Outreachy stuff • Benefit to society (eg saving lives) • NGI interop (at compute) • Others…

  23. Summary

  24. Efficiencaciousness Goals Service • Availability • Performance • Grows as needed • Robust (no SPoF?) People • (Effective) support • Training • Expertise • Availability of…

More Related