HPC@sheffield - where do we go next?

1. HPC@sheffield -where do we go next? John HardingDept. Materials Science and Engineering, Sheffield

3. Simulation Scales www.nsf.gov/news/speeches/bordogna/rochester/sld006.htm

4. Simulations and scales

5. For details see www.prace-ri.eu

6. Grid computing – particle physics

7. Iceberg Current Configuration • Cluster based on Intel 6- core X5650 • 71 nodes: 12 cores and 24GB • 4 nodes: 12 cores and 48GB • 8 NVIDIA Tesla Fermi M2070GPUs • Storage • 45TB NFS mounted mirrored data store • User quota system • 80TB Lustre based parallel file store • Fastdata store a scratch area data is deleted after 90 days

8. How do we use what we have at Sheffield

9. Iceberg main users • Mechanical Engineering • Aerodynamics • Thermofluids • Applied Maths • Solar Physics and Space Plasma Research • Insigneo • Virtual Physiological Human Initiative • Electrical Engineering • Device Electronics: nanoWires and NanoTubes • Lensless microscopy

10. Future of computing in Sheffield, SP2RC approach Solar Physics and Space Plasma Research Centre (SP2RC) Solar Wave Theory Group (SWAT) School of Mathematics and Statistics (SoMaS), Faculty of Science, UoS Space System Laboratory (SSL) Automatic Control and Systems Engineering (ACSE), Faculty of Engineering, UoS • SP2RC at the UoS seeks to understand the nature of key plasma processes occurring in • the solar interior, • the atmosphere of the Sun, from • photosphere to corona, • with particular attention devoted to the various • coupling mechanisms of these apparently distinct • regions.

11. Interfaces for photovoltaics Supercell Fermi energy. n-type Schottky barrier height. Valence band top energy of bulk semi-conductor. p-type Schottky barrier height. Semi-conductor band gap energy. K.T. Butler, P.E. Vullum, A.M. Muggerud, E. Cabrera and J.H. Harding, Phys. Rev. B 83 (2011) Art 235307   

12. Biomechanics (Lacroix)

14. A new, bigger Iceberg • Infrastructure to be added to existing Intel Nodes • Cluster based on Intel Ivy Bridge 8- core E5-X2650v2 • 71 nodes each with 16 cores and 64 GB memory • 8 nodes each with 16 cores and 256GB memory • 8 NVIDIA Tesla Fermi K40 GPUs • Storage • 100TB Lustre based parallel file store • Fastdata store a scratch area data is deleted after 90 days Network • Expand Infiniband spine to facilitate further expansion and addition of nodes by research groups • Update of NFS mounted data – TBD • Older AMD nodes to be decommissioned • Coming by June 2014.

15. Hardware Access Routes • Research groups can • access the general facility – fair share using system scheduler • Purchase hardware and host on iceberg • Purchase a number of compute nodes for a specified of time • Purchase route provides a dedicated access to a resource effectively ‘owned’ by the reseacher

16. Client Access to Visualisation Cluster Iceberg – Campus Compute Cloud VirtualGL Server (NVIDIA GPU) VirtualGL Client

17. Applications Supported Using Graphical Processing Units • Ansys • Matlab • Flame • Beast • AnsysAbaqus • Smoothed Particle Hydrodynamics (DualSPhysics) • Astronomy • MHD Applications (SMAUG, Athena) • Nbody 6

18. N8 HPC • 316 nodes (5,056 cores) with 4 GByte of RAM/core • 16 nodes (256 cores) with 16 GByte of RAM/core • Sandybridge architecture • Nodes connected by infiniband. • 110 Tflop/sec peak. • 174Tb Lustre v2 parallel file system • 109Tb NFS backed-up file system www.n8hpc.org.uk

19. N8 HPC - objectives • Seed engagement between industry and academia around research using e-infrastructure • Develop skills in the use of e-infrastructure across the N8 partnership • Share the asset of skills and equipment across the N8 partnership via the facilitation of networks of people

20. What people have done on N8 Studies of turbulence Shuisheng He Biomolecules on surfaces Shaun Hall MySpine – Damian Lacroix; Sheffield

21. N8 HPC - access • You need a N8 HPC Project Account. We ask that PIs are permanent members of staff at their institution • Contact the local HPC representative at Sheffield – hpchub@sheffield.ac.uk or Iceberg admin. • You will need to produce a project proposal for consideration by the local committee. • You will be expected to produce a report saying what you have done with the resources.

22. ARCHER (www.archer.ac.uk)

24. What there is (Stage 1) • ARCHER compute nodes contain two 2.7 GHz, 12-core E5-2697 v2 (Ivy Bridge) series processors (2632 nodes) • Standard compute nodes on ARCHER have 64 GB of memory shared between the two processors. Smaller number of high-memory nodes with 128 GB of memory shared between the two processors (376 nodes) • In all, 3008 nodes; 72,192 cores.

25. Getting access • ARCHER Access Through EPSRC and NERC Grants – process similar to HECToR. Need to submit a technical assessment before the grant application and attach result to JES form • ARCHER Instant Access – means what it says; limited amounts for pump-priming • ARCHER Access Through the Resource Allocation Panel – stand-alone access request. Next closing date is May 14th (EPSRC remit) • ARCHER access through a consortium – there may be one in your area; need to contact PI.

28. Some facilities available

29. Getting access • PRACE publishes its Calls for Proposals for Project Access twice per year according to a fixed schedule: • Call opens in February > Access is provided starting September of the same year. • Call opens in September > Access is provided starting March of the next year. • See www.prace-ri.eu for details.

30. Some issues • High Performance Computing is essential but does not stand on its own. Simulations are simulations of a real world problem. They must not become divorced from that. All simulation has a multidisciplinary side. • People can always use more computing power – but getting good, reliable software is even better (and much harder) • Hardware is useless without technical support and training – and these tend to be much harder to fund. • Data is becoming a major issue – its capture, curation and analysis. We are already into petabytes here at Sheffield. The world is into exabytes at least.

31. Closing remarks • Industry tends to engage with Academy on problems they need answering. • Needs an expertise offering, even if approached for cycles. • Trusted relationships take time • Important to maintain channels that work • HPC is part of a broader offering • Critical to build the right team • Involve skills/resources in National centers • Must listen to & provide forum for industry • Shape & maintain a relevant offering to industry • Embed in existing activities: N8 industry innovation forum • Links to Universities • Appropriate skills emerging from our graduates.

32. Main Goals we would like to achieve in SP2RC • Forward modelling of the the excitation, progression and dissipation of MHD waves in the solar atmosphere based on the single/multi- highly localised magnetic flux tubes (e.g. magnetic bright points/pores, magnetic elements of Active Regions, vortices and inter-granular lanes) by applying realistic sources (even from high-resolution observations) of wave excitation based on one/two fluid MPI/GPU MHDcodes. • To determine the role of Alfven, kink and longitudinal wavesin the wave energy transport from photosphere to the upper layers of the solar atmosphere. • Perform multidimensional two fluid MHD numerical simulationsat various spatial and temporal scales within the chromosphere, transition region and low corona by taking into account the crucial solar plasma processes, i.e. heat conduction, ionization, proton-neutral hydrogen collisions, in order to obtain an accurate description of the classic wave dynamics and line synthesis. • Apply realistic 3D MHD models of the solar atmosphereinvestigating the mechanisms governing the transport of energy from the solar photosphere to the solar corona.

33. Orszag-Tang Test 200x200 Model at t=0.1, t=0.26, t=0.42 and t=0.58s

34. Timing for Orszag-Tang Using SAC/SMAUG with Different Architetures