Introductory concepts: Computer etiquette

1. Introductory concepts:Computer etiquette Jon Goss MMG Skills Lecture Series

2. Outline • Get organised • Consistency vs efficiency • Compute machine vs file server • Hierarchical calculation strategy • Restarting calculations • Throughput vs turn-around time • Interacting with a batch queue MMG Skills Lecture Series

3. Get organised • You are in an area of study that has the potential to produce vast amounts of data • Why do we need to store it? • It is crucial that you adopt a comprehensive, structured filing system from the very start. • Alpha-numerical file naming with index files – could be chronological. • Potentially many files per directory. • Hierarchical set of directories based on application • Try to keep few files per directory. • You’ll probably have to revise your filing system over time. • Acid test: • When asked to locate a file, can you do it within a few seconds? • Minutes? • Hours? • Days? MMG Skills Lecture Series

4. Consistency vs efficiency • When you begin researching an application, you may adopt a parameter set from some previous calculations to be consistent with them • The foundation work may already be there • Bulk material: basis sets, lattice constants, convergence tests… • This may allow direct comparison between previous and current calculations • Binding / reaction energies • Marker method calculations for electrical levels • Structures for better starting points • It reduces the probability of trivial errors – it does not eliminate them. MMG Skills Lecture Series

5. Consistency vs efficiency • Consistency is good, but not at any cost: • The old parameter set may be inefficient in terms of CPU time per calculation. • New parameters may be better designed (e.g. optimised basis sets) for this specific project • In the long run it may be better to spend time at the start setting your parameters as this gives you a higher level of confidence in the accuracy of the results, and grounds you in the fundamentals of the calculations • You may well find that you have to repeat large numbers of calculations if you have to revise from a sub-optimal setup… • Calculating the same basic values twice is unlikely to be exemplary efficiency MMG Skills Lecture Series

6. Consistency vs efficiency What ever you do, it should be agreed with all involved in the project, and you should choose your strategy carefully at the start: Think! MMG Skills Lecture Series

7. Compute machine vs file server • This distinction is part of “organisation”, and is important in: • Avoiding duplication • Avoiding loss of data • Allowing access for others involved in the project • Keeping access to important files for yourself • Some computers are there for calculations: • Rhodes, Verity, Braid, Hector, HPCx, … • Other machines are designed to store files: • Trueman, Snufkin… MMG Skills Lecture Series

8. File storage • We need to answer three questions: • Why do we need to store files? • Which files do we need to keep? • How can we minimise file-space usage? MMG Skills Lecture Series

9. File storage: why? • Why do we need to store files? • Scientific ethics require us to be able to back up our claims! • They are one of our chief resources for future research. MMG Skills Lecture Series

10. File storage • Which files do we need to keep? • Always AIMPRO standard output, plus • Bandstructures: bandst.out, bandt.plt • EELS / OA: bandst.out, dieln output files • Mulliken: bandt.out • NEB: maybe res.neb • AIMVIEW: bandst.out, dump files (careful with these) • DDS: maybe derivs.txt • DoS: bandst.out , dos.out , dos.dump • Keep all parts of a run • If you have to restart a relaxation, for example, keep all parts. • What don’t we keep in our permanent filing: • Restart dump files! • fort.99, standard error files (e.g. aim2.3.01b.sh.e18371) • Aimpro input files (dat, pseudo-pots, hgh-pot, bandst.dat,…) unless they are modified specifically for this run, and cannot be re-created from the aimpro output. MMG Skills Lecture Series

11. File storage • How can we minimise file-space usage? • Most files we generate are ASCII • They can be reduced in size by compressing them (we’ll look again at this in another lecture): • bzip2 <file> • (You can learn about this by typing man bzip2 on snufkin.) • A typical aimpro output file may be reduced in size by 80% without any loss of data. • ‘Bzipped’ files can be viewed, gres’d and even edited. • You have a quota on most machines • If you exceed it, you’ll be unable to do very much • You may be prevented from logging into the machine! • If there is no quota, you may fill the disk space and affect all other users. MMG Skills Lecture Series

12. Calculation strategy:hierarchy of costs • The majority of the AIMPRO computational time is taken in obtaining total energies: • The self-consistent cycle. • We focus on this part of the calculation for speed. Obtain ρin Generate Ĥ ρin=ρout? Done Obtain ρout MMG Skills Lecture Series

13. Calculation strategy:hierarchy of costs • Commonly, a goal may be reached in a sequential method, minimizing computational effort… • …we want to maximise the amount of science we can do so we need to be able to answer the following questions: • How do the decisions we make in constructing a data file, and how we run the job affect the time & efficiency? • Which factors are most important? MMG Skills Lecture Series

14. Number of atoms Number of different species Self-consistency method Number of basis functions Number of exponents Maximum orbital angular momentum Initialisation charge density basis Number of k-points Location of k-points Spin state Plane-wave basis Number of processors The amount of memory available K-point parallelism Number of symmetry operations DIIS Real-space build The amount of vacuum (molecules and surfaces) Number of images in a NEB Number of k-points in a band structure Number of bands in an EELS run Number of energies in an EELS run Number of atoms included in the derivatives Calculation strategy:hierarchy of costs MMG Skills Lecture Series

15. Number of atoms Number of different species Self-consistency method Number of basis functions Number of exponents Maximum orbital angular momentum Initialisation charge density basis Number of k-points Location of k-points (real/complex) Spin state Plane-wave basis (vacuum) Number of processors The amount of memory available K-point parallelism Number of symmetry operations DIIS (extensive parallelism) Real-space build The amount of vacuum (molecules and surfaces) Number of images in a NEB Number of k-points in a band structure Number of bands in an EELS run Number of energies in an EELS run Number of atoms included in the derivatives Calculation strategy:hierarchy of costs MMG Skills Lecture Series

16. Calculation strategy:hierarchy of costs • For an SCF step: • The time for an energy scales as nα, α~2→4 where n is the dimension of the Hamiltonian. • Going from real arithmetic to complex (at a general k-point), the time increases by a factor of ~½-1 order of magnitude. • Sampling the Brillouin-zone mp23 generally includes 4 (complex) points for a cubic cell. • Spin polarisation doubles the time taken • The number of SCF steps for an energy may increase with spin-polarisation. • Compare the time for an energy for a pppp, pdpp, ddpp and dddd basis. • Compare the time for pppp, gamma point, spin averaged, and dddd, 4 complex k-points, spin polarised. MMG Skills Lecture Series

17. Calculation strategy:hierarchy of costs • The most common calculation we perform is a structural relaxation. • If we do not have an accurate starting structure, it is likely that the optimisation will require multiple optimisation iterations. • Why is this likely to be the case? • The number of structural iterations is approximately independent of how we run AIMRPO, provided the calculation is performed within a ‘reasonable’ set of parameters (i.e. not necessarily convergent ones). • What does this tell us about the energy surface? • A structural iteration generally takes more SCF steps when we’re far from the structural minimum. • This is related to recycling charge densities from previous structures – why does this affect the number of SCF cycles? MMG Skills Lecture Series

18. Calculation strategy:hierarchy of costs • Example: • Substitutional gold in silicon. • We want to know the symmetry, donor and acceptor levels. • We need to check the convergence with • Cell size • Basis • Pseudo-potential – whether the 5d electrons are included in the valence or not • How do we start? Sketch out a strategy to get all the data we need. MMG Skills Lecture Series

19. Calculation strategy:hierarchy of costs • Basis • Design a hierarchical basis set sequence: • C44G* → pdpp → dddd • How do we know where we can start? • And how far we need to go? Y=Sf MMG Skills Lecture Series

20. Calculation strategy:hierarchy of costs • Sampling • Start with simplest viable sampling scheme: • Gamma-point, mp23,… • Use k-point parallelism • The Hamiltonians for different k-points are diagonalised in parallel, rather than in serial • parameter{use_kpar} • If nk<np, then np must be an integer multiple of nk. • If nk>np, then this is not the case. • How do we know what is “viable”? • How does this improve efficiency? • What are the potential pitfalls? MMG Skills Lecture Series

21. Calculation strategy:hierarchy of costs • Spin polarisation • For spin-polarized problems, first relax spin averaged – this makes the calculation run twice as fast. • What does spin averaged / polarised mean? • Can we always relax S=0? • Is it always really helpful? MMG Skills Lecture Series

22. Calculation strategy:hierarchy of costs • Supercell size • Start with a small unit cell and embed it in larger ones: • 64 → 216 → 512 → 1000 atoms • Use an anchor point • Take care over symmetry • What are the implications for timing? MMG Skills Lecture Series

23. Calculation strategy:hierarchy of costs • Starting structures continued… • Use the structure obtained from one charge state to start others. • Recycle similar systems: e.g. use a phosphorus structure to start an arsenic one. • If you’ve already run in LDA and you now want it GGA, scale the structure according to the ratio of standardized lengths (typically lattice constants). • What sort of errors might these short-cuts lead to, if any? MMG Skills Lecture Series

24. Calculation strategy:hierarchy of costs • Recycle the charge-density! • When restarting an incomplete relaxation, or restarting at the end of the relaxation to get some analytical data (e.g. AIMVIEW dumps, band-structures, EELS spectra...) • Use a “restart-dump”! • What does the restart dump contain? • restart{make-dump} • restart{load-dump} • restart{load-dump,override-positions} • How can the restart be used? • How much time might this save? MMG Skills Lecture Series

25. Restarting calculations: more generally • Your default mode of operation is to always write to a restart dump using: • restart{make-dump,file=dump.xxxxx} • You might store them all in one place • filespace{/scratch/njpg/DUMP-FILES} • This is what I do! • The xxxxx is a unique identifier associating the dump-file with the run. • The restart dump files may be very large in terms of disk-usage! • Restarts also exist for other calculations: • NEB • This will be discussed in detail elsewhere in this course • Energy second derivatives with respect to position • You should check the on-line documentation MMG Skills Lecture Series

26. Sometimes we have to bite the bullet apple(?!). After all this… MMG Skills Lecture Series

27. Throughput vs turn-around time • When running a parallel job, there is a scaling penalty: • Doubling the number of nodes will NOT half the time (in general). • Why is this? • It WILL prevent other people from using the nodes. • You will constantly balance throughput (the number of jobs completed per day) with turn-around time (wall time from start to job completion). • In general: • throughput is maximised by adopting the minimum number of nodes for the job, given the batch queue time limits • turn-around time is minimised by adopting larger numbers of nodes (subject to scaling). • Minimising the turn-around time is anti-social behaviour. • Big brother is watching you. MMG Skills Lecture Series

28. Interacting with a batch queue system • The batch queue systems differ from machine to machine. • It is important that you familiarise yourself with: • Memory per node • CPUs/cores per node • Time limits • Scheduling priority • Disk-usage • Scaling (interconnect) • Reliability  • How do these relate to the preceding discussion? MMG Skills Lecture Series

29. Interacting with a batch queue system • Every day, check all of your jobs • Running • Finished • For those still running, check to see: • Is all well? • Can/should this run be terminated? • When might this be? • Maximise the available resources for all users • For those finished: • Restart them, if need be • Incomplete relaxation • Analysis • File them carefully MMG Skills Lecture Series

30. Concluding thoughts • Be organised • Be hardware aware • Be efficient • Be sociable • THINK about what you’re doing. MMG Skills Lecture Series