Parallel Methods for Nano/Materials Science Applications

1. Parallel Methods for Nano/Materials Science Applications (Electronic Structure Calculations) Andrew Canning Computational Research Division LBNL & UC Davis

2. Outline • Introduction to Nano/Materials science • Electronic Structure Calculations (DFT) • Code performance on High Performance Parallel Computers • New Methods and Applications for Nanoscience

3. Milestones in Parallel Calculations 1991 Silicon surface reconstruction (7x7), Phys. Rev. (Stich, Payne, King-Smith, Lin, Clarke)Meiko I860, 64 processor Computing Surface (Brommer, Needels, Larson, Joannopoulos)Thinking Machines CM2, 16,384 bit processors 1998 FeMn alloys (exchange bias), Gordon Bell prize (Ujfalussy, Stocks, Canning, Y. Wang, Shelton et al.)Cray T3E, 1500 procs. first > 1 Tflop Simulation 2005 1000 atom Molybdenum simulation with QboxSC05. (F. Gygi et al. )BlueGene/L, 32,000 processors (LLNL)

4. Electronic Structure Calculations • Accurate Quantum Mechanical treatment for the electrons • Each electron represented on grid or with some basis functions (eg. fourier components) • Compute Intensive: Each electron requires 1 million points/basis (need 100s of electrons) • 70-80% NERSC Materials Science Computer Time (first-principles electronic structure) InP quantum dot (highest electron energy state in valence band)

5. Motivation for Electronic Structure Calculations • Most Materials Properties Only Understood at a fundamental level from Accurate Electronic Structure (Strength, Cohesion etc) • Many Properties Purely Electronic eg. Optical Properties (Lasers) • Complements Experiments • Computer Design Materials at the nanoscale

6. Materials Science Methods • Many Body Quantum Mechanical Approach (Quantum Monte Carlo) 20-30 atoms • Single Particle QM (Density Functional Theory) No free parameters. 100-1000 atoms • Empirical QM Models eg. Tight Binding 1000-5000 atoms • Empirical Classical Potential Methods thousand-million atoms • Continuum Methods atoms

7. Ab initio Method: Density Functional Theory (Kohn 98 Nobel Prize) Many Body Schrodinger Equation (exact but exponential scaling ) Kohn ShamEquation (65): The many body ground state problem can be mapped ontoa single particle problem with the same electron density and a differenteffective potential (cubic scaling). Use Local Density Approximation (LDA) for (good Si,C)

8. Selfconsistent calculation N electrons N wave functions lowest N eigenfunctions Selfconsistency

9. Choice of Basis for DFT(LDA) Increasing basis size M Gaussian FLAPW Fourier grid Percentage of eigenpairs M/N 30% 2% Eigensolvers Direct (scalapack) Iterative

10. Plane-wave Pseudopotential Method in DFT Solve Kohn-Sham Equations self-consistentlyfor electron wavefunctions within the Local Density Appoximation 1. Plane-wave expansion for 2. Replace “frozen” core by a pseudopotential Different parts of the Hamiltonian calculated in different spaces (fourier and real) 3d FFT used

11. PARATEC (PARAllel Total Energy Code) • PARATEC performs first-principles quantum mechanical total energy calculation using pseudopotentials & plane wave basis set • Written in F90 and MPI • Designed to run on large parallel machines IBM SP etc. but also runs on PCs • PARATEC uses all-band CG approach to obtain wavefunctions of electrons • Generally obtains high percentage of peak on different platforms • Developed with Louie and Cohen’s groups (UCB, LBNL), Raczkowski

12. PARATEC: Code Details • Code written in F90 and MPI (~50,000 lines) • 33% 3D FFT, 33% BLAS3, 33% Hand coded F90 • Global Communications in 3D FFT (Transpose) • Parallel 3D FFT handwritten, minimize comms. reduce latency (written on top of vendor supplied 1D complex FFT )

13. PARATEC: Parallel Data distribution and 3D FFT • Load Balance Sphere by giving columns to different procs. • 3D FFT done via 3 sets of 1D FFTs and 2 transposes • Most communication in global transpose (b) to (c) little communication (d) to (e) • Flops/Comms ~ logN • Many FFTs done at the same time to avoid latency issues • Only non-zero elements communicated/calculated • Much faster than vendor supplied 3D-FFT (a) (b) (c) (d) (e) (f)

14. PARATEC: Performance • All architectures generally achieve high performance due to computational intensity of code (BLAS3, FFT) • ES achieves highest overall performance to date: 5.5Tflop/s on 2048 procs • Main ES advantage for this code is fast interconnect • SX8 achieves highest per-processor performance • X1 shows lowest % of peak • Non-vectorizable code much more expensive on X1 • IBM Power5 4.8 Gflops/P (63% peak on 64 procs) • BGL got 478 Mflops/P (17% of peak on 512 procs) Developed with Louie and Cohen’s groups (UCB, LBNL), also work with L. Oliker, J Carter

15. Self-consistent all band method for metallic systems • Previous methods use self-consistent (SC) band by band, with Temperature smearing (eg. VASP code) drawbacks – band-by-band slow on modern computers (cannot use fast BLAS3 matrix-matrix routines) • New Method uses occupancy in inner iterative loop with all band Grassman method (GMCG method) Al (100) surface, 10 layers + vacuum GMCG: new method with occupancy

16. Self-consistent all band method for metals KS - DFT Potential Mixing Vout  Vin

17. The Quantization Condition of Quantum-well States in Cu/Co(100) photon beam in electrons out d 54 Å Copper Wedge Cobalt layer 4 mm scan Copper substrate 0 • Theoretical investigation of Quantum Well states in Cu films using our codes (PARATEC, PEtot) to compare with experiments at the ALS (E. Rotenberg, Y.Z. Wu, Z.Q. Qiu) • New computational methods for metallic systems used in the calculations. • Lead to an understanding of surface effects on the Quantum Well States. Improves on simple Phase Accumulation Model used previously QW states in Copper Wedge Difference between theory and experiment improved by taking surface effects into account

18. Computational challenges (larger nanostructures) atoms nanostructures bulk molecules Infinite (1-10 atoms in a unit cell) 1-100 atoms 1000-10^6 atoms size Challenge for computational nanoscience. Ab initio Method PARATEC • Ab initio method • PARATEC method Ab initio elements and reliability New methodology and algorithm (ESCAN) Even larger Supercomputer

19. Example: Quantum Dots (QD) CdSe • Band gap increase CdSe quantum dot (size) • Single electron effects • on transport (Coulomb • blockade). • Mechanical properties, • surface effects and no • dislocations

20. Charge patching method for larger systems(Wang) Non-selfconsistent LDA quality potential for nanotube Selfconsistent LDA calculation of a single graphite sheet Get information from small system ab initio calc., then generate the charge densities for large systems

21. Motif based charge patching method (Wang) Error: 1%, ~20 meV eigen energy error.

22. N + Folded Spectrum Method (ESCAN)

23. Charge patching: free standing quantum dots In675P652 LDA quality calculations (eigen energy error ~ 20 meV) L-W Wang 64 processors (IBM SP3) for ~ 1 hour Total charge density motifs CBM VBM

25. Nanowire Single Electron Memory Samuelson group Lund, Sweden Nano Letters Vol2, 2, 2002.

26. Nanowire Single Electron Memory (LOBPCG) • Comparison of LOBPCG with band by band CG (64 procs on IBM SP) • Matrix Size = 2,265,837 (nano-wire InP InAs with 67,000 atoms) Using code to determine size regimes in which single electron behavior occurs (~60nm length, ~20nm diameter), also using LCBB code for larger systems. Work carried out with G. Bester S. Tomov, J. Langou

27. Future Directions • O(N) based methods (exploit locality) gives sparse matrix problem • Excited state calculations • Transport Calculations

28. Multi-Teraflops Spin Dynamics Studies of the Magnetic Structure of FeMn and FeMn/Co Interfaces Exchange bias, which involves the use of an antiferromagnetic (AFM) layer such as FeMn to pin the orientation of the magnetic moment of a proximate ferromagnetic (FM) layer such as Co, is of fundamental importance in magnetic multilayer storage and read head devices. A larger simulation of 4000 atoms of FeMn ran at 4.42 Teraflops 4000 processors. (ORNL, Univ. of Tennessee, LBNL(NERSC) and PSC) IPDPS03 A. Canning, B. Ujfalussy, T.C. Shulthess, X.-G. Zhang, W.A. Shelton, D.M.C. Nicholson, G.M. Stocks, Y. Wang, T. Dirks Section of an FeMn/Co (Iron Manganese/ Cobalt) interface showing the final configuration of the magnetic moments for five layers at the interface. Shows a new magnetic structure which is different from the 3Q magnetic structure of pure FeMn. Contact: Andrew Canning (ACanning@lbl.gov)

29. Conclusion Large scale supercomputers First principle calculation New algorithm methodology + + Accurate Nanostructures simulations