1 / 62

ISC14 BoF : Drilling Down: Understanding User– Level Activity on Today’s Supercomputers

ISC14 BoF : Drilling Down: Understanding User– Level Activity on Today’s Supercomputers. Outline. Brief presentation Open discussion Demo. Robert McLay TACC. My Passions. Protect new user but stay out of vet's way Make staff support efficient and effective

milton
Download Presentation

ISC14 BoF : Drilling Down: Understanding User– Level Activity on Today’s Supercomputers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ISC14 BoF: Drilling Down: Understanding User–Level Activity on Today’s Supercomputers ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  2. Outline • Brief presentation • Open discussion • Demo ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  3. Robert McLayTACC ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  4. My Passions • Protect new user but stay out of vet's way • Make staff support efficient and effective • Automate detection, correction, prevention • Make the repeat tickets go away! ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  5. Making a difference… Maintain consistent, compatible software environment Lmod and related tools $ module swap mvapich2 impi Inactive Modules: 1) vasp Due to MODULEPATH changes the following have been reloaded: 1) fftw3/3.3.2 $ module load mvapich2 Lmod Error: You can only have one MPI module loaded at a time. You already have impi loaded. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  6. Making a difference… Detect potential problems and alert users Lariat and related tools TACC: Starting up job 423224 ****************************************************** WARNING: Your MPI Environment is: mvapich2/1.9a2 Your executable was built with: impi/4.1.0.030 ****************************************************** ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  7. Making a difference… Job-level usage data on libraries and applications ALTD (Mark Fahey -- NICS) ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  8. Joining forces… TACC: Starting up job 423224 ****************************************************** WARNING: Your MPI Environment is: mvapich2/1.9a2 Your executable was built with: impi/4.1.0.030 ****************************************************** Job-level usage data on libraries and applications Detect potential problems and alert users XALT ALTD Lariat ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  9. My own not-so-hidden agenda... • Looking for XALT beta users • Hungry for ideas, needs, feedback • Wanting to begin conversation with kindred souls ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  10. Mark FaheyUTK ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  11. ALTD • What it does • Intercepts linker (ln) and job launcher (aprun) • Uses linker tracemap option to get all libraries • Stores all of this in a database • What it gets • Full path of the executable • Static and dynamic libraries used by the executable • What it can be used for • Which executables use the largest number of core hours? • Are they managed by center? Do they use the system efficiently? • Which libraries, applications, or tools are being used? • Are there libraries we should remove? Are there libraries we should install? • What percentage of executables are scripts? • Are these scripts being used because the job starter isn’t sophisticated enough? • Are there any executables with modification times older than 1 year? • Should we ask the user to recompile? ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  12. What Does NERSC Collect? • ALTD • Track library usage both at compile and run time • Torque Logs • Job information, accounting • ALPS Logs • Track applications run time data and options on the Cray systems • Darshan • IO profiling data • IPM • MPI profiling data • Performance Monitoring • Monitoring system performance over the life time of the machines • LMT • Lustredata ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  13. ALTD is enabled on all major computing platforms at NERSC ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  14. Applications of ALTD An ALTD tool to restore the build environment for an application: aryal@edison12:~> linkinfo.sh /global/homes/a/aryal/bin/gvasp5.3.2 User : zz217 Linked on : 2013-01-03 Executable Name: vasp Libraries Used : //usr/lib64/libhugetlbfs.a ../vasp.5.lib/libdmy.a /opt/cray/atp/1.6.0/lib//libAtpSigHCommData.a /opt/cray/atp/1.6.0/lib//libAtpSigHandler.a /opt/cray/libsci/12.0.00/cray/81/sandybridge/lib/libsci_cray_mp.a /opt/fftw/3.3.0.1/x86_64/lib/libfftw3.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpich_cray.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpl.a /opt/cray/xpmem/0.1-2.0500.36799.3.6.ari/lib64/libxpmem.a /opt/cray/pmi/4.0.0-1.0000.9282.69.4.ari/lib64/libpmi.a /opt/cray/ugni/4.0-1.0500.5836.7.58.ari/lib64/libugni.a /opt/cray/udreg/2.3.2-1.0500.5931.3.1.ari/lib64/libudreg.a /opt/cray/alps/5.0.1-2.0500.7663.1.1.ari/lib64/libalpslli.a /opt/cray/alps/5.0.1-2.0500.7663.1.1.ari/lib64/libalpsutil.a /opt/cray/cce/8.1.2/craylibs/x86-64/libpgas-dmapp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/dmapp/4.0.1-1.0500.5932.6.5.ari/lib64/libdmapp.a /opt/cray/pmi/4.0.0-1.0000.9282.69.4.ari/lib64/libpmi.a /opt/cray/cce/8.1.2/craylibs/x86-64/libfi.a /opt/gcc/4.4.4/snos/lib64/libstdc++.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a /opt/cray/cce/8.1.2/craylibs/x86-64/libf.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymath.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcsup.a //usr/lib64/librt.a /opt/cray/cce/8.1.2/craylibs/x86-64/libtcmalloc_minimal.a //usr/lib64/libpthread.a //usr/lib64/libc.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a //usr/lib64/libm.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc.a • Understanding current library usage and plan for future software need • Providing usage statistics to developers and vendors • Restoring the program environment where user applications were built • Assisting with debugging system issues ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  15. ALTD at CSCS • In production at CSCS since 2011 • Rock solid: just a single downtime in two years • Rosa (Cray XE6) since March 2011 • 600K compilations, 2.8M jobs • Todi (Cray XK6/XK7) since October 2012 • 470K compilations, 500K jobs • Daint (Cray XC30) since March 2013 • 100K compilations, 550K jobs • We’ve added an additional SQL table “accounting” which logs more data about the application execution – number of cores used, number of cores claimed, number of threads, MPI processes, processes per node, … • We want to be able to detect situations like the use of a buggy or non-performant library ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  16. How we mine data: ahypotheticsituation A critical bug has been identified in FFTW version 3.3.0.2, affecting code correctness ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  17. First, find which users have linked this library mysql> select distinct username from altd_rosa_link_tags,altd_rosa_linkline where altd_rosa_link_tags.linkline_id=altd_rosa_linkline.linking_inc and exit_code=0 and linkline like '%fftw/3.3.0.2/%' ; +----------+ | username | +----------+ | tkachenn | | boswald | | liang | | robinson | | yunding | | zilia | +----------+ 5 rows in set (4.33 sec) • Querying the ALTD database reveals that several users have applications linked to the buggy library ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  18. Now, check if they are using the buggy application • And it’s confirmed that user “robinson” is running the application linked to the buggy library • It’s now up to the user services group to contact the user and recommend relinking their applications against the newer version of FFTW, which has fixed the bug mysql> select altd_rosa_jobs.* from altd_rosa_link_tags,altd_rosa_linkline,altd_rosa_jobs where altd_rosa_jobs.tag_id=altd_rosa_link_tags.tag_id and altd_rosa_link_tags.linkline_id=altd_rosa_linkline.linking_inc and exit_code=0 and linkline like '%fftw/3.3.0.2/%' and altd_rosa_jobs.username="robinson"; +---------+--------+------------------------+----------+------------+--------+---------------+ | run_inc | tag_id | executable | username | run_date | job_id | build_machine | +---------+--------+------------------------+----------+------------+--------+---------------| | 2410158 | 438583 | /users/robinson/mycode | robinson | 2013-11-05 | 834805 | rosa| | 2410172 | 438583 | /users/robinson/mycode | robinson | 2013-11-05 | 834805 | rosa | | 2410198 | 438583 | /users/robinson/mycode | robinson | 2013-11-05 | 834805 | rosa| | 2410222 | 438583 | /users/robinson/mycode | robinson | 2013-11-05 | 834805 | rosa| +---------+--------+------------------------+----------+------------+--------+---------------| 4 rows in set (0.65 sec) ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  19. This methodology is clearly unmanageable! • Ideally, user support specialists would be alerted automatically to “situations of interest” • Users running applications linked to legacy, less-performant, or buggy libraries • Users running legacy versions of applications • Users building code with legacy compilers • Users making use of their own libs or apps, when more optimized versions are available centrally How can we automate the processes of data mining, reporting and alerting? ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  20. Porting ALTD on IBM x86 and BGP • Neser x_86, • ALTD worked instantly for linking stage ( Intel and GNU compiler) • Postponed “temporarily” tracking the executable ( several OpenMPI versions 1.4.3, 1.5.4, 1.6.4,1.6.5, 1.7.3, 1.8.1) [x2 for Intel and GNU] • More effort needed for BGP • On this machine at least 4 sets of compilers are available : • Native IBM or GNU • For the front (64 bits) or core nodes (32 bits) • Every one of them has its dedicated linker  only patching/wrapping ld and mpirun was no more possible or sufficient • XL_LINKER environment available ( force linker to use ALTD like for Cray compilers) XL_LINKER –F ALTD_PATH+'/alias/dev/xlf_linker/xlf_xx._xx.cfg • Need to specify the appropriate ld with the corresponding .cfg file ISC14 : BoF ALTD at KAUST Supercomputing Lab

  21. New implementation of ALTD based on bash aliases • Each alias points to a unique python script $ALTD_ENV/rerouting/altd_alias.py • an analysis of the command line gives the name of the original command along with some interesting parameters (name of the compiler, executable or job file, requested number of processors) • depending on these information it eventually executes commands to add new information to the database, modifies of the running environment, or patches a job file before submitting it • runs the same command line, this time in an environment unaware of the aliases. • in case of error, the original command runs in an environment unaware of the aliases. The error is logged in a file and a mail is sent to ALTD maintainers. It gathers the encountered exception or error as well as numerous information on the users environment at that time • Known Issue:threaded IBM compiler , mpixf90_r, mpixlc_r • Error at runtime about “Insufficient memory to start application” • disabled aliases for mpi…._r compilers ISC14 : BoF ALTD at KAUST Supercomputing Lab

  22. ALTD Early mining ( 2months in full production) • ALTD fully ported for x86 and BGP • Identified: • few compiled codes, (our users are mostly black-box users or in productions) • Enabled storing jobs even if no ALTD header ( we store tag_id=0) • Linking • many users linked with libraries built in their home directories • Execution • Climate modeling ( mainly pre-processing) • Combustion ( S3D, NGA) • Molecular dynamic ( VASP, NAMD,…) • On going • Relink major software installed by the staff to track them better • additional tracking of other statistics (number of requested nodes vs actual used ) • work on a web-interface for automatic mining. ISC14 : BoF ALTD at KAUST Supercomputing Lab

  23. TACC_Stats • Job-level transparent performance monitoring from HPC compute nodes • CPU performance counters • IB statistics • Lustre statistics • Scheduler job statistics • Host data • OS statistics • Analyses integrate available Lariat data (XALT in the future) ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  24. XALT: Understanding the Software Needs of High End Computer Users • Newly NSF funded project • Is combining the best of Lariat and ALTD • Collecting job-level and link-time level data and subsequent analytics • Building a community around analytics – potentially one of many tools • Will make it available to the community • Optional interface to XDMod/SUPREMME ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  25. XALT Goals • Goal is a census of libraries and applications and automatic filtering of user issues • what additional user problems can we detect and report (perhaps correct) automatically? • How can we leverage lessons learned by the tacc stats team to implement additional automatic filtering? • Plan to add tracking of function calls as well • Want to balance the need for portability with support for site-specific capabilities • Want to simplify the processes system administrators use to install, configure, and manage ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  26. XALT Agenda • New tracking infrastructure – XALT • Alpha version available today • Deployed at NICS and TACC • LANL and CSCS testing it • Some new functionality still to add • Detect function calls • Check runtime environment versus compile time env • xalt-users@lists.sourceforge.net • Want feedback, hungry for ideas ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  27. Thanks to • Richard Gerber and Zhengji Zhao, NERSC • Tim Robinson, CSCS • Bill Barth, TACC • BilelHadri, KAUST • Julius Westerman, LANL ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  28. Contact Info • Mark R. Fahey • mfahey@utk.edu • Robert McLay • mclay@tacc.utexas.edu ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  29. Background Slides ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  30. Lariat User #1 Bill Barth Director of HPC, TACC Co-PI SUPreMM ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  31. TACC Stats • Job-level transparent performance monitoring from HPC compute nodes • CPU performance counters • IB statistics • Lustre statistics • Scheduler job statistics • Host data • OS statistics • Analyses integrate available Lariat data ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  32. Nightly Analyses • Automatically analyzes jobs nightly • Highlights jobs worth looking at • Tries to provide a one-stop view of a job for • Support staff • Sysadmins • And soon, users ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  33. Current Reports • High levels of imbalance • Low Flops (but other activity) • Idle hosts • Catastrophic performance drop ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  34. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  35. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  36. NERSC Job Data Richard GerberZhengji ZhaoNERSC User Services ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  37. What Does NERSC Collect? • ALTD • Track library usage both at compile and run time • Torque Logs • Job information, accounting • ALPS Logs • Track applications run time data and options on the Cray systems • Darshan • IO profiling data • IPM • MPI profiling data • Performance Monitoring • Monitoring system performance over the life time of the machines • LMT • Lustredata ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  38. Expose job data via the web • We try to make as much data available as possible via the web • For users to track usage • For users to check resource utilization • For users to monitor performance • For staff to help debug jobs • For summary reports • The following are web screen shots • All data collection is transparent to users ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  39. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  40. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  41. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  42. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  43. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  44. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  45. ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  46. ALTD is enabled on all major computing platforms at NERSC ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  47. Applications of ALTD An ALTD tool to restore the build environment for an application: aryal@edison12:~> linkinfo.sh /global/homes/a/aryal/bin/gvasp5.3.2 User : zz217 Linked on : 2013-01-03 Executable Name: vasp Libraries Used : //usr/lib64/libhugetlbfs.a ../vasp.5.lib/libdmy.a /opt/cray/atp/1.6.0/lib//libAtpSigHCommData.a /opt/cray/atp/1.6.0/lib//libAtpSigHandler.a /opt/cray/libsci/12.0.00/cray/81/sandybridge/lib/libsci_cray_mp.a /opt/fftw/3.3.0.1/x86_64/lib/libfftw3.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpich_cray.a /opt/cray/mpt/5.6.0/gni/mpich2-cray/74/lib/libmpl.a /opt/cray/xpmem/0.1-2.0500.36799.3.6.ari/lib64/libxpmem.a /opt/cray/pmi/4.0.0-1.0000.9282.69.4.ari/lib64/libpmi.a /opt/cray/ugni/4.0-1.0500.5836.7.58.ari/lib64/libugni.a /opt/cray/udreg/2.3.2-1.0500.5931.3.1.ari/lib64/libudreg.a /opt/cray/alps/5.0.1-2.0500.7663.1.1.ari/lib64/libalpslli.a /opt/cray/alps/5.0.1-2.0500.7663.1.1.ari/lib64/libalpsutil.a /opt/cray/cce/8.1.2/craylibs/x86-64/libpgas-dmapp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/dmapp/4.0.1-1.0500.5932.6.5.ari/lib64/libdmapp.a /opt/cray/pmi/4.0.0-1.0000.9282.69.4.ari/lib64/libpmi.a /opt/cray/cce/8.1.2/craylibs/x86-64/libfi.a /opt/gcc/4.4.4/snos/lib64/libstdc++.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a /opt/cray/cce/8.1.2/craylibs/x86-64/libf.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymath.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcraymp.a /opt/cray/cce/8.1.2/craylibs/x86-64/libu.a /opt/cray/cce/8.1.2/craylibs/x86-64/libcsup.a //usr/lib64/librt.a /opt/cray/cce/8.1.2/craylibs/x86-64/libtcmalloc_minimal.a //usr/lib64/libpthread.a //usr/lib64/libc.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc_eh.a //usr/lib64/libm.a /opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4/libgcc.a • Understanding current library usage and plan for future software need • Providing usage statistics to developers and vendors • Restoring the program environment where user applications were built • Assisting with debugging system issues ISC14 BoF: Drilling Down: Understanding User-Level Activity on Today's Supercomputers

  48. Monitoring Software Usage at CSCS Dr Tim Robinson CSCS Drilling Down: Understanding User-Level Activity on Today’s Supercomputers

  49. We support many, many libs, tools, apps, compilers…

  50. How can wetellif software isactuallybeingused? • Support staff want to answer questions like • is anyone using a legacy version of a certain library or application? • Research teams, software developers, vendors, and funding agencies need to know • is their software being used? Where should they place effort for future development? • User surveys? • Counting “module loads”? A better solution: the Automatic Library Tracking Database (Fahey, Jones, and Hadri, Cray User Group meeting, 2010)

More Related