Doing More With Less: Managing Software at a DoD HPC Center

1. Doing More With Less:Managing Software at a DoD HPC Center Jay Blair CSC High Performance Computing Center of Excellence

2. Introduction • DoD HPCMP • The High Performance Computing Modernization Program (HPCMP) was initiated in 1992 in response to congressional direction to modernize the Department of Defense (DoD) laboratories' high performance computing (HPC) capabilities • Major Shared Resource Centers • Complete HPC environment, including hardware, software, data storage, archiving, visualization, training, and expertise in specific computational technology areas

3. Introduction • ASC MSRC • Wright-Patterson AFB, OH • Over 3000 CPUs on the floor • HP, IBM, SGI & Linux • Thousands of users (Gov, civilian, contractor) • Software usage is mix of home grown and commercial • 80% home grown • 20% commercial • Broken down into computational technology areas • Jobs are managed in a batch only environment

4. The Dilemma Wants and Needs Users Vendors Neverland Management

5. Vendors Cost Users Perfect Solution The Method

6. 5 What’s • What do we have? • What is being used? • What do we need? • What are the options? • What will the future hold?

7. What Do We Have? • Software • What capabilities does the vendor say we have • What is the competition • How many seats/licenses/tokens • What are the lifecycle and maintenance costs • What is the license model • Floating, node locked or token • Open source • Lease / Paid up

8. What Is Being Used? • Log files are your friends • Parsing in Perl • Macrovision tools • SAM Suite aka FLEXnet Manager • Metrics of merit • CPU hours • Number of license hits • Number of users • Maximum concurrent users • Denials (Hours unavailable)

9. What Do We Need? • Vendor NDA • Site specific issues • Hardware/OS support • The alpha is dead… long live the alpha • Security • No root, no gui, no problem • Probe for interest in working relationship • White papers • Tech day • Usage demographics • 36 month technology roadmap

10. What Do We Need? • User Input • What do they need • Functional needs • Experience base • Training budget • HTML survey to targeted group of users • Brief • Highly focused questions • Leverage intranet or in house web tools • Publicize the results • Graph it

11. What Do We Need? • Management thoughts • Corporate initiatives • Left asks: Right what are you doing? • Existing partnerships • Corporate rates, agreements, initiatives… • Leverage • Do not reinvent the wheel • Timeline • Time is money • Access to data • Is effort in the loop • Do you have autonomy?

12. What Do We Need? • Proof is in the pudding… er… benchmark • User requirements drive problem set • Best / worst of class • Know the answer and tell the vendor • Brief vendor • What • Why • Time • Cost/Benefit • Performance

13. What Are the Options? • Usual Suspects • Add • Remove • Retain • Users want the same or more as long as you pay • Management wants less because they do pay • Vendors want more because those payments on that Jag and the timeshare in Cancun aren’t getting any cheaper!

14. What Are the Options? • Fallacy of Removal • Cost savings for removal is never 100% • More than likely have to add to remove • Consider freezing maintenance on paidups • Consider divesture to another group • Trade in value to OEM or competitor • Consider Open Source • Free as in free or as in beer?

15. What Will the Future Hold? • The Art and Practice of Prediction • Historical trends • Current usage • User-centric • Project-centric • Business methodology • Make and Break • Simulation • If budget allows… Pad

16. Anne Bob Cost per User Cost per Hour Chris Dan Ellen Analysis Methodologies • Return on Investment • Ingredients • Hours • Costs • Users

17. Hours Jobs Cost Utility – Area Number of Users Analysis Methodologies • Software Utility • Ingredients • Users • Hours • Number of Jobs • Cost

18. All Software for FY04 Cinco Software for FY04 Cost per Hour Cost per Hour Dos Uno Zeta Beta Tres Seis Delta Alpha Cinco Epsilon Cuatro Gamma Analysis Methodologies • Historical Baseline • Examine costs over time • Across all applications • Across groups of similar functionality

19. Max Concurrent User Number of Jobs Jan Feb May April June March Analysis Methodologies • Software Usage • Ingredients • Usage • Hours Cinco Software Vendor Epsilon

20. Example CSM • Computational Structural Mechanics • In 2002 there were four major commercial codes • Alpha, Beta, Gamma, Delta • Maintenance tail was excessive • Contract with most expensive vendor was at end of life • Some paid-up licenses and some leases • Number of licenses was large Goal was to reduce cost w/o adversely affecting users

21. Example CSM What Is Being Used?

22. Example CSM Describe your usage of CSM software

23. Example CSM How much alpha code experience do you have?

24. Example CSM How many resources do you request per job?

25. Example If you could run a parallel job, would you?

26. Example CSM Features of Importance

27. Example CSM • User Defined Benchmarks • Pulled from vendor documentation • Industry standards • User examples • 25 problems • Answers had to have decks and results • Verification performed • “Trivial” – Nameless Vendor • 6 week timeline

28. = Support via 3rd party package = Not Available Example CSM CSM Solvers Feature Matrix

29. CSM Decision • Codes are functional equivalent at the 85%+ level • Removed 2 packages (leased codes) • Added seats to 2 packages • Added 1 new package • 6 weeks of training Net savings in excess of $3M over 5 years No unhappy users

30. Example CCM • Computational Chemistry Codes • In 2004 there were several major commercial codes • Alpha, Beta, Gamma, Delta, Epsilon, Zeta • Large portion of costs in single code • Contract with most expensive vendor was at end of life • User requirements were undefined hence vendor was in control of situation Goal was to right size w/o adversely affecting users

31. Example CCM

32. Hours Jobs Cost Number of Users Example CCM GRAPHITE / ROI

33. Example CCM What Is Being Used?

34. Cost per User Cost per Hour Cost Zeta Beta Delta Alpha Epsilon Gamma Example CCM What are the costs across CCM?

35. Cost per User Cost per Hour Cost Features Example CCM What are the features of Alpha are we usingand their cost?

36. CCM Decision • Not all features needed • Reduced number of features within a package • Pushed high $/hr or $/user features to users that need them • Supported the idea of a core set of necessary features • Offset some use to Government or Open Source codes • Multiyear lease deal with vendor Net savings in excess of $800K over 5 years Users are more aware of metrics

37. Summary • Software Utility is possible in a Government environment • Users will come along if you make a good case • Look to commercial offerings to increase value add • Macrovision FLEXnet Manager • Information is power with OEM vendors • Decreasing costs can be done so that users do not suffer

38. Thanks! • You’ve Got Questions: Excellent, I have answers!