1 / 32

Linux on Power – Hints and Tips

Linux on Power – Hints and Tips. Daniel Martin November 19 th 2013 daniel.martin@uk.ibm.com Find me on developerWorks. 1. Contents. Porting from Intel x86 to Linux on Power - Thoughts IBM POWER Linux Tools Repository IBM POWER Linux Tools Repository – Install

deepak
Download Presentation

Linux on Power – Hints and Tips

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Linux on Power – Hints and Tips Daniel Martin November 19th 2013 daniel.martin@uk.ibm.com Find me on developerWorks 1

  2. Contents • Porting from Intel x86 to Linux on Power - Thoughts • IBM POWER Linux Tools Repository • IBM POWER Linux Tools Repository – Install • IBM POWER Linux Performance Recommendations • IBM POWER Linux Java Install and Performance Recommendations. • Redhat on IBM POWER – Tips • Obtaining examples - SPEC.org • Leveraging IBM – developerWorks, VLP, IIC, Webinars 2

  3. Porting from Intel x86 to Linux on Power - Thoughts • IBM POWER7 and POWER7+ recommended distros - • The following distributions are optimised for POWER7 and POWER7+ processor-based servers: • SUSE Linux Enterprise Server 11 SP1, any subsequent service packs. SLES 11 SP2 is the latest service pack. • Red Hat Enterprise Linux 6, and subsequent updates. RHEL 6.4is the latest update. • PowerLinux Tools – • There are a number of related tools that you should install to get the very best out of your Linux on Power systems. These are: IBM Linux hardware diagnostic aids and productivity, IBM Software Development Kit and the IBM Advance Toolchain. 3

  4. IBM POWER Linux Tools Repository + Installation 4

  5. IBM POWER Linux Tools Repository • The IBM POWER Linux Tools Repository enables the use of standard Linux package management tools to provide access to the following: • IBM POWER Linux hardware diagnostic aids and productivity tools • IBM Software Development Toolkit for POWER Linux servers • IBM Advance Toolchain for POWER Linux servers • The Tools repository supports the following versions of Linux: • Red Hat Enterprise Linux 5 (RHEL5) • Red Hat Enterprise Linux 6 (RHEL6) • SuSE Linux Enterprise Server 10 (SLES10) • SuSE Linux Enterprise Server 11 (SLES11) 5

  6. IBM POWER Linux Tools Repository - Install Installing the IBM POWER Linux hardware diagnostic aids and productivity tools. Create your repo information and then update the new repo: # vi /etc/yum.repos.d/ibm-power.repo [ibm-power-repo] name=IBM POWER Tool baseurl=http://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/6/ppc64 enabled=1 gpgcheck=0 # yum update Once the updates are downloaded, create the related libraries: # yum install ibm-power-repo.ppc Confirm what repos are now available: # yum repolist enabled 6

  7. IBM POWER Linux Tools Repository - Install Update the installed packages to the latest version available, enter: # yum update Then choose one of the following to install the recommended packages. For an HMC or IVM-managed system (in most cases), enter: # yum install ibm-power-managed-rhel6 OR For a standalone system, enter: # yum install ibm-power-nonmanaged-rhel6 I recommend installing these additional packages: Performance Management # yum install ibmPMLinux NMON– Performance monitoring # yum install nmon POWER Energy Management # yum install pseries-energy NUMA policy for processes or shared memory # yum install numa* 7

  8. IBM POWER Linux Performance Recommendations 8

  9. IBM POWER Linux Performance Recommendations • Environmental Settings – • These are a number of recommended environmental settings based on a system with 16 POWER7+ cores and 128GB RAM. • fs.file-max = 4878674 • This relates to the maximum number of open files/file descriptors: • # cat /proc/sys/fs/file-max • 201561 • This will set the limit to be applied to the system after the next reboot: • vi /etc/sysctl.conf • Adding the line: • fs.file-max = 4878674 • Run the following to re-read the sysctl.conf file and apply it to the current system: • # sysctl -p 9

  10. IBM POWER Linux Performance Recommendations • Setting the maximum number of open file descriptors. • Displaying your current limits: • # ulimit -n • 1024 • Changing those limits on your current session: • # ulimit -n 1048576 • Preserving the change on system reboot: • # vi /etc/security/limits.conf • Add the lines: • * hard nofile 1048576 • * soft nofile 786432 • Then repeat the same principles. • This relates to the maximum number of processes available to a single user: • # ulimit -u unlimited • This relates to the maximum stack size: • # ulimit -s 524288 10

  11. IBM POWER Linux Performance Recommendations • The previous slides cover a few environmental settings, the following are some which relate to the CPU and its features which may help improve performance. • Disabling ‘preempt scheduler’: • # mount -t debugfs debugfs /sys/kernel/debug# echo NO_WAKEUP_PREEMPT > /sys/kernel/debug/sched_features# umount /sys/kernel/debug • It can be re-enabled via: • # echo WAKEUP_PREEMPTION > /debug/sched_features 11

  12. IBM POWER Linux Performance Recommendations Disable Idle Power Saver - Idle Power Saver can be disabled by accessing your browser and utilising the Advanced System Management Interface (ASMI) console. From the HMC, navigate to System Configuration -> Power Management Mode Setup -> Disable Power Save Mode. Set the Idle Power Saver value to Disabled, then click on the "Save settings" button at the bottom of the page.   12

  13. IBM POWER Linux Performance Recommendations • Disable IPv6 • IPv6 support can be disabled in the Linux kernel by adding the following options to the command line in the boot loader configuration: • ipv6.disable_ipv6=1 ipv6.disable=1 • Disabling IPv6 support in the Linux kernel guarantees that no IPv6 code will ever be run while the system is booted. Always remember that if you run this command then you cannot enable IPv6 until the system is rebooted. A lighter touch is to let the kernel boot with IPv6 support and then use the sysctl facility to dynamically set a kernel variable to disable IPv6: • sysctl -w net.ipv6.conf.all.disable_ipv6=1 • The example above disables IPv6 on all interfaces. You can also opt to just disable IPv6 support on specific interfaces: • sysctl -w net.ipv6.conf.eth0.disable_ipv6=1sysctl -w net.ipv6.conf.eth1.disable_ipv6=1 • If your workload only uses IPv4 (in order to allow you to disable IPv6 support in the JVM) add the flag: • -Xaggressive • There is no reason for the JVM to spend any time trying to use IPv6. 13

  14. IBM POWER Linux Java Installation – Performance Recommendations 14

  15. IBM POWER Linux Java Installation Red Hat Java install - Firstly here are a couple of binaries that you will need to ensure that the Java ‘installanywhere’ script works: # yum install libstdc++.ppc64# yum install rpm-build Then you can download the related package (in our case it’s Java7) into the install directory (/usr). Once that is complete, run the *.bin file: Java 7 # ./ibm-java-sdk-7.0-4.0-ppc64-archive.bin Java 6# ./ibm-java-sdk-6.0-13.0-linux-ppc64.binThis should enable you to simply follow the prompts from the install script, and as long as you have sufficient space all should configure correctly.Check the version as follows: # /usr/<install-path>/bin/java -version 15

  16. IBM POWER Linux Java Performance Recommendations • Further Recommendations: • Turn off Hardware data prefetch with the Linux command "ppc64_cpu --dscr=1“. • The default page size is 64 KB page size, this will normally provide good performance gains over 4 KB pages. For further potential performance gains, enable 16 MB page size -  • # cat /proc/meminfo • Predominantly the best performance is from 32-bit Java, that is unless the memory requirement of the application requires you to be running in 64-bit mode.  Make sure libgcc.ppc and glibc.ppc are installed otherwise the 32bit Java will not include the correct libraries. • /root/.bashrc  - set the path in your profile to Java 32 or 64 bit. • Most applications benefit from SMT. However, some applications do not scale with an increased number of logical CPUs on an SMT enabled system. One way to address such an application scalability issue is to change to a lower SMT mode with fewer logical CPUs; for example, changing from SMT4 to SMT2.  Details of SMT will follow. 16

  17. Redhat on IBM POWER – Tips 17

  18. Redhat on IBM POWER – Tips The server is now installed, below are a few commands to assist you in tailoring your environment. POWER SMT on Red Hat. How does Linux handle SMT modes, as IBM POWER7 processors support Simultaneous Multi-Threading (SMT)?To toggle the CPUs offline/online consistently across the cores:# ppc64_cpu --smt=xLinux numbers all of the CPUs sequentially across the cores:# ppc64_cpu --smt=1 This will "turn off" cpu1,cpu2, cpu3 of the four CPUs associated with each core, leaving cpu0 running. 0,1,2,3 is the example for the first core. This repeats throughout the CPUs:# ppc64_cpu --smt=2 This ensures that cpu0, cpu1 are "on", and cpu2, cpu3 are "off" for each core:# ppc64_cpu --smt=4 This ensures that all cpus are "on“With SMT=1 mode (aka SMT off), the Linux scheduler should shift to the appropriate scheduling mode. 18

  19. Redhat on IBM POWER – Tips Useful Commands. If you want to get virtual machine configuration data, then the ’lparcfg’ file contains information relating to the IBM Power Virtual Machine: [root~]# cat /proc/ppc64/lparcfg The interactive command version of this is ‘lparstat’ [root~]# # lparstat System Configuration type=Shared mode=Uncapped smt=Dedicated lcpu=1 mem=2051328 kB cpus=32 ent=0.40 %user %sys %wait %idle physc %entc lbusy vcsw phint ----- ----- ----- ----- ----- ----- ----- ----- ----- 0.13 0.17 0.05 99.65 0.00 0.00 0.30 103475 0 [root]# lparstat 5 2 System Configuration type=Shared mode=Uncapped smt=Dedicated lcpu=1 mem=2051328 kB cpus=32 ent=0.40 %user %sys %wait %idle physc %entc lbusy vcsw phint ----- ----- ----- ----- ----- ----- ----- ----- ----- 0.05 0.00 0.00 99.95 0.00 0.00 0.05 130373 0 0.00 0.00 0.00 100.00 0.00 0.00 0.00 130475 0 The –i flag lists details of the LPAR configuration. [root ~]# lparstat -i Node Name : server1.iic.hur.cdn Partition Name : server1 Partition Number : 6 Type : Shared Mode : Uncapped Entitled Capacity : 0.40 Partition Group-ID : 32774 Shared Pool ID : 0 Online Virtual CPUs : 1 Maximum Virtual CPUs : 4 Minimum Virtual CPUs : 1 Online Memory : 2051328 kB Minimum Memory : 1024 Desired Variable Capacity Weight : 128 Minimum Capacity : 0.10 Maximum Capacity : 4.0 Capacity Increment : 0.1 Active Physical CPUs in system : 32 Active CPUs in Pool : 32 Maximum Capacity of Pool : 32.0 Entitled Capacity of Pool : 290 Unallocated Processor Capacity : 0 Physical CPU Percentage : 40 Unallocated Weight : 0 Memory Mode : Shared Total I/O Memory Entitlement : 2147483648 Variable Memory Capacity Weight : 0 Memory Pool ID : 65535 Unallocated Variable Memory Capacity Weight : 0 Unallocated I/O Memory Entitlement : 0 Memory Group ID of LPAR : 32774 Desired Variable Capacity Weight : 128 19

  20. Redhat on IBM POWER – Tips • Linux Performance Customer Profiler Utility (lpcpu) • This script captures potentially interesting performance data (profile information, system information, and some system configuration information) on a single pass approach. Gathering all of this information at once allows the context to be understood and the performance profiling data to be analyzed. • The script will check to ensure you have the basic tools installed on your system. • This script takes advantage of all of the "normal" performance tools used on Linux: • iostat • mpstat • vmstat • perf • meminfo • top • sar • oprofile • perf • In addition, relevant system information is gathered with the profiler output into a single tarball saved on your system. By default the file is saved in /tmp. Useful Commands continued: CPU settings utility Determine cpu frequency: [root ~]# ppc64_cpu --frequency min: 4.31 GHz (cpu 2) max: 4.31 GHz (cpu 3) avg: 4.31 GHz Obtain the number of cores installed: [root ~]# ppc64_cpu --cores-present Number of cores present = 1 Obtain the number of cores currently online: [root ~]# ppc64_cpu --cores-on Number of cores online = 1 [root@eg02ph05 ~]# 20

  21. Redhat on IBM POWER – Tips RHEL Kernel image too large? I have seen this problem a few times whilst trying to install (DVD/ISO or Network) RHEL on IBM POWER:Welcome to yaboot version 1.3.14 (Red Hat 1.3.14-35.el6)Enter "help" to get some basic usage informationboot: linuxPlease wait, loading kernel...   Elf64 kernel loaded...Loading ramdisk...Claim failed for initrd memory at 02000000 rc=ffffffff After some searching on Bugzilla I discovered my problem, and from this I did a little testing to resolve the issue which I have detailed below:Boot the IBM POWER server and enter the ‘OpenFirmware’ prompt with selection 8 in the IBM menu, and run the following command at the prompt:0 > printenv real-base If you see the following the I-------------- Partition: common -------- Signature: 0x70 ---------------real-base                2000000             2000000  21

  22. Redhat on IBM POWER – Tips Continued: RHEL Kernel image too large? The issue is that firmware is expecting an image of 32MB, and in our example for RHEL6 it is 16MB (or smaller) so this is how to correct this issue: 0 > setenv real-base 1000000 This sets the expected image to 16MB, which we can then verify: 0 > printenv real-base -------------- Partition: common -------- Signature: 0x70 ---------------real-base                1000000              2000000 Now perform a reboot:0 > reset-allThe system should then boot off of the RHEL image and the installs should now work as expected. In this instance it was a virtual server so until the LPAR is removed from the HMC it will continue to work. Ref: 2000000 is 32MB 1800000 is 24MB 1000000 is 16MB c00000 is 12MB 22

  23. Obtaining examples - SPEC.org 23

  24. Obtaining examples - SPEC.org The previous section covers a few changes that can be made in relation the operating system, and it can be difficult to find reference material and example systems to test comparably too. I would also recommend looking at the www.spec.org website. Searching in the ‘SPECjEnterprise2010’ section will also yield some useful results. 24

  25. Obtaining examples - SPEC.org You can view all of the results or search for a direct example: The website helpfully includes all of the related environmental information, such as DB2, JVM, ulimits, and other parameters. 25

  26. Leveraging IBM – Resources and Support 26

  27. PowerLinux Resources and Support Hardware & Facility Access Special discounted ISV Developer Box Purchase or lease PowerLinux servers for development or in-house use at special discounted rates. Virtual Loaner Program (VLP) Reserve and provision IBM Power Systems online for migration, porting, testing and solution demonstrations. ibm.com/systems/vlp IBM Innovation Centers and Linux Technology Centers Get remote and on-site access to IBM Power Systems with dedicated support at no charge. Build customized environments, conduct optimization, or host client workshops. IBM Innovation Centers ibm.com/partnerworld/wps/servlet/ContentHandler/isv_com_tsp_iic_overview IBM Linux Technology Centers ibm.com/linux/ltc/index.html Technical Assistance ISV Application Porting Assistance Get help to port and enable your applications to IBM Systems platforms. ibm.com/partnerworld/mem/support/trs_develop_rapidport.html Chiphopper Assistance to port, test, and support your existing Linux x86 applications on other IBM Systems and middleware platforms—at no cost. ibm.com/isv/go/chiphopper IBM Migration Factory Address your customers’ concerns when migrating from Solaris/HPUX/ x86 to PowerLinux with migration expertise and methodologies from IBM ibm.com/systems/migratetoibm/factory IBM developerWorks – A Technical Community for PowerLinux Get access to blogs, wiki, and message boards for recent news, discussions, events and assistance. ibm.com/developerworks/group/tpl SDK & Tools IBM Installation Toolkit for PowerLinux A simple tool to install and configure Red Hat Enterprise Linux and SUSE Linux Enterprise Server, open source workloads, as well as IBM value-added software on Power Systems. ibm.com/webapp/set2/sas/f/lopdiags/installtools Software Development Toolkit for PowerLinux All-in-one solution integrating Linux and IBM tools such as OProfile, Perf, Valgrind, and Autotools for Linux and FDPR for IBM. You can even use the Eclipse IDE directly on the Power Systems server or in x86_64 clients for remote development. ibm.com/support/customercare/sas/f/lopdiags/sdklop.html 27

  28. Technical Support Face to face or remote Proof of concept IBM products consulting Prototyping Testing Porting IBM platforms Validation Integration Performance/scalability testing Sizing guides SmartCloud Education and events Extensive program of technical education on the latest IBM technologies LoB events focused on: Maximising your Relationship with IBM Industry focused seminars New business models Services offered at the IIC Hursley All at no charge • Infrastructure (onsite and virtual) • Latest hardware and software • Industry and cloud solutions • Technical expertise • Solutions Laboratories • Smarter Planet • Smarter Cities • Smarter Commerce • Smarter Energy • Smarter Government • Intelligent Operations Centre 28

  29. Facilities, Infrastructure and Skills World-class facilities Located in Hursley Park, site of IBM’s largest non-US Development Laboratory 6 project / engagement rooms 3 conference rooms 2 Dedicated classrooms, + access to 4 more Hursley Innovation Theatre SmarterPlanet Lab, Retail Lab, E&U Lab, Local Govt. Lab, Sustainability Demos Technology and Servers System x, (incl. Nehalem) System i inc i520, i570, IBM POWER inc POWER6 570, POWER7 740/50/770/80, & POWER7+ 760/70/80 Storage inc SVC, SFS,Tape, SONAS, V7000 DS8800 & XIV Fibre connected Blades, & SmartAnalytics All Middleware engagements accommodated, including most SWG acquisitions (e.g. iLog, FileNet, SPSS, Cognos, etc.) PureSystems(PureApplication, PureFlex & PureData) Infrastructure High bandwidth infrastructure Private network separate from internal IBM network VPN Support to allow remote working Dual SAN Fabric to dedicated Storage Large number of load generation machines Soft switching KVM system to allow customers to work anywhere in centre Skills and People Skilled “development” knowledge and experience Wide range of technical expertise across SWG and STG Great networks inside IBM’s technical communities Technical events management and coordination 29 29

  30. Locations 42 Globally Networked Locations • North America (6) • Austin • Chicago • Dallas • San Mateo • Toronto • Waltham • Latin America (2) • Sao Paulo • Mexico City • CEEMEA (12) • Bratislava • Bucharest • Budapest • Casablanca • Istanbul • Johannesburg • Kiev • Ljubljana • Moscow • Prague • Warsaw • Nairobi Results • Build skills • Build solutions • Close deals and improve your bottom line • Execute marketing tactics Staff • Full-time IBM technical experts for hands-on assistance and knowledge transfer • Access to IBM marketing and sales teams Classrooms • Technical workshops and seminars • Business seminars and collaboration events Meeting rooms • Customer briefings • Sales closure meetings • Business Partner Demo rooms Technology support Access to the latest IBM hardware and software: • Onsite at the Centers • Remotely via secure Virtual Private Network (VPN) • Europe (13) • Copenhagen • Dublin • Stockholm • Hursley • London • Stuttgart • Zurich • Amsterdam • Barcelona • La Gaude • Tel Aviv • Milan • Paris • Japan (1) • Tokyo • Asia Pacific (8) • Bangalore • Kuala Lumpur • Manila • Seoul • Shanghai • Sydney • Vietnam • Ho Chi Minh 30 30

  31. Learn more about PowerLinux Power Systems Linux Portal (Product Information) www.ibm.com/systems/power/software/linux/ The PowerLinux Community (developerWorks) www.ibm.com/developerworks/group/tpl/ plus.google.com/communities/100156952249293416679 @thinkpowerlinux

  32. Sessions: #1: Exploiting Virtualisation on IBM Power Systems with PowerVM #2: VIOS - how to get going #3: Controlling processor resources in virtualised partitions #4: Deeper dive into Active Memory Sharing #5: Virtualisation Best Practices #6: Deeper dive into Shared Storage Pools and Thin Provisioning #7: Virtualisation and the world of 10Gbit Ethernet #8: Deeper dive into Active Memory Expansion #9: VIOS maintenance #10: Active Memory Deduplication #11: IBM i Suspend/Resume #12: Dual VIOS Upgrade walk-through #13: Shared Storage Pools ... from Experience #14: IBM i Live Partition Mobility #15: Capture and Deploy partitions #16: Virtual Partition Manager for IBM i #17&18: Updating Power Systems, I/O and HMC #19&20: Power7 Affinity and Performance #21: Power Advisors (VIOS, LPAR and Java) #22: PowerSC – What/Why/How #23: Dynamic Platform Optimizer (DPO) #24: Removable Disk Cartridge (RDX) on Removable Mass Storage (RMS) #25: Electronic Service Agent #26: Active and Dynamic Systems Optimizer (ASO/DSO) #27: Shared Storage Pools Phase 3 #28: Whole POWER Machine Monitoring #29: Monitoring POWER/AIX/VIO with IBM Tivoli Monitoring #30: Tricks of the Power Masters #31: External Storage and IBM i – sizing/modelling #32: Power Systems Configuration Best Practices @JyotiDodhia Power Systems Technical Webinar Series A series of technical webinars on Power Systems (including virtualisation): Informal Usually about an hour each Many include demos Aimed at: Technical audience - operators, systems administrators and technical specialists Those using / planning to use IBM's Power based systems Customers / Business Partners / IBMers UK & Ireland audience initially, but spread to other countries Join the 760+ who have registered already... More info, replays, and registration at: http://tinyurl.com/PowerSystemsTechnicalWebinars or send an email to jyoti_dodhia@uk.ibm.com 32

More Related