Trouble Shooting

1. Trouble Shooting

2. What are System Logs for Blue Gene/P • A System Admin needs to refer to these logs for troubleshooting or failure analysis. • Components of BG/P control system will store all output into the BG/P system log directory, /bgsys/logs/BGP. • bgpmaster • mcserver • mmcs_db_server • navigator • mpirund • IO node logs

3. Naming of system logs except for IO node logs • Each system log except for IO node logs will be stored under /bgsys/logs/BGP whose name is based on the timestamp of its start time. A fine difference based on specification is explained below. • naming of past logs for bgpmaster and mcserver • <service node name>-<component name>-yyyy-mmdd-hh:mm:ss.log e.g. ibmsn-mcserver-2007-0406-11:31:52.log • naming convention of past logs for mmcs_db_server, navigator, and mpirund • <service node name>-bgdb0-<component name>-yyyy-mmdd-hh:mm:ss.log e.g. ibmsn-bgdb0-mmcs_db_server-2007-04-09-10:41:21.log, • Each ongoing log file is linked by a file whose name is *-current.log in the same directory. e.g. ibmsn-bgpmaster-current.log, ibmsn-bgdb0-navigator-current.log

4. Naming of IO node logs • Each IO node log will be stored under /bgsys/logs/BGP whose name is based on the location. • naming of logs for IO node • Rxx-My-Nzz-Jnn.log e.g. R00-M0-N00-J00.log • The following logs also exist in this directory. • diagnostics logs in /bgsys/logs/BGP/diags • installation logs • ex. VerifyCables-yyyy-mmdd-hh:mm:ss.log • InstallServiceAction-yyyy-mmdd-hh:mm:ss.log • service action logs • ex. ServiceNodeCard-Rnn-Mn-Nnn-yyyy-mmdd-hh:mm:ss.log • ServiceLinkCard-Rnn-Mn-L0n-yyyy-mmdd-hh:mm:ss.log • ServiceBulkPowerModule-Rnn-B-P0n-yyyy-mmdd-hh:mm:ss.log

5. Access to System Logs via command line • The system logs can be accessed from command line • only to go /bgsys/logs/BGP, and find the log of the time zone of the purpose target system component $ bgpadmin@bgpdd1sys1:~> cd /bgsys/logs/BGP/ $ bgpadmin@bgpdd1sys1:~> ls bgpdd1sys1-mcserver-2007-1010-14:41:46.log bgpdd1sys1-mmcs_db_server-2007-1010-14:11:51.log bgpdd1sys1-mpirund-2007-1010-14:11:51.log R10-M1-N08-J00.log bgpdd1sys1-navigator-2007-1010-13:45:20.log bgpdd1sys1-bgdb0-mmcs_db_server-current.log@ bgpdd1sys1-bgpmaster-2007-1010-12:11:51.log bgpdd1sys1-bgdb0-mpirund-current.log@ bgpdd1sys1-bgdb0-navigator-current.log@ bgpdd1sys1-mcserver-current.log@ bgpdd1sys1-bgpmaster-current.log@ bgpdd1sys1-bgpmaster-2007-1009-15:28:18.log

6. Access to System Logs via BG Navigator access to current system logs • The system logs can be also accessed from the BG Navigator • Click “System Logs” on the top page of BG Navigator • Click “Current”, or selected by default, on the System Logs home page, if you are looking for a current log • You can also chose the other tabs, for “All” or “Old” logs as following page • Click the log file name which you are looking for e.g. mmcs_db_server current log • Display the log file via Log Viewer on a new pop-up window 1 2 3 4

7. Access to System Logs via BG Navigator access to past system logs • Click “System Logs” on the top page of BG Navigator as above • Click “All” or “Old” tab, on the System Logs home page, if you are looking for a past log • If needed, expand the bullet of the filter option and enter the timestamp • Click the log file name which you are looking for • Display the log file via Log Viewer on a new pop-up window 1 2 3 4 5

8. Viewing logs via Log Viewer • Log Viewer has these features which assist you in the following areas: • Movement: paging backwards and forwards, moving to a specific timestamp • Refresh: refreshing your log manually or automatically, in viewing a current log • Filtering: selecting only the specific log file lines, can be applied to the entire log file or to the lines that are currently being viewed. • Highlighting: • auto highlighting, for example, error message in red character • manual highlighting particular word(s) • Hints: Hover your mouse over any of the log file viewer icons, and the viewer provides a hint that describes that icon’s function 4-1 3 2 1 4-2 5