Zinsight: Understanding Traces from System Z

Zinsight:Understanding Traces from System Z Steve Heisig, Wim De Pauw IBM T.J. Watson Research heisig@us.ibm.com wim@us.ibm.com WASL 2009

System Z OS Level Tracing: • Starting point: A problem has been detected by software or human analysts. • A System Dump contains a snapshot of the Trace Tables • ~2 Million, ~100 byte trace events per second • per cpu • Syscall level events • Optional Branch Trace • ~50 event types • Used for Problem Diagnosis and Performance Tuning • Why can’t I saturate the cpu? • Who is consuming the cpu?

Raw Trace Entries • --------------------------------------------------- SYSTEM TRACE TABLE ---------------------------------------------------- • PR ASID WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3 PSACLHS- PSALOCAL PASD SASD TIMESTAMP-LOCAL CP • UNIQUE-4 UNIQUE-5 UNIQUE-6 PSACLHSE DATE-10/26/2006 • 003-0038 007C58B0 SSAR ... 0038 0038 • 03-0038 007C58B0 SSAR ... 0038 0038 • 03-0038 007C58B0 PR ... 7 0FA3DACE 00_135BB110 0038 • 03-0038 007C58B0 PC ... 8 0479E3E1 02B00 • 03-0038 007C58B0 PC ... 2 01421F34 00343 IeaMQry • 03-0038 007C5250 PR ... 2 0479E3E0 7F5F29D8 0038 • 03-0038 007C5250 PC ... 2 0479E34A 02B01 • 03-0038 007C5250 PR ... 0 01421F34 012B8336 001E • 03-0038 007C5250 PR ... 2 0479E34A 7F5F29D8 0038 • 03-0038 007C5250 PR ... 2 11303529 1191F11A 0038 • 03-0038 007C5250 PC ... 8 11C473E1 02D08 • 03-0038 007C5250 PR ... 2 11C473E1 1191F11A 0038 • 03-0038 007C5250 EXT 1005 078D2400 988FDEAE 00001005 00000000 00000000 0038 0038 12:22:55.911696 14 • 00000000 • 03-0038 007C5250 SVC 18 078D2400 CB0B130C 00000000 4B5EB188 4B0B3FCA DevType 12:22:55.911757 14 • 03-0038 007C5250 SVC 3C 070C0000 854D512E 00000000 00000100 007FD84C Estae 12:22:55.911757 14 • 03-0038 007C5250 SVCR 3C 070C0000 854D512E 00000000 00000000 007FD84C 12:22:55.911759 14 • 03-0038 007C5250 SVC 3C 078C0000 854D5334 00000004 00000084 4B0B3FCA Estae 12:22:55.911759 14 • 03-0038 007C5250 SVCR 3C 078C0000 854D5334 00000000 00000000 4B0B3FCA 12:22:55.911760 14 • 03-0038 007C5250 SVCR 18 078D2400 CB0B130C 00000004 00000004 00000000 12:22:55.911760 14 • 03-0038 007C5250 PC ... 8 0 4B0B4B6D 00180205 • 03-0038 007C5250 SSAR ... 002C 002C • 03-0038 007C5250 SSAR ... 0038 0038 • 03-0038 007C5250 PR ... 7 4B0B4B6D 11FA1BC2 0038 • 03-0038 007C5250 PC ... 8 0 4B0B4B6D 00180205 • 03-0038 007C5250 SSAR ... 002C 002C • 03-0038 007C5250 SSAR ... 0038 0038 • 03-0038 007C5250 PC ... 7 0 0FA3DACE 0018D901 • 03-0038 007C5250 SSAR ... 002E 002E • 03-0038 007C5250 SSAR ... 0038 0038 • 03-0038 007C5250 SSAR ... 0038 0038 • 03-0038 007C5250 SSAR ... 0038 0038 • 03-0038 007C5250 PC ... 7 0479E34A 02B01

Graphical representation, navigation, searching and statistics

Zinsight extracts execution flows to reduce the amount of data for the user

Finding bottlenecks is easy

Multiprocessor analysis shows the impact of MP on your applications

Zinsight usage

Conclusion: Zinsight as a QA tool • Pro-active analysis of performance • Problem determination for difficult cases • Combination of visualization and pattern extraction to handle large complexities • Zinsight technology can be applied to different systems

Next Steps: • Register and overlay traces from other levels of the system • Millicode, Hypervisor, IO Assist processor, Hardware Accelerators, Component Traces • Larger Traces • Not capable of fitting into memory or being processed deterministically • Use Temporal Data Mining techniques, sampling to build up a statistical picture • Tweaking Hardware to make Tracing cheaper

Zinsight: Understanding Traces from System Z