Too Deep & Too Wide

1. Too Deep & Too Wide Using Soft LimitsTo Avoid Hard Crashes NY/NJ Websphere MQ Family User GroupIBM 590 Madison Ave, New York CityTuesday March 25, 2003 T.Robert Wyatt, Vice PresidentBank Of America Differentiated ServiceWebsphere MQ Support NY/NJ MQ Family Users’ Group

2. Objectives • Demonstrate why the default queue depth and message length values pose a potential risk • Discuss one approach to mitigating that risk NY/NJ MQ Family Users’ Group

3. Scenario For Disaster Unrealistic requirements: “Need to be able to queue up an entire day’s worth of large XML messages” • Increase MAXDEPTH • Increase MAXMSGL NY/NJ MQ Family Users’ Group

4. Whoops! One day the channel goes down and the system performs exactly as designed: • Queues begin to fill up • Underlying filesystem fills • MQ begins to cut FDC files • MQ and possibly the OS are halted NY/NJ MQ Family Users’ Group

5. Recovery • Free up disk space any way possible • Move any diagnostic files • Delete non-essential files • Move older MQ logs if linear NY/NJ MQ Family Users’ Group

6. Recovery When (or if) QMgr becomes responsive again: • Stop all channels • RCDMQIMG if linear • Shut down the QMgr NY/NJ MQ Family Users’ Group

7. Recovery Restarting the QMgr: Delete files or add space • Clean up the file system • Remount the filesystem to larger disk and copy files Results in extended outage, high probability of client impact NY/NJ MQ Family Users’ Group

8. What Happened? 5,000 x 4mb messages = 20GB 5k x 4mb persistent msgs > 40GB Def DeadQ 640,000 x 4mb = 2.5TB! When the capacity specified by soft limits exceeds the physical limits, then the soft limits do not exist. NY/NJ MQ Family Users’ Group

9. Hard Limits Application constrained only by limits of physical disk space • Failure is sudden • Failure may be catastrophic • Affects OS & co-resident apps • Slow manual recovery • High client impact NY/NJ MQ Family Users’ Group

10. Soft Limits Application constrained by MQ • System degrades gracefully • Failure isolated to specific apps within MQ • Recovery is automated – normal operation of MQ restores service • Minimal client impact NY/NJ MQ Family Users’ Group

11. Tuning Discipline Insure soft limits < physical limits • Reduce MAXDEPTH, MAXMSGLor both • Reduce settings on SYSTEM.DEF*as well as application queues • Reduce DeadQ Settings! NY/NJ MQ Family Users’ Group

12. Tuning Discipline Disk Space Available* MAXDEPTH MAXMSGL DSA = MAXMSGL x MAXDEPTHMAXDEPTH = DSA / MAXMSGLMAXMSGL = DSA / MAXDEPTH*Use ½ DSA for persistent msgs NY/NJ MQ Family Users’ Group

13. Tuning Discipline Know your messages! • Soft limits must be appropriate to the business to be enforceable • Standardize, preferably automate • Verify on each machine • No exceptions Too shallow? Too narrow? Buy more disk! NY/NJ MQ Family Users’ Group

14. Apply System Wide System-wide tuning parameters • Channel MAXMSGL • QMgr MAXMSGL • Max uncommitted messages • Log file size and extents • Use separate, preferably dedicated, filesystems for /var/mqm/qmgrs and /var/mqm/logs NY/NJ MQ Family Users’ Group

15. Resources IBM Websphere MQ Manuals http://www-3.ibm.com/software/ts/mqseries/library/manualsa/ • Quick Beginnings Guide • System Administration Manual Support Pacs http://www-3.ibm.com/software/integration/support/supportpacs/ • Capacity planning pacs • Performance reports NY/NJ MQ Family Users’ Group

16. Resources MQSeries.net: http://www.mqseries.net Vienna List Server: http://www-3.ibm.com/software/ts/mqseries/support/usergroups/listserver.html Download presentation: http://www.t-rob.net/Downloads/2Deep2Wide.ppt T.Rob: T.Rob@t-rob.net NY/NJ MQ Family Users’ Group