1 / 28

Maintaining Large Vista Installations

Maintaining Large Vista Installations. Amy Edwards, Ezra Freelove, & George Hernandez July 12, 2007. Agenda. Comparisons Who is USG Automation Monitoring Maintenance More Tricks Questions?. (All prod clusters) now: 1-10 11-20 21-50 50-70 70+ Ours in bold.

nusa
Download Presentation

Maintaining Large Vista Installations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Maintaining Large Vista Installations Amy Edwards, Ezra Freelove, & George Hernandez July 12, 2007

  2. Agenda • Comparisons • Who is USG • Automation • Monitoring • Maintenance • More Tricks • Questions?

  3. (All prod clusters) now: 1-10 11-20 21-50 50-70 70+ Ours in bold (All prod clusters) by December: 1-10 11-20 21-50 50-70 70+ Informal Poll - Number of nodes

  4. Informal Poll – Number of DB Instances Including secondary and non-production • 1-2 • 3-6 • 7-10 • 10+ • Ours in bold

  5. Vista Architecture

  6. GeorgiaVIEW Project • University System of Georgia (USG) • Vista 3.0.7 • Host 32 institutions & multiple consortial programs • >150,000 active students • Active is 100+ actions • >11,000 active sections / term

  7. Issues • Handling performance issues • Capacity planning • Upgrades • Replication • JMS sensitivity • Integration

  8. Automation • Rolling Restarts • Managed nodes restarted weekly • except JMS • Log cleanup to preserve space • Error reporting • application, tracking, vulnerabilities • Thread dumps • Sync admin node with backup • LDIS batch integration

  9. Monitoring • Nagios • http://www.nagios.org/ • Sends alerts • Stats • Custom AJAX web app • Watch changes of over time • AWStats • http://www.awstats.org/

  10. Nagios Example

  11. OS / Hardware Load Temperature Free space Database Tablespace free space Listener Oracle processes Application Direct-login Weblogic processes Java MBeans Default/Primary Pending Requests Current Count Java Heap Current JDBC Waiting for Connection Current Count Multicast Messages Lost Primary count Nagios Monitors

  12. Stats • Short and long term analysis • 21 months of data • Graphs all Nagios data collected • Flexible creation of reports • Built with AJAX

  13. Stats Examples I of III

  14. Stats Examples II of III

  15. Stats Examples I of III

  16. AWStats • Records data from web server logs • Custom script grabs data from webserver.log files • Runs daily

  17. AWStats Examples I of II

  18. AWStats Eamples II of II

  19. Specialized Nodes • Admin • JMS • Institutional Admin • Integration • Chat

  20. JMS Node • Provides special services • Mail, LC creation, chat • Failure or migration of JMS node hinders usage • Services do not migrate well • Allow targeted migration • OTHERS: Pin JMS to a specific node

  21. Integration • Batched LDIS data files • Cron runs nightly • Files broken up by: • type • “reasonable” number of records • Done on Inst node • Issues with import can kill node

  22. Touching Nodes • ssh & dsh • Touch groups of nodes at once • Useful for: • Installs • Gathering logs • Locating a session

  23. Maintenance Page • Hosted on opposite f5 • Two versions • Scheduled maintenance • Unscheduled outage • In an f5 outage, move DNS to other f5 so message still appears

  24. Installs and Upgrades • Silent install scripts • Test in both development environments • Create against a small database • Get results of time to complete against a full size copy of production • Install to production

  25. Powerlinks and Custom Development • Test in development • Try to break • Pilot in production • Release to all

  26. Questions?

  27. Want More? • To view my resources and references for this presentation, visit www.scholar.com • Simply click “Advanced Search” and search by ezrafreelove and tag: ‘bbworld07’

  28. Contact Information • Ezra Freelove ezra.freelove@usg.edu • Amy Edwardsamy.edwards@usg.edu • George Hernandezgeorge.hernandez@usg.edu

More Related