A System Administrator’s Guide to Preliminary Troubleshooting Techniques

1. A System Administrator’s Guide to Preliminary Troubleshooting Techniques Presented By: Darryl Wright and Volker Kleinschmidt April 12, 2005

2. And now a word from our lawyers… • Any statements in this presentation about future expectations, plans and prospects for Blackboard and other statements containing the words "believes," "anticipates," "plans," "expects," "will," and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including the factors discussed in the "Risk Factors" section of our most recent 10-K filed with the SEC. In addition, the forward-looking statements included in this press release represent the Company's views as of April 11, 2005. The Company anticipates that subsequent events and developments will cause the Company's views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company's views as of any date subsequent to April 11, 2005. Blackboard, in its sole discretion, may delay or cancel the release of any product or functionality described in this presentation.

3. Objectives • Provide Blackboard® System Administrators with an overview of their roles, responsibilities, and their impact on the success of the Blackboard Academic Suite™ at their institutions • Highlight best practices for keeping your Blackboard Academic Suite running smoothly and problem free • Improve and standardize the troubleshooting process on campus. Methods for better troubleshooting. • Describe ticket submission and escalation process within Blackboard • Open Discussion: The evolution of your Blackboard Networked Learning Environment™

4. System Administrator Roles I • Back-end administrator Installations; webserver admin; database configuration and maintenance; application monitoring; snapshot and data integration; and even OS maintenance tasks • Front-end GUI administrator Course shell creation; user enrollment; export/import/copy; control over feature availability and design; set policies and business rules for account creation, content preservation etc.; portal administration • System Extension Developer Creation of Building Blocks, Application Bridges and Portal Modules using Java and Blackboard’s APIs

5. System Administrator Roles II • CMS Help Desk Manager Troubleshooting issues reported via email, phone, or a ticket system, or self-discovered; escalation of issues to Blackboard Technical Support; training and supervision of general help desk staff • Trainer & Documentation Author One-on-one training and seminars for faculty, staff, and students; creating online documentation and tip sheets • Instructional Designer In some cases the system administrator may also train instructors and staff on pedagogically sound uses of technology, and participate in the creation of course materials and the tracking and evaluation of student performance

6. Which roles do you fill at your institution? Back-end administrator Front-end administrator System Extension Developer CMS Helpdesk manager Trainer / Documentation Author Instructional Designer Show of hands

7. Back-end administrator Front-end administrator System Extension Developer CMS Helpdesk manager Trainer / Documentation Author Instructional Designer BBADMIN-L BLKBRD-L BB-OPEN_SRC BB-FS BB-ID User community mailing lists

8. Evolution of System Administration Roles & Practices INSTITUTIONAL GROWTH EXPLORATORY SUPPORTED STRATEGIC MISSION CRITICAL TRANSFORMATIVE:THE NETWORKED LEARNING ENVIRONMENT Business Continuity Strategic Owner High Performance / High Availability System Update Deployment Plans Identified system manager Student /Faculty Training Programs Help Desk Management Integrated Production Environment Monitoring, Backup & Maintenance Practices Long Range Planning Change management practices Multiple People Custom Instructional Tools Single Server Single Admin Ticketing System Help Desk Phase 5 Phase 2 Phase 3 Phase 4 Phase I

9. Best Practices • Be Prepared • Avoiding is better than solving • Have a well-established troubleshooting process on campus • Train your users to be aware of pitfalls • Document local policies and differences • Top 10 tips for System Administrators

10. Best practices - Top 10 tips for System Administrators • Maintain current Operations Workbook • Document all server and application changes • Maintain a Test server environment • Set up and test a Disaster Recovery Plan • Plan ahead for increased usage • Train users well to avoid problems • Keep it clean - purge old records • Manage and monitor your application logs • Graph system metrics (MRTG, cricket) • Participate in user community

11. Network diagram(workbook) • Use Visio templates or simple flowcharting tool • Make sure to label machines with function, hostname and IP-address • Indicate ports used on each connection • Include servers, firewalls, routers, proxies, load-balancers • Keep it up-to-date

12. Disaster Recovery Plan • Ready adequate replacement hardware • Set up full daily backups to it • Test recovery regularly • Train multiple staff on recovery • Document all necessary steps in standard location (Workbook, Backup/Recovery guide, local customizations) • Don’t commit replacement system to other production-grade tasks

13. Sample performance graphs CPU-busy statsweekly graph dual weekly graph for total number of logins per dayand number of recent logins(last 10 mins)

14. Part II Troubleshooting

15. Troubleshooting Process on Campus The SOAP Method - A Basic Troubleshooting Method Subjective- Who?, What?, Where? and When? Objective - Steps to recreate, Error messages, log collection and access information Assessment - Review steps, errors, logs and assess whether it can be resolved or has to be escalated Plan- Resolution steps or Workaround Examples of SOAP interaction: - Used by the Helpdesk- troubleshooting end-user issues - Used by the System Administrator - recognizing, analyzing, reporting system-issues, focused collection of info for escalation

16. SOAP at the Helpdesk Troubleshooting User-Reported Issues • User- or course-specific issue? • if so, need exact steps and location • Browser-specific? • test on different machines and browsers • Recent changes? • OS or software update, configuration change, malware infection • Restore known-good configuration • system restore, spyware removal, virus scan

17. Example:Resolving a login problem • Account exists? • Correct password used? Caps? • If campus-wide pwd, does it work elsewhere? • Correct URL used? • Machine-specific? • Browser configuration (SSL, JS, cache) • Spyware and virus check • Create account, reset password, clean machine

18. SOAP for the GUI admin • Word-of-mouth vs. direct report • Issue affects multiple users? • Issue repeatable? • Issue known in user community? • Useful info in logs? • Report results of this investigation when filing support case

19. Example:Disappearing forum posts • Group forum or course forum? • Who has forum admin rights? • Can authors delete own posts? • whodunnit: check modperl access logs for POST requests in timeframe in question • Inform users that forums are not personal mailboxes and read messages should not be deleted • Restore backup to test machine and send text of lost messages to users for re-posting

20. SOAP for the back-end admin • Intermittent problem vs. event driven • Issue affects server performance? • Issue repeatable? • Issue tracking in the system logs? • Useful info in server OS logs? • Report results of this investigation when filing Blackboard support case.

21. Example:Database Connection Unavailable • Blackboard application not working • Users reporting error messages upon connection • Problem is not browser specific. • Review of Log files − Server OS, DB Logs and Blackboard log files • Log results • Blackboard log files revealed ‘full transaction logs’. • Server OS logs reported low drive space on DB server. • Database Maintenance Plans • System Administrator scheduled Database maintenance routines, Data integrity checks, Backup and truncation of logs. Updated Blackboard maintenance routines to include nightly and weekly DB backups

22. The Escalation Process within Blackboard Client Support • Clarifying the process – towards realistic user expectations -sample case lifecycle -sample case interaction between customer and Blackboard support • How and where can you as the system administrator help to resolve cases quickly and smoothly? - SOAP

23. Sample case lifecycle • Customer: case creation on Behind the Blackboard® • TSM: replicates problem • TSM: researches KB, Wiki, tracking system for similar cases and known resolution • TSM: obtains access info from customer • TSM: creates bug report and escalates case • PSE: verifies problem and attempts resolution • PSE: escalates to Development • Dev, PSE: Analysis of customer impact; prioritization • Dev: Developer assigned; target release version determined • Dev: Issue is replicated; root cause is determined • PSE: communicates between customer and development • Dev: solution is created • Dev: fix goes to QA • Customer: Beta-release testing • Dev: Software updated released

24. Roadblocks in the lifecycleor: why do things take so long? • Communication failures • Everyone is too busy • Incorrect or incomplete information • Customer impact not clear • Issue is not high priority • Other issues hold up a release

25. Sample case interaction • C: Our system keeps going down • S: Symptoms please? Logs? Access? • C: Supplies some logs & access info • S: Need recent log and perf-report • S: Access not working, SSH blocked • C: now SSH works • S: escalating • S: perf-report useless, no DB info in it • … you get the picture…

26. Or how it should be… • C: occasionally modperl crashes with CPU exhaustion • C: logs and perf-report from latest crash, access info • S: this has been reported by other customers – have you recently updated any building blocks? • C: yes, block XYZ • S: any courses with course menu links to this BBB? If so, export sample such course, restore to test server, and see whether accessing that course makes modperl CPU usage rise over all bounds • C: verified, here’s sample course export • S: workaround: remove bad menu items in DB

27. How can YOU help? • Clear problem description • Precise steps for replication • Preliminary analysis (SOAP) • Supply requested data promptly • Check case status online • Focus on important issues • We are all only human

28. Open Discussion The Evolution of theNetworked Learning Environment(John Fontaine, Jeff Kelley, John Knight) • Managing the growth of your installation of Blackboard Academic Suite • Scaling to the Blackboard Enterprise Application • Strategy and planning for future growth • Blackboard Consulting’s role in evolution of the NLE • Consider Blackboard ASP Services

29. Evolution of System Administration Roles & Practices INSTITUTIONAL GROWTH EXPLORATORY SUPPORTED STRATEGIC MISSION CRITICAL TRANSFORMATIVE:THE NETWORKED LEARNING ENVIRONMENT Business Continuity Strategic Owner High Performance / High Availability System Update Deployment Plans Identified system manager Student /Faculty Training Programs Help Desk Management Integrated Production Environment Monitoring, Backup & Maintenance Practices Long Range Planning Change management practices Multiple People Custom Instructional Tools Single Server Single Admin Ticketing System Help Desk Phase 5 Phase 2 Phase 3 Phase 4 Phase I

30. Wrapping up… • Innovating Together in ‘05: • Use SOAP method for troubleshooting • Preparation is better than problem solving • Resources Available: • Operations Workbook http://behind.blackboard.com/b3/sysadminls/refcenter/r61/wkbk.htm • Training Center http://behind.blackboard.com/b3/sysadminls/training/index.htm − Web Forums http://behind.blackboard.com/b3/sysadminls/forums/index.htm • Follow up Contact(s): − Support site http://behind.blackboard.com − Product Support - Domestic 1-800-788-5264, International 1-202-715-6019 − dwright@blackboard.com − vkleinschmidt@blackboard.com • IF YOU ONLY REMEMBER 1 THING: • Know your system well!

31. Thank You!