1 / 45

Blackbird: Accelerated Course Archives Using Condor with Blackboard

Blackbird: Accelerated Course Archives Using Condor with Blackboard. Sam Hoover, IT Systems Architect Matt Garrett, System Administrator. Blackboard @ Clemson. @. Blackboard @ Clemson. End of Semester archives of all online courses in Blackboard since implementation in 2004

lark
Download Presentation

Blackbird: Accelerated Course Archives Using Condor with Blackboard

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Blackbird: Accelerated Course Archives Using Condor with Blackboard Sam Hoover, IT Systems Architect Matt Garrett, System Administrator

  2. Blackboard @ Clemson @

  3. Blackboard @ Clemson • End of Semester archives of all online courses in Blackboard since implementation in 2004 • 77 GB Oracle 10.2.0.4 DB tied to a 1.3 TB Content system with over 13 million files • Spring 2010: 4610 active Blackboard courses, 31,372 total courses in Blackboard • Full system backups once a week, nightly incremental backups of entire system

  4. The Archive Problem • Blackboard is a mission critical system • Why is 85.5 hours for archives a problem? • Start of new semester vs. normal operations • Time between semesters is short and getting shorter • Faculty have to wait to set up next semester’s courses • End of semester processes

  5. Why do we need course archives?

  6. Student Add / Drop at start of semester

  7. Loss of course content or an entire course

  8. CRLT archive uses • Grade disputes

  9. Blackboard EoS archives

  10. The Archive Problem

  11. The Archive Problem • Blackboard provides a script for executing batch archives given a list of courses as input. • Weekly archive process at Clemson began in Fall 2006 after an accidental deletion of many courses. • Started out splitting the course list into four equal chunks and giving each server ¼ of the total course list. All four servers usually finished within 2 hours of each other, total time for the batch was < 24 hours. • By Fall 2008, archiving the active courses took 85.5 hours, and the servers finished at widely varying times.

  12. The Archive Problem

  13. The Archive Problem • Who wants to work weekends?

  14. Blackboard archive script • /usr/local/blackboard/apps/content-exchange/bin/batch_ImportExport.sh • Archive/Restore: The Archive Course function creates a record of the Course including User interactions. It is most useful for recalling Student performance or interactions at later time. The archive package is saved as a .ZIP file that can be restored to the Blackboard system at another time. In effect, Archive/Restore acts as a backup tool at the individual course level.

  15. The Archive Problem

  16. The Archive Problem Potential Solutions?

  17. Throw money at the problem?

  18. Add more servers?

  19. The Archive Problem

  20. Potential solutions • Write our own job scheduler? • Could we take advantage of the other 3 (CPUs)? • How do we monitor performance so end user (Blackboard) experience isn’t impacted? • Use a DB to store and manage the queue? • What about security? • Has anyone else out there already done this?

  21. Project Blackbird +

  22. Condor to the rescue? • Job scheduler? Check • Multi-core capable? Check • Manage the queue? Check • Performance monitoring? Check • Security? Check • Has anyone done this before? No

  23. Steps in the weekly archive process • Determine what to archive (active courses, orgs) • Build a course list • Create Blackbird submit files • Submit DAGMan job to Condor • Monitor Condor queue • Receive email notification when all courses have been archived • Look for errors and verify archive integrity

  24. Custom Condor Configuration • DAGMAN_MAX_JOBS_IDLE = 25 • DAGMAN_MAX_JOBS_SUBMITTED = 50 • SLOTS_CONNECTED_TO_CONSOLE = 0 • SLOTS_CONNECTED_TO_KEYBOARD = 0 • ## Force Condor to use Blackboard Private Network • NETWORK_INTERFACE = Private Blackboard Net

  25. DAGMan example JOB UniqueCourseID /path/to/condor/submit/job/file/UniqueCourseID.bbCondor JOB UniqueCourseID2 /path/to/condor/submit/job/file/UniqueCourseID2.bbCondor JOB UniqueCourseID3 /path/to/condor/submit/job/file/UniqueCourseID3.bbCondor JOB UniqueCourseID4 /path/to/condor/submit/job/file/UniqueCourseID4.bbCondor JOB UniqueCourseID5 /path/to/condor/submit/job/file/UniqueCourseID5.bbCondor JOB UniqueCourseID6 /path/to/condor/submit/job/file/UniqueCourseID6.bbCondor SCRIPT POST UniqueCourseID6 /usr/local/CMSIntegration/bin/weeklyArchiveChecker.pl

  26. Condor Submit example universe = vanilla requirements = (OpSys=="LINUX") && ((Arch=="INTEL") || (Arch=="X86_64")) executable = /usr/local/bin/condorSubmitArchive.pl arguments = shoover-S0000BKBRD_401001,/san/weeklyArchives/20091008/ getenv = True log = /usr/local/logs/bbCondorLogs/archive20091008.log notification = Error notify_user = DCIT2803_BB_ON_CALL-L@clemson.edu transfer_executable = False when_to_transfer_output = ON_EXIT queue 1

  27. Blackbird archive solution

  28. The Archive Problem

  29. Blackbird archive solution

  30. Blackbird Benefits • Reduced total archive time from > 85 hrs to < 24 hrs • Job scheduling – servers finish about the same time • Zero impact to Blackboard Performance • Automatic suspension/resumption of archives if Load reaches threshold on any core • Email notification upon completion of all archives • Load balancing – archive jobs are distributed as cores become available • Takes advantage of all available CPU cores instead of just one core per server

  31. Project Blackbird +

  32. Blackbird Benefits

  33. Blackbird Benefits

  34. What did it take to implement? • Have one or more multi-core (CPU) machines • A large amount of shared storage for archives • Choose one machine as your Central Manager • Install and configure Condor on each machine • Automate course list creation (Query DB or Directory) • Automate Condor submit files and Condor DAGMan file creation • Automate the whole thing with cron • Check log files for errors upon archive completion

  35. Where else could I use this? • Any system that does batch processing that can be broken up into many jobs • Recently implemented on our MySQL server to export all of the MySQL databases • Reduced the export time from 10 hours to 3.5 hours on a single, quad core machine

  36. Recent updates • 64 Bit Red Hat 5.4 OS and JVM 1.6 • Maximum (affordable) RAM per machine – 32 GB • Web page to view Blackbird Condor Pool status • Duplicate archives • Error checking logs • Redo any courses with errors or not completed • Major Blackboard upgrade from 7.3 to 9.1 end of June

  37. What’s next? • New machines have 2 x Quad Core CPUs with HyperThreading so Condor sees 16 Cores • Add out of warranty machines to the Blackboard Condor Pool (keep users off of them) • Monitoring of queue (web page) • Use ClassAds to specify architecture and memory requirements for large archive jobs • Write code to query DB and find out what courses have changed, backup any course that has changed on a daily basis • Automate installation and configuration

  38. Please provide feedback for this session by emailing DevConFeedback@blackboard.com.  The subject of the email should be title of this session: [Blackbird: Accelerated Course Archives Using Condor with Blackboard]

More Related