100 likes | 338 Views
Single P oint Failure: The case study of RBS. CS/SE 6361 FINAL TERM PAPER. Shahed Shuman. What was the issue?. On the day of June 19 2012 The RBS payment processing system suddenly stopped , None of the overnight job running the Bank balance
E N D
Single Point Failure: The case study of RBS CS/SE 6361 FINAL TERM PAPER ShahedShuman
Whatwas the issue? • On the dayof June 19 2012 The RBS paymentprocessing system suddenlystopped, None of the overnight job running the Bank balance calculationhappend, all of the transactions processing were disrupted, ATM machines were not operational, online bank balance wasshowing incorrect balance! • The day before the Incident, an incorrect upgrade was Installed to the RBS CA-7 Software, CA-7 is a job scheduling / workflow automation software package sold by CA technologies. Resulting in total stoppage of Payment processing jobs. • RBS moved the CA7 maintenance and support to offshore .RBS alsocut down theironshore CA-7 resources As a result ,no CA-7 expert wasleft in RBS whowereexperiencedenough to diagnose the issue and revertthe corruptedupdates
Root cause • Proper CA-7 update requirements were not followed by the offshore team. • Systems upgrade happened on a weekday without proper testing, it caused major business disruption • The issue was not found until it was already in the RBS production server. • RBS didn’t have any requirement available to tackle these kinds of technical emergency • RBS did not have resources to implement any corrective measure • The delay to implement the fix and subsequent media coverage caused the issue to be longer as CA technologies took longer time to avoid another blunder. the error was made when backing out of an upgrade from CA-7 v11.1 to v11.3. The CA-7 upgrade took place at the weekend of 16/17th June and a problem was noticed on Monday which prompted a back-out from the upgrade on Tuesday night. In the back-out, an "inexperienced operator" made the wrong move and the day's data was wiped from the system. This created the backlog(http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_offshoring_software_glitch/)
Requirements Prioritization AHP Requirement Priority Normalized