1 / 7

Single P oint Failure: The case study of RBS

Single P oint Failure: The case study of RBS. CS/SE 6361 FINAL TERM PAPER. Shahed Shuman. What was the issue?. On the day of June 19 2012 The RBS payment processing system suddenly stopped , None of the overnight job running the Bank balance

kibo-morrison
Download Presentation

Single P oint Failure: The case study of RBS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Single Point Failure: The case study of RBS CS/SE 6361 FINAL TERM PAPER ShahedShuman

  2. Whatwas the issue? • On the dayof June 19 2012 The RBS paymentprocessing system suddenlystopped, None of the overnight job running the Bank balance calculationhappend, all of the transactions processing were disrupted, ATM machines were not operational, online bank balance wasshowing incorrect balance! • The day before the Incident, an incorrect upgrade was Installed to the RBS CA-7 Software, CA-7 is a job scheduling / workflow automation software package sold by CA technologies. Resulting in total stoppage of Payment processing jobs. • RBS moved the CA7 maintenance and support to offshore .RBS alsocut down theironshore CA-7 resources As a result ,no CA-7 expert wasleft in RBS whowereexperiencedenough to diagnose the issue and revertthe corruptedupdates

  3. Root cause • Proper CA-7 update requirements were not followed by the offshore team. • Systems upgrade happened on a weekday without proper testing, it caused major business disruption • The issue was not found until it was already in the RBS production server. • RBS didn’t have any requirement available to tackle these kinds of technical emergency • RBS did not have resources to implement any corrective measure • The delay to implement the fix and subsequent media coverage caused the issue to be longer as CA technologies took longer time to avoid another blunder. the error was made when backing out of an upgrade from CA-7 v11.1 to v11.3. The CA-7 upgrade took place at the weekend of 16/17th June and a problem was noticed on Monday which prompted a back-out from the upgrade on Tuesday night. In the back-out, an "inexperienced operator" made the wrong move and the day's data was wiped from the system. This created the backlog(http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_offshoring_software_glitch/)

  4. ProblemInterdependency Graph

  5. Solutions –

  6. Requirements Prioritization AHP Requirement Priority Normalized

  7. Questions?

More Related