Clues and the Scientific Method for System Changes

1. Clues and the Scientific Method for System Changes Change Management and Downtime Procedures

2. Session Description • During this session we will deduce ways to manage system changes • Methods for different change types will be explored • Documentation of a Change Request and the anatomy of Change Management authorizations • Tips and tricks to execute downtimes

3. Two main expectations of the services provided by IT • The services should be stable, reliable, and predictable • The services should be able to change rapidly to meet evolving business requirements • These expectations are in conflict. The objective of change management is to enable IT service management to meet both expectations—to enable rapid change while minimizing the possibility of disruption to services.

4. Change Management • Implement a set of standard processes for managing changes • Reduce time to implement change without loss of service • Expedite decisions with built-in approval process • Increase visibility and communication of changes to both the business & service support staff

5. Information Technology Infrastructure Library (ITIL) • Provides a set of best practices for change management • Makes it easier for IT professionals to roll out and prioritize changes efficiently, without negatively impacting customers • Process designed to understand and minimize risks while making IT changes • ITIL defines the best practices that IT organizations use to deliver value to customers via the concept of “services.” • Companies who use ITIL are able to standardize the way they plan, deliver, and support IT services to their internal or external customers.

6. Change Management Team • The change initiator • recognizes and identifies the need for change • The change coordinator • assesses requests for change that originate from incident management, problem management, release management, or optimization • registers changes as needed to handle requests for change or receives change requests from other change initiators; assigns to implementers • determines the risk and impact for requested changes; prepares implementation plans by creating tasks; and monitors the progress of changes.

7. Change Management Team • The change manager • responsible for managing change procedures, receiving and prioritizing change requests, evaluating the risk level associated with requests, and keeping thorough records of the outcome of each change • The approver • decides whether to approve or reject changes • The change advisory board • responsible for authorizing changes and further evaluating requests when the change manager determines that there is a high risk associated with these requests • The board takes into account the impact that a requested change may have on all affected parties • The change implementation team • the specialists on your team who are responsible for actually making changes

8. Types of IT changes • There are different types of change requests that are typically managed in different ways: • Standard changes • changes to a service where the implementation process and the risks are known upfront • changes are managed according to procedures that are already in place • don’t require approval from a risk management perspective • Normal changes • must go through the change process before being approved and implemented • If they are determined to be high-risk, a change advisory board must decide whether they will be implemented • Emergency changes/ Urgent • arise when an unexpected error or threat occurs • needs to be addressed immediately • A security threat is another example of an emergency situation that requires changes to be made immediately

9. Standard Changes • Applies to changes involving little risk • Standard changes are pre-approved, meaning that they do not have to be reviewed by change management • Has accepted and established processes/procedures to provide a specific change requirement. List procedure number in the Change Request • Used to deliver services published in the Service Catalogue • Activities performed to support and maintain services, the infrastructure, or to enable a project to deploy • May require peer review

10. Standard Service Catalog

11. Standard Change Procedures

12. Standard Change Log

13. Request for Routine Change • If you are creating a request for change, you are responsible for documenting details that will help others understand what change needs to be implemented and why you are making the request. The initial change request submission often includes details about the risk and implementation steps, • Details that may be found in a change request include: • Incidents that necessitate the change • Description of how the change would be implemented • The impact that the change would have on all associated systems • A risk assessment • Contact information for everyone involved in the change • An outline of who will need to approve the request • A backup plan to follow in case the change is not successful

14. Contents of a Change Requests • Build documentation • Detailed work instructions and step-by-step procedures for the implementer to follow when completing the change. • This enables the reviewers and the Change Manager the ability to determine the procedure's risk and the need to request additional steps be added to mitigate the risk (e.g. backup before starting the change). • Validation documentation

15. Example Ticketing System

17. Planning the Change • A change plan outlines the course that the change will take, the resources that are needed to complete the change, and a timeline for implementation.

20. Approving Change • A request for a new Standard change is approved through the Standard Change Management process. • Request for Change (RFC) to approve the Standard Operating Procedure (SOP) to establish a Routine change. • After it is approved, every time it is used, an RFC should be logged and reference the approved procedure • A request to approve a Routine change is submitted to the approver • Advice from the Change Advisory Board (CAB) can be asked if required

22. Security Access to Production

23. Implementing Changes • Once the change has been made, tests must be done to determine whether the desired results have been achieved • If the change is not successful, remediation methods may be used to determine what went wrong and if needed implement the backout

24. Downtime Labels • Determine if you will use downtime labels • Prepare enough for length of downtime • Procedure to create downtime labels • Procedure to use downtime labels

27. Recovery

28. Downtime Epic Release • Service Request for early release of EPIC lab orders for sweeps • IT analyst • Verify EPIC Orders were released by 11PM • Procedure is for EPIC analyst to page

29. Downtime • Reports • Do not schedule any reports for kick off during downtime • AR jobs reschedule • Move its

30. Downtime • Microbiology Downtime Server • Server and report printer which capture prelims every 12 hours. • Converted to PDF files • Can be printed on demand • Verify last file back up was successful

31. Downtime • Update Test Patients with new Billing number • Send Lab Team reminder to review procedure for testing on test patients in Production • ward/physician use • minimize leak to a real client or report physician JoAnn- I went ahead this morning and updated the visits in SOFT so that they route to our dummy wards ROCG, TRCG, GPCG.  They’re all ready for tomorrow.  Visits are: 99999272163 99999262062 99999282056

32. Downtime • IT analyst • Reminder call to Service desk • Update Hot Line message • Assume On Call pager • Create communication forum for use during install between client and SCC • Create a Web Ex or Lync as back up if needed • Send to participants • Email between point persons (IT installation analyst and SCC installation analyst) • Send as placeholder • Open Line of Communication between Client and SCC at 20 mins prior to installation • Record milestones, issues • Respond to any additional permissions as needed

33. Post Downtime • Page testers to begin General Functionality testing • Call key lab areas such as Stat Labs to verify no immediate concerns • Page Lab Leader pager group with update • Open Skype for Testers – start recording • Testers note progress • Testers note issues and TMS numbers • Email with SCC still open • Provide updates to SCC implementer on progress • Report issues and TMS numbers • Work to resolve all open issues

34. Downtime Issues • System not up as expected • Make immediate assessment and decision regarding notifications and incident command execution • System up with issues • Determine impact of issue on end users • Work with SCC to resolve • Document TMS numbers, Issue and Resolution steps • Include time started and resolved

36. Create Hot Line Message • The IT Systems Downtime Hotline Message is a recorded message that resides at the following phone number 248-551-XXXX. • This message is used to communicate statuses/updates when IT Systems are unavailable or are functioning in less than optimal manner and it is necessary to communicate statuses to many users. This service is used for Scheduled and Unscheduled Downtimes

37. Open a Customer Communication • This is an opportunity to listen to the customer/user to obtain additional information about the incident request to help the investigation, as well as provide confidence to the user that their incident issue has been received, logged, has an owner and is actively being worked to restore service as quickly as possible. • Start a Microsoft Skype session for incident customer/user communication, if needed for more effective user communication than sending a routine incident notification message.  • Use the pre-established number and Conference ID for an incident customer/user update MS-Lync session.  Use this direct link to

39. Downtime Completed • Close Lync Session with Testers • Store recorded file on Change Management Entry • Send out all clear page to lab leadership • Return pager assignment to on-call person • Return SOFT ID roles back to look ahead 3 hours • Close Change Management • Attach session recordings/summaries • Close email with SCC • Compose and send a Summary of the Load and send to IT and Lab Leadership

40. Reviewing Change Performance • post-implementation review • understand whether your change procedures are working as expected • reviewing records to determine whether the change was successful or failed, and recording details about the time to determine the accuracy of estimates that were made before a request was fulfilled • Reviewing change performance gives you the opportunity to fine-tune your change management process for better results in the future

41. Closing the Process • Once the change process is complete, you must be sure that the entire process has been documented in a database that all stakeholders can access

42. Completed Change Requests

43. Working Request thru to Change

44. Slice and Dice Management Tips