Raoul Blignaut IT Head - Citigroup South Africa

1. CMG Australia 2005 Understanding the evaluation of IT risks in the context of the broader social influences of such evaluations Raoul Blignaut IT Head - Citigroup South Africa Public

2. South Africa IT Cluster Head Risk Management Desktop / Network / OPS / Projects Team Building Public

3. Abstract / Overview Post WorldCom, ENRON and a raft of other spectacular business failures, the business world is rapidly moving towards a risk-based approach to management with the advent of the likes of SOX and BASIL2. Although this started with the financial and accounting areas of large corporations this has rapidly encompassed the entire organization, the Information Technology division not withstanding. This presentation looks at understanding the fundamentals of IT evaluation (in the context of evaluating IT risks) as it is the individual human actors and social groups who are completing the regular assessment of the risks, that can be the weakest (or strongest) links in the evaluation process. The worlds best IT controls, checks, risk indicators and monitoring is only as good as the individual or group who does the final analysis and assessment (and thus judgment) of the risk area under review. This paper provides and a view some of the weakness identified in the assessment of risk in the realm of IT and proposes some proactive approaches to mitigating these within your organization. Public

4. Information Systems Evaluation • Evaluation • is the process of coming to an understanding about value • Evaluation & measurement • evaluation relies on measuring various attributes of the object being valued • The social nature of evaluation • although there are components of evaluation that are more formal the process itself (i.e. coming to an understanding) is in fact an informal social & political process What does this have to do with IT Risk Assessment? Public

5. IT Risk Assessment • This presentation proposes that any Risk Assessment process, being an assessment (estimate the size or quality of)*is in fact an evaluation (assess, appraise)*and thus if viewed in the narrower context will be dysfunctional * Oxford dictionary • Information systems (IS) evaluation is typically aimed at identification and quantification of metrics & costs (Symons, 1990) • This narrower view is claimed to be dysfunctional(Walsham, Farbey, et al, 1996) • This presentation explores the broader sociological, psychological and political dynamics of assessing IT risks within the context of Citigroup, IT South Africa • It was found that both formal and informal approaches need to be drawn together to provide a balanced multifaceted view • understanding the context, processes and the content provides a mechanism for interpreting in-context IT / IS assessments Public

6. Context Process Content Attribute Metric L M 1 1 Measurement procedure L M 2 2 Measurement Information Systems Evaluation Public

7. Assumptions about the nature of evaluation as social science Subjective Objective Ontology Nominalism Realism Epistemology Anti-positivism Positivism Human Nature Voluntarism Determinism leads to Methodology Ideographic Scientific A continuum on which one can understand the various dynamics that influence evaluation Public

8. Assumptions about the nature of evaluation within society Order (sociology of regulation) Conflict (sociology of radical change) Stability Integration Functional co-ordination Consensus Social order Consensus Solidarity Change Conflict Disintegration Coercion Structural conflict Modes of domination Emancipation A continuum on which one can understand the social forces that influence your evaluation Public

9. Subjective Objective Two dimensions, four paradigms Conflict (sociology of radical change) Order (sociology of regulation) Public

10. Subjective Objective Two dimensions, four paradigms Conflict (sociology of radical change) Dialogic (post-modern) Critical (late-modern) Interpretive (pre-modern) Normative (Modern) Order (sociology of regulation) Public

11. Example: risk review is objective! Determinism Controls and Compliance review the responses for: ·         timeliness (all responses are received on or before the due date) ·         completeness (all items on the risk assessment matrix must have an entry) ·         validity (all entries relate to the specified control objective) ·         relevance (all responses are relevant to the business unit, and also to the time period under review) ·         accuracy (all responses are consistent with other related responses). ·         reliability (all responses provide adequate evidence of the testing undertaken and the results of the testing). Where IT Controls and Compliance identify and errors, omissions and/or inconsistencies within the responses and/or where they determine that the responses do not provide sufficient evidence… Realism Positivism Public

12. … example wording used to enforce domination and control … Example : Operational Risk & Risk Assessment Definitions • Sarbane Oxely – The regulation that say we must be able to evidence that, as a bank, we are in control of the processes that impact the books or the bank and any financial reporting. THIS is for YOU. Management go to jail when this is not under control – not line operations! • Risk Assessment – This is the entire controls environment we have put in place to ensure that you do NOT go to jail. It includes everything form the IT Process, global policies and standards to tracking to the quarterly risk management assessment. … attempting to enforcing social order and consensus Public

13. Subjective Objective Shifting Paradigms! Conflict (sociology of radical change) Dialogic (post-modern) Critical (late-modern) Personal / ‘Actual’ (informal) Normative (Modern) Interpretive (pre-modern) Organisational View (formal) Order (sociology of regulation) Public

14. Two dimensions, four paradigms Social Discourse Interpretive Critical Dialogic Normative Basic Goal Hope Social Metaphor Org. metaphor Narrative Style Problems Addressed Org Benefits Mood Social Fear Law-like relationship between objects Progressive emancipation Economic Marketplace Scientific / technical Inefficiency / disorder Control Optimistic Disorder Unified culture Values Social Community Romantic Meaninglessness Quality work life Friendly Depersonalisation Unmask Domination Reformation Political Polity Therapeutic Domination Participation Suspicious Authority Reclaim conflict Claim space for lost voices Mass Carnival Ironic Marginalisation Diversity Playful Normalisation Public

15. Formal vs Informal IT Assessments • Many organizations base their risk assessment approach around the narrower definition of evaluating attributes & metrics … the normative / ordered view of the world • IT senior management often subscribe to the ‘measured world’ / business school approach to management • This has lead to the belief that the ‘metrics’ themselves will provide the ‘answer’ … the interpretive view is not often considered Public

16. In summary • Understand subjective vs. objective paradigms … and the strengths & weaknesses of each • Understand how power, coercion & socio- political structures, contextually dominate the organizational landscape • Understand that evaluation is not measurement • Acknowledge that as a social process all evaluations are essentially agreed realities Public

17. A Risk Evaluation Methodology Public

18. Risk, risk impact and risk exposure • Definition of risk • ‘the potential for realization of unwanted, negative consequences of an event’ • the degree of uncertainty (probability between 0 & 1) regarding the occurrence of the problem and • a (negative) effect on the environment if the problem occurs • Magnitude of loss is known as risk impact • risk impact multiplied by the probability is referred to as the risk exposure (RE) • Risk Reduction Leverage (RRL) • used to determine the effectives of possible counter measures by comparing costs to its expected benefit • RRL = (REbefore – RE after) / Cost of Measure Scientific Nominalism …however the actual evaluation will be based on an agreed reality! Public

19. A risk evaluation methodology (five step cyclic methodology) • Risk Foundation • Risk Evaluation • Corrective Action • Risk Reporting • Risk Foundation Evaluation Cyclic / Self-improving Process Scientific Public

20. A risk evaluation methodology (although methodology is more scientific, the content of each step is significantly more subjective) • Risk Foundation • Risk Evaluation • Improvement Plan • Risk Reporting • Risk Foundation Evaluation Anti-positivism Positivism Evaluating the risks and formulating assessment questions Ideographic Evaluate (answer) the questions Nominalism Realism Plans to fix the problem Report on issues and improvements (tracking) Re-evaluate the questions and the frequency Public

21. What to do! Break up Information Technology (IT) into the core management / process disciplines Consider the key ‘questions’ to be asked & answered in each discipline … build a foundation based on risk (probability & impact) … however in reality this will be a subjective agreed reality 3. List the key ‘attributes’ that can be used in evaluating the questions (Key Risk Attributes) – KRAs and a threshold … only to be used as input into judging Nominalism Realism Positivism Public

22. Risk EvaluationUnderstanding the management of your business and associated risks using the core IT managementdisciplines as a base Public

23. 1. Core IT Disciplines What are the main TI disciplines? • Nine (9) core IT management disciplines (1) Architecture & Strategy (2) Change Management (3) Capacity Planning (4) Disaster Recovery (5) Information Security (6) Problem Management (7) Project Management (8) Software & Systems Management (9) Vendor and Outsource Management (10) … Public

24. Attribute Metric M L 1 1 Measurement procedure L M 2 2 Disciplines + Key Risks + KRAs / Thresholds Typically a few key questions per discipline K E Y R I S K S What are the main TI disciplines? • Nine (9) core IT management disciplines (1) Architecture & Strategy (2) Change Management (3) Capacity Planning (4) Disaster Recovery (5) Information Security (6) Problem Management (7) Project Management (8) Software & Systems Management (9) Vendor and Outsource Management (10) … Decide on attributes & thresholds to breach Public

25. Capacity PlanningExample Public

26. Risk Foundation Public

27. Collecting the Data Public

28. Risk Foundation Capacity planning: typical capacity data collection example Risk Description Capacity & performance data collection must be in-place for IT resource entities. The lack of data collection may lead to outages and system failures… Risk Review The IT entities inventory must be reviewed against key components and data collection verified / monitored. Key Risk Attribute (KRA) The total number of entities with key configuration components that are identified as not being monitored. KRA failure trigger – 10% threshold of key entities / configuration components identified as not monitored Public

29. Risk Foundation Capacity planning: question the questions! Risk Description Capacity & performance data collection must be in-place for IT resource entities. The lack of data collection may lead to outages and system failures resulting in the loss of business and / or reputation … Risk Review The IT entity inventory must be reviewed against key configuration components and data collection verified / checked. Key Risk Attributes (KRA) The total number of entities with key configuration components that are identified as not being monitored. KRM failure trigger – 10% of key entities / configuration components identified as not monitored Anti-positivism What data? (there are many tools and systems that collect data?) What is an ‘IT resource entity’? Where is this entity inventory? (make one up based on assets?) Nominalism What is considered key! Who defines what is key, and then who verifies this? Coercion Disintegration Modes of domination Inventory or what is KEY can be modified to manipulate KRM Public

30. Data Collection (example – can explain how you got to a conclusion!) Total Population (total ‘Inventory’) Nominalism Basic set maths Some type of validation Some level of collaboration Who says they ‘logical’? who concurs they key?… the business? who concurs what to measure (Business, CMG?) ‘Resource Entities’ (subset based on logical groupings) ‘Key configuration Components’ Vitalsuite Best/1 Perfmon Intervals Sampling Thresholds Regressions + + + … NOW? ‘what to measured’ Would not see wood from the trees if you tried to measure everything… Further collaboration from: 1) Technical Papers 2) Best Practices 3) Conference material 4) Other experience Anti-positivism … so does it does not actually matter what you measure (to a point) … but can you explain why you measure. Public

31. Risk Evaluation Public

32. Risk Evaluation Complexities A simple but practical example: In South Africa we have 100 ‘servers’ in total 50 are ‘production’ – are these the KEY systems? or is it a subset of 5 of these (i.e. the transactional systems)? - on the above, what are the keyconfiguration components? - disk, CPU, network cards (all or both or one)… - what about DR servers? Santon (Primary Site) Network links L O N D O N In South Africa we 25 network links… 20 are ‘production’ – are these the KEY systems? or is it a subset of 5 of these ? (e.g. SA -> London, gateway to Central Bank, Stock Market… etc links)? Midrand (Disaster Recovery) Nominalism … agreed reality… to the CEO his Laptop is KEY!!! Public

33. Risk Evaluation Complexities A simple but practical example: KRM can be breached OR not depending on how decide to interpret the environment. i.e. if I chose the 100I could in theory miss some really important entity / component and yet not exceed my KRM (i.e. 10% of 100 = 10) … ie collect data for 91 systems, but not the 9 transactional systems… Conflict Disintegration Coercion Public

34. Evaluating the Data Public

35. Risk Evaluation (example network utilization… no problems but mail is still slow)) KRM Public

36. Risk Evaluation (graphs from regional network team… what does this mean?) Public

37. Risk Evaluation (more graphs from regional network team… what does this mean?) Public

38. Philosophy (not even multiple sources could actually answer the specific risk evaluation) …the world is more complex than a set of metrics and measurements Heidegger (1889 - 1976): started to explore the concept of Dasein or ‘being IN the world’. This relates that understanding is not theoretical or intellectual and it might not even be articulated. It allows for practical understanding that becomes before the narrower objectivist view of the world. & = = Understand Risk? & & … its not only about being accuracy, relevance or validity … its also about experiencing, feeling & understanding! Public

39. Summary of Potential Weaknesses • Risk Foundation & Evaluation • risk questions can be found to be very complex, even when initially considered simple. Complexities include: • complexities in understanding the question • the subjectivivity of questions being open to interpretation • thus although the key risks can be identified, the understanding of how these risks applied to the local environment is open to interpretation • the complexity of any foundation leads such risk evaluations being highly interpretive and open to possible ‘miss-understanding’ • often completed by staff as a ‘best effort’ on own / single persons interpretationn • if same person is completing the evaluation every quarter / year, the same interpretation is levied, based on that persons perception • individuals tend to want to pass themselves (halo-effect) Public

40. Summary of Strengths • Risk Foundation & Evaluation • subjective questions ALLOW for interpretation based on local context - however one must apply broader thinking (i.e. based on an interpravist approach) • although the understanding of how risks applied is open, this gives you FLEXIBILITY to decied how you want to interpret it - this should be an agreed reality (and one you can convince an auditor of…) • demonstrate some interpretive thinking (i.e. consider multiple angles) and understanding of the subject & complexities - just looking at the narrower simplistic KRM value would be dysfunctional • lead people by example, draw pictures, be excited about the complexities and encourage debate and discussion! • workshop as a combine effort and open this as a challenge to THINK! Public

41. Two dimensions, four paradigms Social Discourse Interpretive Critical Dialogic Normative Basic Goal Hope Social Metaphor Org. metaphor Narrative Style Problems Addressed Org Benefits Mood Social Fear Law-like relationship between objects Progressive emancipation Economic Marketplace Scientific / technical Inefficiency / disorder Control Optimistic Disorder Unified culture Values Social Community Romantic Meaninglessness Quality work life Friendly Depersonalisation Unmask Domination Reformation Political Polity Therapeutic Domination Participation Suspicious Authority Reclaim conflict Claim space for lost voices Mass Carnival Ironic Marginalisation Diversity Playful Normalisation Public

42. Conclusion • Formal and informal evaluation run together in parallel and are closely inter-linked. Objectivist data and measurement must be interpretive and understood. 2) Management and evaluators need to understand that IT risk evaluation is more than the positivistic, elite or a priori view reported but a complex socio-political process. • Understand sociological and psychological influences and the dimensions of conflict & disintegration provides a deeper insight into a ‘being-in-the-world’ view of the risk evaluation process. • The risk foundation (questions) are open to provide a strong basis for you to understand your environment and control risks based on local context . Public

43. References • A critical analysis of information systems evaluation, R Hirschheim & S. Smithson, Templeton Collage Oxford (1986) • Dealing with risk: a practical approach, F.J. Heemstra & R. J. Kusters, Journal of Information Technology (1996), 11, 333-346. • Risk Management for information systems development, P.L. Powell & J.H. Klein, Journal of Information Technology (1996), 11, 309 – 319. • Sociological paradigms and organizational analysis, B. Gibson (1979). • The mechanics of managing IT risk, R. N. Charette, Journal of Information Technology (1996), 11, 337-378. Public