Assessment instruments for nuclear safety culture: what are we measuring?

Assessment instruments for nuclear safety culture: what are we measuring? Antonio C. O. Barroso IPEN – CNEN/SP

Presentation roadmap • Origen of the term • IAEA guidance • Has the concept permeated? • Assessment instruments • What are we measuring?

Origen of the term … Chenorbyl • From the aftermath, at IAEA Equipments Systems Procedures Controls Management People A socio-technical system Organizacional Culture

Origen of the term - IAEA in 1986 INSAG1 – The Chernobyl Accident “A vital conclusion drawn from this behaviour is the importance of placing complete authority and responsibility for the safety of the plant on a senior member of the operations staff of the plant. Of equal importance, formal procedures must be properly reviewed and approved must be supplemented by the creation and maintenance of a “nuclear safety culture”. Let’sgettheproperunderstandingof

IAEA guidance - basedonSchein Monica Haageat ICTP 2010 m.haage@iaea.org

IAEA guidance - duringthe 90’s IAEA INSAG 4 1991 IAEA INSAG 1 1986 IAEA INSAG 7 1992 IAEA SRS 1 1998 IAEA SRS 11 1998 A “definition” “Safety Culture is that assembly of characteristics and attitudes in organizations and individuals which establishes that, as an overriding priority, nuclear plant safety issuesreceives the attention warranted by theirsignificance”.

IAEA guidanceand ...March 2011 IAEA TECDOC 1321 2002 IAEA TECDOC 1707 2013 2011-03-11 INSAG Series 15 2002 IAEA SRS 74 2012

Hastheconceptpermeated? • Humans alone and in groups are the root agents of organizations and, as such, we could build upon the definition stated in INSAG 4 • Safety Culture is the assembly of systems, characteristics, mindset and attitudes at the organizational and individual levels which assures that: • (a) as an overriding priority, nuclear safety issues receive the attention warranted by their significance; and • (b) adequate resources, information and actionable knowledge are empowered at the decision / action points where safety issues are dealt with.

Hastheconceptpermeated? a bibliometricview • FromScopus data base untilDec/2011 • From 1997 -2011: nuclear safety culture seemed to be left out of the agenda of the nuclear industry. • In relevant journals and conferences, the interest on the subject has been near stagnant, both in terms of publications and citations. • At the same time for the other areas taken together, there is a vigorous increasing interest on the subject

Hastheconceptpermeated? a bibliometricview

Has the concept permeated? a bibliometric view • It could be unfair to compare numbers of nuclear area to all others, but it is wise and fair to compare the growth rates Are we happy with that?

Assessment instruments I don’t agree that “What You Can't Measure, You Can't Manage”, but I do agree that “What You Can Assess, You Can Better Manage” IAEA documents provide thorough, diverse and valuable guidance, but they are only sketchy concerning quantitative assessment models. Note that there are other assessment methods and we need them all, but I am going to focus on the quantitative ones.

Assessment instruments - reflective or formative • Is safety culture a reflective or formative construct? Health Diet (reflective) Health conservation (formative) Health Cons. Health Diet

Assessment instruments - reflective or formative Blood alcohol content Drunkness level Formativeconstruct Reflectiveconstruct # Wine glasses # beer cans # vodka drinks # whisky doses Blow test Physical reaction test Clocked memory test Observable variables (indicators) not necessarily correlated (should be complete) Observable variables (indicators) that should be correlated (some are enough)

Assessment instruments – reliability and validity

Assessment instruments – reliability and validity • Dimensionality of the construct • Content validity • Convergent validity • Discriminant validity • Predictive validity • Reliability “validation is cumulative” multiple studies, different settings …

Assessment instruments - “validated” modelsof nuclear safetyculture? Web ofscience – Apr/2014

What are we measuring? • Article 1 - LEE, T. Assessment of safety culture at a nuclear reprocessing plant. Work and Stress, v. 12, n. 3, p. 217-237, 1998. • Safety culture assessment of Sellafield reprocessing plant focusing safety related attitudes. Initial questionnaire had 172itens and gather data from 5926 participants. Initially EFA by PCA resulted in 38 factors, but later reduced to 19 and 81 items. Predictive validity was done against the number of reported accidents with significant time loss. • The model was not further evolved and it would be unpractical bytodays standards

What are we measuring? • Article 2 - Lopes de Castro, B.; Gracia, F. J.; Peiró, J. M.; Pietrantoni, L.; Hernández, A. Testing the validity of the International Atomic Energy Agency (IAEA) safety culture model. Accident Analysis and Prevention, v. 60, p. 231-244, 2013. • They have used as model the 5 characteristics (factors) and 37 attributes (indicators) recommended by AEA Safety Report Series no. 42 (2005) and have tried to validate it. • The model failed to present adequate psychometric capacity. Most of the indicators were not good manifestations of their factors, low face validity and only moderate content validity. Most alarming was the lack of discriminating validity.

What are we measuring? • Not much, or • Not nuclear safety culture • But other segments are! • For instance in Hospitals and similar organizations many steps of the “cumulative” validation have been done • a model for patient safety culture, developed in 2005, has been applied, adapted and validated in different settings (10) and countries (5)

Assessment instruments for nuclear safety culture: what are we measuring?