IRRS REFRESHER TRAINING Lecture 2

1. IRRS REFRESHER TRAININGLecture 2 Methodology of the review

2. Outline PART I • General review principles • Team work • Daily team meetings • Counterpart and reviewer involvement in data collection PART II • Reviewing additional written material • Taking notes • Interviews • Direct observations • Site visits Refresher IRRS Training - Methodology of the Review

3. Learning objectives After finishing this lecture the trainee will understand: • The general review principles behind an IRRS mission • How team work is organized and conducted in IRRS • The purpose, conduct and output of Daily Team Meetings • The roles of reviewers and counterparts in data collection • The role, methodology and contribution of different data collection methods in IRRS: review of additional documentation, note taking, interviews, direct observation, site visits Refresher IRRS Training - Methodology of the Review

4. The GreatChallenge The IRRS team has, in a short time and in a new environment, to: • look for issues • identify findings • agree findings with counterpart • write a report This Training Lecture provides recommendations on how to work in order to accomplish these tasks in the best way Refresher IRRS Training - Methodology of the Review

5. Contents of Part I General • General review principles • IRRS Team Goal • Team work • Daily team meetings • Counterpart and reviewerinvolvement in data collection Refresher IRRS Training - Methodology of the Review

6. General review principles Refresher IRRS Training - Methodology of the Review

7. General review principles (1) The reviewer should be Well prepared: Use the IRRS Guidelines, IAEA Safety Standards,ARM including the self-assessment and the action plan Reviewers should not refer to their national practice! Efficient: Open minded, rely on facts, look for examples Take notes Professional Team member Refresher IRRS Training - Methodology of the Review

8. General review principles (2) Remember: As a reviewer you always have to be flexible and receptive Recognize that there are several ways to accomplish a certain objective or task Do not try to impose your own solutions on the counterpart but give examples if necessary to illustrate an issue A relaxed interview situation is best for all parties You and your counterpart are colleagues, not competitors or adversaries, and should share the same interest in having a constructive peer-review Refresher IRRS Training - Methodology of the Review

9. Team Work Refresher IRRS Training - Methodology of the Review

10. IRRS Team Goal What are we here to do? A joint search for improvement of nuclear regulation through providing recommendations and suggestions, and to illustrate good practices • Review against the IAEA Safety Standards and best international practice by • reviewing documents • interviewing • observing and • Write a report consistent with the IRRS Template, and all in a sprit of good teamwork Refresher IRRS Training - Methodology of the Review

11. Team work, methodology (1) An IRRS mission can not succeed without good team work. Good team workers: • Support and help other team members • Seek guidance or assistance from other team members • Use available good English speakers for language review • Accept and use critical comments • Do not be defensive • Critical comments from the TL/DTL should be understood as a challenge to the issue, not a criticism of the team member´s work Refresher IRRS Training - Methodology of the Review

12. Team work, methodology (2) Seek and accept assistance or guidance from the TL or DTL early with respect to: • Problems with counterparts • Problems with interpreters • Difficulties in writing working notes • Personal problems • Problems with team activities or relationships • Let the team leader know if your counterpart invites you home or out Refresher IRRS Training - Methodology of the Review

13. Team work, expectations (1) • Be open minded and critical in thought, objective in discussions and writing • Work hard: • Evenings are work time • Write and update working notes • Define your issues • Prepare for the next day • Relax on free Sunday and enjoy the social programme • Prioritize your effort • You can not cover everything in IRRS Guidelines to the same depth • Use discussions and observations to identify performance and importance Refresher IRRS Training - Methodology of the Review

14. Team work, expectations(2) • Use • IAEA Safety Standards • The summary reports of the IRRS findings as reference material • Prepare well for team meetings • Focus on potential issues • Respect the time of others • Stay on schedule • Review work - daily • Working notes writing - daily • Remember: you represent IAEA Be a team worker, help other reviewers! The Team Leaders are there to support you all! Refresher IRRS Training - Methodology of the Review

15. Daily team meetings Refresher IRRS Training - Methodology of the Review

16. Daily team meetings, purpose Meeting of the whole team and the host country Liaison Officer at the end of each working day, until the first draft of the mission report has been completed. Its purpose is • To discuss the main observations of the day • To share findings and insights regarding technical and policy issues related to the different review areas Refresher IRRS Training - Methodology of the Review

17. Daily team meetings … are not meant for general activity reporting by reviewers • The team leader has to establish an effective style and conduct of these meetings to prevent wasting reviewer´s time meant to work on their reports Refresher IRRS Training - Methodology of the Review

18. Daily team meetings, conduct Common mistakes: • The meeting has no standardized structure • The TL asks around the table for the day´s impressions • The team members have not prepared to present conclusions and concerns only, but provide a general report about the activities of the day • The TL does not limit the time for presentations and discussions, the meeting can take several hours • The TL does not provide clear conclusions and directions for the further work • The Liaison Officer interferes in discussions in a defensive way rather than answering questions and providing clarifications Refresher IRRS Training - Methodology of the Review

19. Daily team meetings, conduct (3) Summary of Daily Team Meetings Is meant to summarize daily observations and cross-cutting issues, it is not an activity report Team members need to thoroughly prepare the report to the meeting Group-wise Both the Group-reports and the entire meeting need to be limited in time Should be closed with conclusions and instructions for the next day Refresher IRRS Training - Methodology of the Review

20. Counterpart and reviewer involvement indata collection Refresher IRRS Training - Methodology of the Review

21. Counterpart involvement The reviewer should encourage the counterpart to: • Actively contribute • Help obtain correct and complete information • Assist in the search for improvement • Develop a sense of ownership • Keep an open minded approach • Not to be defensive Refresher IRRS Training - Methodology of the Review

22. Reviewer involvement The reviewer needs to • Make preparations • Control the length of host organization´s presentations • Ask questions for clarification • Concentrate on important aspects • Keep copious notes • Check related documentation • Check processes or records • Check any performance statistics/indicators Refresher IRRS Training - Methodology of the Review

23. END OF PART I Refresher IRRS Training - Methodology of the Review

24. Contentsof Part II Data collection methods • Reviewing additional written material • Interviews • Good interviewing practices • Direct observations • Site visits Refresher IRRS Training - Methodology of the Review

25. additional written material Refresher IRRS Training - Methodology of the Review

26. Requesting and reviewing additional written material(1) • During the mission, additional material will need to be requested • Typical examples are: • Lower level Management System documents, such as procedures and work instructions • Annual activity planning and follow up documents • Training plans and programmes for the regulatory staff • Inspection reports • Event reports • Safety analysis reports or parts of such reports • MoU with other national bodies Refresher IRRS Training - Methodology of the Review

27. Requesting and reviewing additional written material (2) Refresher IRRS Training - Methodology of the Review • For somedocuments fulltranslations can often not be provided, then • Ask for a translated summary • Ask the counterpart to explain the contents of the document • Use the Google translator, if possible • The regulatory body´s intranetmaycontaintherequireddocuments

28. interviews Refresher IRRS Training - Methodology of the Review

29. Purpose of interviews to gather information not covered by the written material to seek clarification of written information provided toprovide an opportunity for host regulatory body staff to discuss their practices and professional opinions with the reviewers Refresher IRRS Training - Methodology of the Review

30. Goodinterviewingpractices(1) Prepare properly for each interview: • Reviewalldocuments and activities availableprior to the interview • Develop several questions to cover the area • Formulate to avoid simple answers like ”yes” and ”no” • Ask questions only to persons who are familiar with the area under review Refresher IRRS Training - Methodology of the Review

31. Good interviewing practices (2) • Make sure your words are clear and easy to understand • Keep the right distance between you and the interviewee, consider cultural aspects • Control your body language and try to interpret the body language of your interviewee • Maintain a professional attitude at all times Refresher IRRS Training - Methodology of the Review

32. Good interviewing practices (3) At the very beginning • Let the interviewee calm down and show a positive attitude • Explain review targets* • Ask for a short presentation, if needed, just to explain main activities, roles and responsibilities in the reviewing area * For nervous interviewees, emphasise that you ask because you would like to know how the organization works, not in order to inspect or to assess individual performance Refresher IRRS Training - Methodology of the Review

33. Good interviewing practices (4) How to ask? • Keep questions simple and focussedonfacts • Use open ended questions • Use a funnel approach: frombroad leading to specific questions • Avoid devious or trick questions Refresher IRRS Training - Methodology of the Review

34. Good interviewing practices (5) How to listen? • Be interested - show enthusiasm • Maintain eye contact • Listen to the answer before asking next question • If you don´t understand the answer, ask for clarification • Do not let note taking interfere with your listening • Request copies of documents for later study • Don´t assume, ask questions Refresher IRRS Training - Methodology of the Review

35. Direct observations Refresher IRRS Training - Methodology of the Review

36. Direct observations (1) • Opportunities to observe work activities should be taken both at the regulatory body offices and during site visit • Activitiestoobservemay be • meetings, • audits, • inspections, • exercises, trainingsessions • Use of electronicdocumentation Refresher IRRS Training - Methodology of the Review

37. Direct observations (2) Remember: • A precondition for using observation as a data source for conclusions is that the context of the observed activity is understood by the reviewer • Prepare by checking relevant procedures and if necessary ask the counterpart to explain the significance of the activity and to translate (discretely) during the activity • Be aware that presence of an observer could affect the observed activity • Observations should be properly recorded in technical notes and reflected in the mission report. Refresher IRRS Training - Methodology of the Review

38. Site visits Refresher IRRS Training - Methodology of the Review

39. Site visits, purpose Certain interviews and direct observations of regulatory practices and activities (typically inspections) should be carried out on the sites of authorized organizations Such visits will enable a better understanding of how the regulatory body discharges its responsibility and give an opportunity for reviewers to meet with licensees Refresher IRRS Training - Methodology of the Review

40. Site visits, types of information Through observations of work activities and interviews with the regulatory body staff and the licensee staff types of information to be gathered may include: • Perceived and actual roles and responsibilities of the regulator at the visited site • Resources to fulfil those responsibilities, including facilities, equipment and staffing • Knowledge, skills and abilities of the regulatory inspectors to perform inspections • Effectiveness of the regulatory processes • Elements relevant to policy issues • Relations between the regulatory body and the licensee Refresher IRRS Training - Methodology of the Review

41. Site visits, conduct • The visit should start with an opening statement to the site host informing about the scope of the IRRS mission, the purpose of the visit and questions to be asked • It is important that staff of the visited facility understand in advance that the visit is not an IAEA review of their facility but is conducted solely to observe practices and actions of the regulatory staff Refresher IRRS Training - Methodology of the Review

42. Site visits, conduct(2) • On site the IRRS reviewers should concentrate on observing the site inspector conducting a normal inspection or other routine regulatory tasks • Before the walk around, the inspector should brief the reviewers about the planning process for the inspection • During the walk around the reviewers should act as observersand not interfere in the inspection activities • Reviewers may ask the inspector to explain what he is doing if this does not disturb his/her activities • After the inspection, the inspector should inform the reviewers about his/her observations and conclusions and what the next steps are Refresher IRRS Training - Methodology of the Review

43. Site visits, conduct (3) • Such information will provide a valuable complement to the information provided by the regulatory body itself • During interviews with licensees, the preference is that host regulatory body staff is not present, in order to facilitate open and frank discussions • After the visit, reviewers should discuss their findings with the full IRRS team at the next Daily Team Meeting Refresher IRRS Training - Methodology of the Review

44. END OF PART II Refresher IRRS Training - Methodology of the Review

45. Summary (1) An IRRS is not possible without good team work Every reviewer needs to prepare well and be efficient, professional and skilled Reviewers are assigned to review groups (A-F) Good team workers are flexible and open minded The daily team meeting is the most important instrument for discussing conclusions and agreeing recommendations, suggestions and good practices Refresher IRRS Training - Methodology of the Review

46. Summary (2) To collect data during the mission: Additional documentation needs to be reviewed Copious notes should be taken at the earliest opportunity Lists of questions should be prepared for interviews Interviews should be conducted in a friendly and professional manner and encourage the counterpart to provide substantial information, don´t assume – ask Direct observation of work activities could provide valuable additional information Site visits are vital for understanding of the regulatory effectiveness and the relations with the licensees Refresher IRRS Training - Methodology of the Review

47. THANK YOU FOR YOUR ATTENTION This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission Photo: V. Friedrich Refresher IRRS Training - Methodology of the Review