Detecting Improper Laboratory Practices

1. Detecting ImproperLaboratory Practices A Toolbox for Assessors

2. Module 1 Orientation

3. Module 1 - Topics • Impact of Improper Practices • Purpose and Scope of Training • Assessment Process Flow Chart • Assessor Roles and Responsibilities • Assessor Skills and Qualifications • Responding to Improper Practices • Definitions and Examples

4. Impact of Improper Practices • Sound decision-making requires reliable data of known and documented quality • Laboratories are coming under increasing scrutiny for improper practices • Improper practices result in millions of dollars in unreliable data • Impacts • Public Health • Regulatory Programs • Industry • Laboratories • Individuals

5. Purpose and Scope of Training • This course presents tools for identifying improper laboratory practices during routine assessments • Presumes knowledge of laboratory assessments such as: • National Environmental Laboratory Accreditation Program (NELAP) • Good Laboratory Practices (GLP) • Drinking Water Certification program • Pre-qualification (e.g., project-specific assessments)

6. Purpose and Scope of Training • This is supplemental training for experienced assessors • Determining intentional wrongdoing is a job for the investigator

7. The Assessment ProcessFlow Chart

8. Assessor Roles and Responsibilities • Verify adherence to specifications (quality system documents and SOPs) • Be alert to deficiencies (non-conformance to a standard) • Probe further when a deficiency is observed • Have contacts from whom to seek advice • Follow your assessment plan and observe your assessment program’s policies

9. Assessor Roles and Responsibilities • Be alert to patterns of behavior that could indicate a systematic weakness • Document findings and gather sufficient objective evidence to support deficiencies • Inform management of deficiencies • Recognize good practices

10. Assessor Skills & Qualifications • Assessment training and experience • Verbal and written communication skills • Interviewing • Documenting deficiencies • Expertise in areas being assessed • Laboratory instrumentation (including computer skills) • Data collection, reduction, analysis, and reporting • Record-keeping

11. Assessor Skills & Qualifications • Knowledge of applicable specifications • Relevant federal and state regulations • Laboratory assessment and quality systems standards • Methods • Minimum QA and QC requirements Flexibility is essential: Assessors must adapt as they go!

12. Responding to Improper Practices • Act impartially and observe due process (innocent until proven guilty) • DO NOT take actions that could compromise a future investigation • DO NOT overstep your authority (e.g., interrogate laboratory analysts until they “confess”)

13. Responding to Improper Practices (cont’d) Important Reminders • There may be a number of explanations for an observation • When in doubt, gather the objective evidence and records and seek advice • Be candid with laboratory management during the exit briefing and encourage open dialog about your observations

14. Responding to Improper Practices (cont’d) Circumstances that require follow-up: • You have been approached by a “whistle-blower” or otherwise suspect deliberate wrong-doing • You observe practices that could result in a potential threat to public health and safety • You observe a pattern of improper practices that raises concerns about data integrity

15. Responding to Improper Practices (cont’d) • Your organization should have specific procedures for managing assessment findings, including making referrals • Before you go on-site, make sure you have identified points of contact for both technical assistance and referrals

16. Terms Defined Red flag Deficiency Improper practice Fraud System vulnerability Objective evidence Data assessment

17. Definitions Red Flag or Warning Sign: An observation that indicates the potential for, or leads the assessor to suspect, system vulnerabilities or improper practices Examples: • Weaknesses in internal assessments/corrective action • High staff turnover in key areas • Unclear roles and responsibilities • Too few QC records to support data output

18. Definitions Deficiency: An unauthorized deviation from acceptable procedures or practices; a defect in an item; non-conformance with a specification Examples: • The laboratory has no arrangements for annual internal audits (ISO 17025 4.3.1, NELAC 5.xx 2002) • Lack of written SOPs (GLP standards 40 CFR Part 160 Section 160.81)

19. Definitions Improper Practice: A scientifically unsound or technically unjustified omission, manipulation, or alteration of procedures or data that bypasses the required QC parameters, making the results appear acceptable Any alteration of data such that the data are unauthentic or untrue representations of the experiment or test performed

20. Definitions Improper Practice - Examples: • Reporting failed QC results as acceptable • Reporting post-digested spikes as pre-digested • Changing instrument clock setting or altering date/time information • Manipulating peak area without approval/documentation • Selectively dropping calibration points

21. Definitions Fraud: A deliberate deception practiced so as to secure unfair or unlawful gain Reminder: An assessor can not determine whether an improper practice is fraud. A determination of fraud is the conclusion of a legal process which must evaluate the elements of both “intent” and “unlawful gain”. Until that point, the practice in question is an allegation of misconduct. (Webster’s II)

22. Definitions System Vulnerability: Any aspect of a Quality System that allows improper practices to occur and to go undetected Examples: • Inadequate training • Ineffective internal assessments • Lack of independent QA reviews • Lack of management controls

23. Definitions Objective Evidence: Data supporting the existence or truth of something Data Assessment: An in-depth review and reconstruction of data from raw/source data through final reporting

24. Any Questions?

25. Module 2 Pre-Assessment Activities

26. Topics • Requesting Documents and Records • Pre-Assessment Review • Pre-Assessment Red Flags • Developing the Assessment Plan

27. Requesting Documents and Records • Qualifications statement • Organizational chart • Quality Systems manual • Resumes for staff performing key functions • Previous assessment reports • Standard Operating Procedures (SOPs)

28. Requesting Documents and Records • Proficiency Testing (PT) sample results • Data Packages • Request specific data packages • Designate the type of data to be sent • Include cross-section of procedures

29. Pre-Assessment Review Questions During the pre-assessment review you should answer the following questions:

30. Pre-Assessment Review 1) Do initial signs point to a well-established internal assessment and corrective action program? 2) Does the pre-assessment submittal document proficiency in requested services?

31. Pre-Assessment Review 3) Does the technical depth appear to be adequate? 4) Have there been recent, significant organizational changes?

32. Pre-Assessment Review 5) Was the laboratory’s initial response prompt, complete, and organized? 6) Is management’s commitment to data integrity clearly spelled out?

33. Pre-Assessment Red Flags • Weaknesses in internal assessments/corrective action • Inadequate demonstration of proficiency • Questionable technical depth • Recent, significant organizational changes • Lack of responsiveness • Lack of management commitment to data integrity

34. Developing the Assessment Plan An assessment plan is a written document that identifies the scope and objectives of an assessment and includes: • Laboratory information including primary contacts • Names of assessment team members • Areas to be assessed • Documents reviewed during pre-assessment • Documents to review on-site • List of laboratory staff to interview • Schedule of assessment team activities while on-site

35. Developing the Assessment Plan 1) If you find weaknesses in internal assessments/corrective action, then: • Examine corrective action reports • Interview QA staff • Interview analysts

36. Developing the Assessment Plan 2) If you question the laboratory’s proficiency, then: • Interview analysts • Request process demonstrations • Review training records • Conduct data assessment

37. Developing the Assessment Plan 3) If you have concerns about technical depth, then: • Review additional resumes • Check training records • Interview key staff • Interview selected analysts

38. Developing the Assessment Plan 4) If you find recent organizational or ownership changes, then: • Verify that the quality systems manual has been updated • Include a cross-section of staff for interviews • Confirm that SOPs are current • Confirm that training records are up to date • Confirm that internal assessment files and corrective action records are up to date

39. Developing the Assessment Plan 5) If the laboratory has been unresponsive, then: • Reissue your request • Report concerns during opening meeting • Conduct on-site interviews of technical management, QA staff, and customer service staff

40. Developing the Assessment Plan 6) If a strong management commitment to data integrity is not apparent, then: • Interview staff at all levels • Review ethics or data integrity policy • Review internal assessment records and corrective action reports • Review training records • Perform data assessment

41. Summary • Note areas of concern during pre-assessment for follow-up • Identify additional resources for technical questions • Determine what information you will need to obtain on-site

42. Any Questions?

43. Module 3 Interviewing

44. Topics • Effective Interviewing Skills • Setting the Stage • Using Core Questions • Useful Interview Phrases • Know How to Respond to Questions • Interviewing Do’s and Don’ts

45. Effective Interviewing Skills • Organizational skills • Maintaining a non-threatening demeanor • Ability to listen • Ability to control situations and manage time • Ability to read body language

46. Effective Interviewing Skills • Ability to redirect • Ability to follow a lead • Accurately documenting interview • Ability to recognize when to refer to other experts • Know the right questions to ask

47. Setting the Stage • Start slowly • Put interviewees at ease • Tell them • who you are • why you are there • what you want to talk about • what are you going to do with the information

48. Using Core Questions • Asking several employees the same set of core questions can provide a sense of how the laboratory functions. Begin with: • What are your responsibilities? • How do you like working here? • How long have you been working here? • How do you handle overtime?

49. Using Core Questions (cont’d) • Move into job function/responsibilities • What do you do in a typical day? • Who is your backup? • Explain the data review process? • What is your repeat rate? Do you get a lot of rush jobs? • Tell me about recent training you have received • Does the laboratory have a bonus program? How does it work?

50. Using Core Questions (cont’d) • Draw out more specific information • How are you assigned work? • How do you document which work you did? • How do you interact with your quality assurance staff? • What do you do, and where do you go when you have a problem?