What is the Decision Rule all about? ISO/IEC 17025 2005 & 2017

1. What is the DecisionRule all about?ISO/IEC 170252005 & 2017 By Maryam Sourani Reference: John Wilson Accreditation and Metrology Services (Pty) LTD

2. What is a decisionrule Let’s start with what is adecision?

3. What is a decision rule Let’s start with what is a decision?

4. What is a decisionrule Let’s start with what is adecision? Years Where will the next reading probablybe?

5. What is a decisionrule Let’s start with what is adecision? Years YesLikely

6. What is a decisionrule Let’s start with what is adecision? Years Not likely butpossible

7. What is a decisionrule Let’s start with what is adecision? Years Are these readingsin Specification?

8. What is a decisionrule Let’s start with what is adecision? Nominal Are these readings in Specification? Before we decide, we need to know thelimits (specification)

9. What is a decisionrule Let’s start with what is adecision? Nominal So are the readingsin Specification? LowerLimit

10. What is a decision rule • In the current version of ISO/IEC 17025:2005itstates in clause 5.10.4.2 “When statements of conformance are made, the uncertainty of measurement shallbe taken intoaccount” • A statement of conformance is something like “The calibration of this instrument shows that all the readings are withinthe • manufacturer’s specifications” or “This instrument complies with the requirementsof the XYZstandard”.

11. What is a decision rule Let’s look at some metrology UpperLimit Nominal Onthis graph, the uncertainty of measurement has not beentaken intoaccount. LowerLimit

12. What is a decisionrule Let’s look at some metrology UpperLimit Nominal The Uncertainty bars arenow included LowerLimit John Wilson2016

13. What is a decision rule Let’s look at somemetrology UpperLimit Nominal So are these readingsin Specification? LowerLimit

14. What is a decisionrule Let’s look at somemetrology UpperLimit Nominal Are these readingsin Specification? LowerLimit

15. What is a decisionrule Let’s look at somemetrology UpperLimit Nominal Three differentgroups ofreadings. LowerLimit

16. What is a decisionrule • Let’s look atsomemet rology • UpperLimit • Fail Undecided Nominal So are the readingsin Specification? Pass LowerLimit

17. What is a decisionrule • Let’s look atsomemet rology • UpperLimit • Fail Undecided Nominal Sodoesthismeetthe requirements of ISO/IEC17025? Pass LowerLimit

18. What is a decisionrule • Let’s look atsomemeetology • UpperLimit • Fail Undecided Nominal The undecidedcould however lead to lots ofarguments. Pass LowerLimit John Wilson2016

19. What is a decisionrule • Let’s look atsomemet rology • UpperLimit • Fail Undecided Nominal The undecidedcould however lead to lots ofarguments. Pass LowerLimit Readings 1 to 4 are clearly “pass” and within Specification. Reading 7 is clearly “fail” and outsideof Specification. It is readings 5 and 6 that cause thearguments.

20. What is a decisionrule • Let’s look atsomemet rology • UpperLimit • Fail Undecided Nominal The undecidedcould however lead to lots ofarguments. Pass LowerLimit If you are the user, then the undecided are not acceptable but if you are the supplier you could argue that that they are acceptable because of the uncertainty associated with thereadings.

21. What is a decisionrule • Let’s look atsomemet rology • UpperLimit • Fail Undecided Nominal The undecidedcould however lead to lots ofarguments. Pass LowerLimit In this case we have a series of readings over a period of time so we have an understanding of how the artefact behaves and can reasonably predict what will happen in the future. This reduces the risk involved in adecision.

22. What is a decisionrule • Let’s look at somemetrology • UpperLimit Nominal Similarly – if we had several of theLowerLimisame instruments, such as a production run, and get the above results, we know a lot more about the behaviour of theproducts

23. What is a decisionrule • Let’s look at somemetrology • UpperLimit Nominal If we had only one (or one set) of readings, would we view it anydifferently? Is this a pass or fail? There may be a lot of money or risk resting on thedecision.

24. What is a decisionrule • Let’s look at somemetrology • UpperLimit Nominal Many cal labs would not make a decision based on this information but would rather pass the information to the customer to decide for themselves. So this would not be a statement ofconformance but merely a calibrationreport.

25. What is a decisionrule • Let’s look at somemetrology • UpperLimit Nominal If however the lab was tasked to give a statement of compliance, what would the decisionbe? Clearly it depends who’s side you are on. Different labs could give different decisions. So the customer needs to be told what “decision rule” you applied to the statement ofcompliance. John Wilson2016

26. What is a decisionrule Decision rules have been used for many years, especially in the calibration and physical testing industries. Their use has expanded because they are applicable to any situation in which data is used to make a decision. The terminology for decision rules includes the following zones and types of zones.

27. What is a decisionrule In a simple decision rule, the rejection and acceptance zones line up with the specification zone. The simple decision rule is illustrated in Figure 1. Many compendial monograph specifications may be considered simple decision rules. Figure 1: A Simple Decision Rule for a Specification with both Upper and Lower Limits. If the measurement result lies in the acceptance/specification zone, product is accepted, otherwise it is rejected.

28. What is a decisionrule A stringent acceptance zone reduces the probability of accepting out-of-specification product by increasing the risk of rejecting in-specification product. A guard band zone is established that offsets the specification and acceptance limits to achieve this. The size of the guard band depends on the desired probability of making each type of wrong decision. The stringent acceptance zone can be accompanied by a relaxed rejection zone, which allows the rejection of product even though a measurement result within the specification zone by the guard band amount is obtained. Many companies use such a decision rule and call it “internal release limit.” This decision rule is illustrated in Figure 2 for a specification with both upper and lower limits and in Figure 3 for a specification with an upper limit only.

29. What is a decisionrule Figure 2: A Stringent Acceptance Zone and Relaxed Rejection Zones Decision Rule for a Specification with both Upper and Lower Limits. Product is accepted if the measurement result lies within the acceptance zone and rejected otherwise. The guard band is established by the acceptable probability of making a wrong decision and the uncertainty.

30. What is a decisionrule Figure 3: A Stringent Acceptance Zone and Relaxed Rejection Zone Decision Rule for a Specification with an Upper Limit Only. A transition zone decision rule is illustrated in Figure 4 for a specification with an upper limit only. The transition decision rule should specify the required actions when a measurement result within the transition zone is obtained. The USP <905> "Uniformity of Dosage Units" (6) specification is an example of a decision rule that uses a transition zone. If the acceptance value of the first 10 dosage units is less than or equal to L1%, the requirements for dosage uniformity are met. If not, additional testing is performed. The region greater than or equal to L1% can be considered the transition zone.

31. What is a decisionrule Figure 4: A Transition Zone Decision Rule for a Specification with an Upper Limit Only. Decision rule construction requires four components: the measurement result, its measurement uncertainty, the specification limit or limits, and the acceptable level of the probability of making each type of wrong decision. Choice of a decision rule is a business consideration that takes into account the cost of rejecting an in-specification product, the cost of accepting an out-of-specification product, the uncertainty associated with the measurement, the distribution of the measurand’s characteristic, and the cost of making the measurement. The analytical quality-by-design (QbD) approach uses risk analysis and probability to determine these four components, hence clearly defining the use of a procedure through a decision rule. The relationship between these four components allows one to determine if an analytical procedure is fit-for-use and, furthermore, set acceptance criteria for the analytical procedure to meet. Measurement uncertainty and its relationship to decision rules will be explored in future columns

32. What is a decisionrule • The current wording of 17025:2005 leaves it very vague and open to a wide range of interpretation. • TheILAC G-8:2009 has tried to provide some clarity by writing this guideline on how to look at pass/fail conformity assessment. • TheJCGM 106:2012 “Evaluation of Measurement Data- the Role of measurement uncertainty in conformance assessment is a further guideline on how to deal with the problem.

33. What is a decisionrule • In the current version of ISO/IEC 17025it • states in clause 5.10.4.2 “When statements of conformance are made, the uncertainty of measurement shall be taken intoaccount”

34. What is a decisionrule • The ILAC G-8:2009 has tried to providesome clarity by writing this guideline on how to look at pass/fail conformityassessment. • Compliance: If the specification limit is not breached by the measurement result plus the expanded uncertainty with a 95% coverage probability, then compliance with the specification can be stated (See Case 1 of Fig.1). This can be reported as “Compliance” or “Compliance – The measurement result is within (or below) the specification limit when the measurement uncertainty is taken into account”. In calibration this is often reported as“Pass”;

35. What is a decisionrule • The ILAC G-8:2009 has tried to providesome clarity by writing this guideline on how to look at pass/fail conformityassessment. • Non-compliance: If the specification limit is exceeded by the measurement result minus the expanded uncertainty with a 95% coverage probability, then noncompliance with the specification can be stated. (See Case 4 of Fig.1) This can be reported as “Non-compliance” or “Non- compliance – The measurement result is outside (or above) the specification limit when the measurement uncertainty is taken into account”. In calibration this is often reported as“Fail”;

36. What is a decisionrule • The ILAC G-8:2009 has tried to providesome clarity by writing this guideline on how to look at pass/fail conformityassessment. • If the measurement result plus/minus the expanded uncertainty with a 95 % coverage probability overlaps the limit, it is not possible to state compliance or non-compliance. The measurement result and the expanded uncertainty with a 95 % coverage probability should then be reported together with a statement indicating that neither compliance nor non-compliance was demonstrated. • This is what most SANAS labs usetoday

37. What is a decisionrule • TheJCGM 106:2012 “Evaluation of Measurement Data- the Role of measurement uncertainty in conformanceassessment” isa further guideline on how to deal with the problem. • This document address a lot more detailand gives different ways of interpretingresults • This goes intothe“knowledge of the measurand” , Bayes Theorem, conformance probabilities and muchmore”

38. What is a decisionrule • So the bottom line is that much more knowledge on the subject is required to enter into an educated discussion withthe • customer about the probability of “False accepts” when a product should failand • “False rejects” when a product shouldpass. • It is suggested that a good study of JCGM 106:2012 ismade. • It is available free off the BIPM website • There is also a lot more informationavailable • on the Web related to thissubject.

39. What is a decisionrule • So why all thefuss? • Again, it depends if you are the useror the supplier and it boils down to the issue of the probability of “False Accept” or “False Reject”. What is the risk to bothparties. • Whilst the current 17025 and the ILACG- 8:2009 give direction it is not always enforced. • Very few people know about the JCGM106.

40. What is a decisionrule • So what is proposed in the new 17025? • Analysis of theresults • 7.7.1 Evaluation ofconformance • When statement of conformity to a specification or standard for test or calibration is requested, the laboratory shall: • document the decision rules employed taking into account the level of risk associated with the decision rule employed (false accept and false reject and statistical assumptions associated with the decision rule employed); • apply the decision rule. • NOTE For further information see ISO/IEC Guide98-4.

41. What is a decisionrule • So breaking this downfurther • When statement of conformity to a specification or standard for test or calibration is requested, the laboratory shall: • The key here is “when it is requested”which implies it is requested by thecustomer. • This means that “contract review” must take place and a clear definition agreed on BEFORE the job isstarted.

42. What is a decisionrule • During Contractreview • The laboratory needs to be prepared to discuss what the decision rule options are regarding the compliance statement. • They also need to understand what the customer may require and where the risk of “false accept” or “false reject” lies so that they do not get drawn into legal battles or compensationclaims. • As with all statistical analysis, it is open to different interpretations and care must be taken to ensure a correct agreement betweenparties.

43. What is a decisionrule • During Contractreview • There are many models which may be applied and they will vary greatly depending on the application. • You may be testing a new prototype whereno previous or group history is available or you may be checking a widely manufactured and accepted item against a manufacturer’s specification. The approach will be very different.

44. What is a decisionrule • Different approaches to eachcondition One reading(or set of readings on aprototype Oneinstrument over a longtime One batch ofthe same model instrument

45. What is a decisionrule • Different approaches to eachcondition • A knowledge of “Guardbands” and the waythey • are implemented isadvisable. UpperLimit Guardband NewLimit Nominal NewLimit Guardband LowerLimit

46. What is a decisionrule • Different approaches to eachcondition • UpperLimit • Guardband • Limit -upper • Nominal • Guardband Limit -lower • Lower Limit

47. What is a decisionrule • Different approaches to eachcondition • For example – if the TAR method is used and a 4:1 TAR is applicable then the rulesare • Both the UUT and STD must be workingin normal operatingspecifications • The STD must be calibrated and withinthe normal calibrationinterval • The UUT reading must be < 70% of the normal specification limit of theUUT • (This is a 30%guardband)

48. What is a decisionrule • Different approaches to eachcondition • Similarly, if the guidance of the ILAC G8 is used, then it must be clearly understood and stated as such. • The same will apply to the JCGM106. • There are also many ISO/SANS/Industry standards that define clearly what decision rule must beapplied.

49. What is a decisionrule • In theLab • … the laboratory shall: • a) document the decision rules employed taking into account the level of risk associated with the decision rule employed (false accept and false reject and statistical assumptions associated with the decision ruleemployed); • Take the time to clearly document what has been agreed and ensure that lab procedures allow for the correct treatment of the readings anduncertainties.

50. What is a decisionrule • In theLab • … the laboratory shall: • a) document the decision rules employed taking into account the level of risk associated with the decision rule employed (false accept and false reject and statistical assumptions associated with the decision ruleemployed); • Might need to change test procedures to cater for the decision rule that will be applied. It has to be such that anybody in the lab will know what is required and what the limitsare.