The National Metrology Laboratory of South Africa

1. The National Metrology Laboratory of South Africa Meeting the requirements of clauses 7.6 and 8.1 of ISO/TS 16949NAAMSA Metrology Interest Working Group – 16 February 2005 Eddie Tarnow Metrologist Automotive Metrology Laboratory

2. Discussion overview Points for discussion • •ISO/TS 16949: clause 7.6; Control of monitoring and measuring devices • Selection of the correct test & measuring equipment • Calibration of equipment • Verification/validation of equipment • Calibration intervals • Measurement traceability • Calibration status of equipment • Calibration integrity • Assessing the impact of “out of calibration” equipment • Calibration records • Validation of software used for testing measurement

3. Discussion overview Points for discussion continued • •ISO/TS 16949: clause 7.6; Control of monitoring and measuring devices • Laboratory requirements • Internal Laboratory • Laboratory scope • Competency of personnel • Testing of the product • Ability to test according to national/international specifications/methods • External Laboratory • Acceptance of laboratory to the customer

4. Discussion overview Points for discussion continued • •ISO/TS 16949: clause 8.1; Measurement, analysis and improvement, General • Demonstrating conformity of the product • ISO/TS 16949: clause 8.2.4; Monitoring and measurement of product • Measuring the characteristics of the product • Summary

5. ISO/TS 16949: Clause 7.6 Control of devices

6. Clause 7.6 Selection of appropriate test & measuring equipment • Have the required tests/measurements been clearly identified? • Has an analysis been conducted to determine the measurement accuracy required? What uncertainty is acceptable? • What instruments have been identified that will perform the measurement/test? (Are they fit for the intended purpose?) • Have the instrument accuracy specifications been correctly interpreted? (especially taking into account environmental tolerance) • Has cost vs accuracy been adequately evaluated? (TUR) • high accuracy – increased calibration costs, fewer calibration service providers, shorter calibration intervals, longer downtimes due to calibration • lower accuracy – increased risk of non-conforming product, lower cal costs, many cal service providers, longer cal intervals, quicker cal turnaround time • Is local technical support available?

7. ISO/TS 16949: Clause 7.6 Control of devices

8. Clause 7.6 Calibration of test & measuring equipment • Definition: Calibration– comparison against a standard of known higher accuracy, adjustment to agree or noting of the difference. • Is a full calibration conducted, or just a partial? • Is the instrument calibrated according to the manufacturer’s recommended performance verification procedure to within specification or merely to within a stated uncertainty? • Has the instrument been adjusted to get it to within spec? Or have correction factors simply been determined? • Have the results been recorded/reported “as found” as well “as left? (before adjustment & after adjustment) • Has the instrument been calibrated at the same points as previous calibration, thereby providing a useful history? • Has the calibration service provider evaluated the results for conformance to accuracy specification and reported a statement to that effect?

9. ISO/TS 16949: Clause 7.6 Control of devices

10. Clause 7.6 Verification/validation of test & measuring equipment • Definition: Verification– check that nothing has changed since the calibration • Definition: Validation– check to see the instrument is indeed fit for purpose • Has the required accuracy been proven by the calibration? (validation) • Is there evidence that the instrument accuracy remains within the specification? (If compared to last cal & no adjustment performed, verification) • Are measurement checks performed in between calibrations to maintain confidence in the original calibration status? Are the results of these checks recorded?

11. ISO/TS 16949: Clause 7.6 Control of devices

12. Clause 7.6 Calibration intervals • Definition: Calibration interval– the interval between consecutive calibrations to ensure the required accuracy is maintained. • Has a suitable calibration interval been scientifically determined taking into account the following factors? • desired accuracy, • stability of the instrument, • risk analysis of the impact of an instrument NOT within the required accuracy, • history of previous calibrations, • how often adjustment is required and when last was it performed, (previous cal history destroyed after adjustment), • influence of the cal service provider – • use of the same calibration method & • calibration at the same points • Does the calibration service provider determine the cal interval?

13. ISO/TS 16949: Clause 7.6 Control of devices

14. ISO/TS 16949: Clause 7.6 Measurement traceability • Definition: Measurement traceability – unbroken chain of comparisons between the UUT and the national measurement standard, each step of which has an estimated uncertainty associated with it. • Can the last step of the traceability chain be identified/described? (Is the identity of the standard used known and unique?) • Was the uncertainty of the standard used for the calibration acceptable? (Was the test uncertainty ratio valid?) • Was the calibration service provider SANAS accredited & was the calibration performed within the laboratory’s published accreditation schedule best measurement capability (BMC)? • Was the calibration procedure used validated and approved by the manufacturer of the instrument? (Were there any deviations from the manufacturer’s procedure and if so were the changes validated?)

15. CALIBRATION POINT100 mm ± 1 mm CALIBRATION POINT200 mm ± 1 mm IMPORTEDTRACEABILITY MEASUREMENT POINT150 mm ± ?mm TRANSFERREDTRACEABILITY ISO/TS 16949: Clause 7.6 Measurement traceability continued • Is there evidence of the competence of the metrologist who performed the calibration? • Is there an uncertainty of measurement reported with the results? • Does the instrument provide “direct” or “indirect” traceability for the intended measurement/test?

16. ISO/TS 16949: Clause 7.6 Control of devices

17. ISO/TS 16949: Clause 7.6 To adjust or not to adjust during calibration • Is there record of adjustments having been made during the calibration? • If adjustment has been performed, are the results before AND after the adjustment reported? • Does the calibration service provider have the technical ability to perform adjustments? (manufacturer’s technical procedure for performing adjustments as sequence, & influences of one adjustment on another, can be critical) • Advantages of adjusting to within spec • eliminates the need to apply corrections, • facilitates ease of use for the operator, • instrument can be used in other applications, • can reduce the cost of calibration, • simplifies the estimation of uncertainty of measurement when using the instrument

18. ISO/TS 16949: Clause 7.6 To adjust or not to adjust during calibration continued… • Disadvantages of adjusting to within spec • adjustment immediately invalidates (destroys) previous history, (this can be managed) • can increase the cost of calibration (some calibration service providers refuse to perform adjustments or do not have the technical ability – typically a problem if not using the authorised agents as the calibration service provider) • the natural drift of the instrument may be disturbed (the instrument may become more stable if left undisturbed for a long period of time)

19. ISO/TS 16949: Clause 7.6 Control of devices

20. ISO/TS 16949: Clause 7.6 Calibration status • Can the calibration status, (valid or invalid calibration), easily be determined by the user? (How easily can the user inadvertently use an “out-of- calibration” instrument? Who’s responsibility?) • On what basis is “Valid Calibration” status assigned to an instrument? • Is the calibration expiry date indicated on the calibration label? • If partially calibrated, is there indication to the user to prevent accidental usage for another application? • Is there a unique link between the calibration status information and the instrument to avoid accidental applicability to the wrong instrument? • How will the quality system prevent the use of an “un-calibrated” instrument? • If the calibration was conducted by an accredited calibration service provider, does the SANAS logo appear on the associated documentation and calibration label?

21. ISO/TS 16949: Clause 7.6 SANAS accredited calibration laboratory logos Fig 1: Old Logo Fig 2: New Logo

22. ISO/TS 16949: Clause 7.6 Control of devices

23. ISO/TS 16949: Clause 7.6 Calibration integrity • Is the calibration validity protected in some way? (Can the user accidentally or intentionally interfere with the calibration?) • Who is responsible for ensuring calibration integrity? (Cannot always be the calibration service provider as the user may have access to adjustments) • Which methods of calibration integrity protection are employed? • Integrity seals over adjustment access points or chassis opening points? Password protection – who keeps the password? • If an entire system is calibrated, does the protection apply to the entire system or only one instrument within the system? • If in-house integrity seals are used, can they be accessed and applied by the instrument user? • How is integrity sealing applied when more than one calibration service provider is used to calibrate the full capability of an instrument and one’s adjustment may affect the other’s calibration?

24. ISO/TS 16949: Clause 7.6 cont. Control of devices cont.

25. ISO/TS 16949: Clause 7.6 Evaluating the impact of an “out-of-calibration” instrument • Is there communication from the calibration service provider regarding an “out-of-calibration” instrument and any subsequent adjustment? • Was the error larger than the required uncertainty of the measurement or test? • Was the instrument out of cal at a point directly applicable to the measurement function being used or was it on another range, function? • Has a risk profile of the measurement been drawn up? Will an erroneous measurement have major consequences which cannot be rectified? • What actions have been implemented to reduce the risk such as:- • reducing the calibration interval, • performing in-between-calibration verification checks, • repeating the measurements using more than one instrument

26. ISO/TS 16949: Clause 7.6 cont. Control of devices cont.

27. ISO/TS 16949: Clause 7.6 Calibration records • Have the records of calibration been reported on a calibration certificate? • If only an in-house document is generated are at least the following recorded:- • measurement results, • identity of the metrologist & authorised signatory conferring validity to the calibration, • estimated uncertainty of the measurements, • before and after adjustment results, • unique identity of the instrument, • description of the calibration procedure/method followed, • date & location of the calibration, • measurement standards used, • conformance statement if required

28. ISO/TS 16949: Clause 7.6 Verification records • Does a verification policy exist defining verification intervals and measurement points? • Is verification performed straight after calibration to establish a “signature”? • Are the results of verification measurements recorded and analysed? Do they demonstrate the ongoing ability of the instrument to meet the requirements of its intended purpose? • What action is implemented when the results obtained after verification differ markedly from the previous results? • How and by whom are the verification measurement points chosen?

29. ISO/TS 16949: Clause 7.6 General instrument records • Does each instrument have a history file? • Are details of maintenance work recorded in the file? (In particular any possible invalidation of calibration resulting from maintenance) • Are costs of maintenance and calibrations recorded to track lifecycle costs?

30. ISO/TS 16949: Clause 7.6 cont. Control of devices cont.

31. ISO/TS 16949: Clause 7.6 Software validation • Is there evidence that the software is fit for its intended purpose? (proof that the software produces the correct measurement/test results) • Has this been “validated” by comparing software generated results with results obtained by means of manual measurements? • How is the integrity of the software protected? • can a different version be inadvertently installed over the original version? • is a copy of the original version kept in a safe place? • is a particular version easily uniquely identifiable? (traceability) • if changes are made, are these adequately documented and controlled? (can the user of the software make changes?) • Is the integrity of the entire measurement process controlled including sampling, measurement and result storage?

32. ISO/TS 16949: Clause 7.6.1 Measurement system analysis

33. ISO/TS 16949: Clause 7.6.1 Statistical analysis of measurement result variation • Is the measurement result variability statistically analysed using a method agreed to by the customer • Analysis of “repeatability” and “reproducibility” Definition ofRepeatability “Closeness of agreement between successive measurement results obtained using the same system under the same conditions” Definition ofReproducibility “Closeness of agreement between successive measurement results obtained using the same system under changing conditions”

34. ISO/TS 16949: Clause 7.6.1 Estimation of uncertainty of measurement • Is an uncertainty of measurement estimated for each measurement/test performed and is it recorded with the result? • Is this uncertainty estimated according to the requirements of the ISO document entitled, “Guide to the expression of uncertainty in measurement” or ISO 14253-2, “Guide to the estimation of uncertainty in GPS measurement, in calibration of measuring equipment and in product verification”? • Are decisions of product conformance based on ISO 14253-1, “Decision rules for proving conformance or non-conformance with specifications”?

35. ISO/TS 14253-1 Result of a measurement, y, and result of measurement, complete statement y’

36. ISO/TS 14253-1 Example of conformance proved

37. ISO/TS 14253-1 Example of non - conformance proved

38. ISO/TS 14253-1 Example of neither conformance nor non - conformance proved

39. ISO/TS 14253-1 Uncertainty of measurement reduces conformance range

40. ISO/TS 16949: Clause 7.6.2 Calibration/verification records

41. ISO/TS 16949: Clause 7.6.2 Calibration/verification records • Highlighted points from above:- • Do the calibration records reflect any engineering changes made to the instrument? (engineering changes can take place without the knowledge of the customer) • Does the contract review, between a calibration service provider and the customer, record that special mention must be made in the reporting that an instrument’s performance is suspect for whatever reason?

42. ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

43. ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Laboratory scope • Is there evidence of a laboratory scope listing the methods or tests/measurements which can be performed by the laboratory? • Are the uncertainties, (Best Measurement Capabilities), for these methods or tests/measurements quoted? • Is the technically responsible person, who confers validity to the tests/measurement reports, identified?

44. ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

45. ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Adequacy of laboratory procedures/methods • Are documented test/measurement methods available? • Have they been authorised for use in the laboratory by a person with appropriate signing power? • Have they been declared technically capable of delivering the desired technical results, i.e. validated, by an appropriately technically competent person? (changes made since validation?) • Do they reflect sufficient detail to ensure that the correct results will still be obtained by junior personnel if a senior personnel member is off sick or injured? • Do the methods address the entire measurement process including sampling, performing of the measurements, capturing and storing of results, estimation of uncertainty of measurement as well as reporting the result? • Are methods reviewed for ongoing relevance at planned intervals?

46. ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

47. ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Competence of laboratory personnel • Since the responsibility of assessing whether or not the metrologist is competent is an internal one:- • Have criteria for the required competence been drawn up? • Is there evidence that the metrologist’s competence has been assessed against these criteria? • Was the metrologist’s competence assessed by means of physically witnessing a measurement/test AND comparing the results obtained with results of known accuracy? • To what extent have audit sample measurements, or participation in proficiency testing schemes, provided proof of acceptable measurement capability? • Does the metrologist have a training file containing records of relevant training and experience? • Is there sufficient depth of staff and is a designated “stand-in” identified in the quality system?

48. ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

49. ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Testing of the product • Are suitable test methods available for testing of the product? • Does the laboratory have access to the required technical specifications in order to declare conformance or not? • Are the uncertainties of measurement in the test adequately estimated? • Do decisions of conformance or non-conformance take the estimated uncertainty of measurement into account? • Has an impact analysis been conducted to evaluate the risk of rejecting product which actually passes spec, and accepting product which fails spec?

50. ISO/TS 16949: Clause 7.6 Laboratory requirements.