Auditing to Sufficient Depth

1. Auditing to Sufficient Depth Prepared/Presented by: Ben Tuley SAI Global - Senior Consultant (214) 274-8646 – ben.tuley@saiglobal.com Lloyd (Sonny) Crile – Boeing Supplier Quality (316) 304-5294 – lloyd.l.crile@boeing.com

2. Auditing to Sufficient Depth • Why is this important? • ANAB auditors issued over 75 NCRs in 20 areas in 2005-2007 for not auditing to sufficient depth • Many were repeat NCRs • Undermines the integrity of aerospace audits • Serious concern within aerospace community

3. Auditing to Sufficient Depth • 75 NCRs in these 20 areas

4. Auditing to Sufficient Depth • Why is this important? • OEMs beginning to conduct validation audits (independent checks of suppliers with AS91XX certifications) • OEMs can (& do) complain directly to ANAB regarding unresolved issues related to CB conformance to established aerospace requirements, including not auditing to sufficient depth • ANAB will respond very promptly (i.e. visit CB office within a few days)

5. Auditing to Sufficient Depth • What should auditors do? • Understand the requirements to which the audit must be conducted • Research/get training in topics they do not understand • Understand the processes and interactions and involvement of all stakeholders • Understand what objective evidence is required for an auditee to show conformance with a requirement • Accomplish fair, unbiased, detailed audits to the established criteria • Avoid “soft” auditing or “glazing over” topics

6. Auditing to Sufficient Depth • What will we address in this session? • Document control (including configuration management) • Procurement (including purchasing and supplier control) • Product acceptance software

7. Auditing to Sufficient Depth Document Control and Configuration Management Requirements (13 ANAB NCRs in this area in the past three years, primarily in configuration management)

8. Auditing to Sufficient DepthDocument Control 4.2.1 General: The quality management system documentation shall include… f. quality system requirements imposed by the applicable regulatory authorities

9. Auditing to Sufficient DepthControl of Records 4.2.4 Control of Records • The documented procedure shall define the method for controlling records that are created and/or retained by suppliers. • Records shall be available for review by customers and regulatory authorities in accordance with contract or regulatory requirements.

10. Auditing to Sufficient DepthControl of Documents and Records Questions on Document and Record Control?

11. Auditing to Sufficient DepthConfiguration Management 4.3 Configuration Management: The organization shall establish, document and maintain a configuration management process appropriate to the product. Note: Guidance on configuration management is given in ISO 10007

12. Auditing to Sufficient DepthConfiguration Management What is configuration management? Per ISO 10007: “configuration: interrelated functional and physical characteristics of a product defined product configuration information” “configuration management: coordinated activities to direct and control configuration”

13. Auditing to Sufficient DepthConfiguration Management What is configuration management? Working level definition: The cumulative actions an organization takes to ensure that the exact status of a unit or component is known at any given time and to ensure that the status is as planned Status refers to the makeup of the unit or component (i.e. which revision is being built?; what pieces or items are in the unit or component?)

14. Auditing to Sufficient DepthConfiguration Management 4.2.3 Document Control vs. 4.3 Configuration Management: Document control typically addresses controlling procedures, work instructions, contracts and other such documents Configuration management typically addresses controlling a product print or drawing and items that are used to build the product

15. Auditing to Sufficient DepthConfiguration Management • While we will address configuration management in some detail, ISO 10007 should be available at your registrar • Per AQMS Standards, configuration management process must be documented (may change in next rev) • Configuration management will move to clause 7.1 in next revision

16. Auditing to Sufficient DepthConfiguration Management Configuration management actions: Technical and organizational activities comprising: • configuration identification • configuration control • configuration status accounting • configuration auditing (next rev adds configuration planning)

17. Auditing to Sufficient DepthConfiguration Management Configuration Planning • Develop a plan • Describe CM actions, responsibilities and authorities throughout product life cycle Configuration Identification • Identify configuration items • Develop a baseline Configuration Control Define a change methodology Define the information storage procedure

18. Auditing to Sufficient DepthConfiguration Management Configuration Status Accounting • Gather & record data on a build • Identify deviations Configuration Audits • Functional Configuration Audit • Verifies that a CI meets performance and functional requirements cited in the configuration documentation • Physical Configuration Audit • Verifies that the as-built unit conforms to product configuration documentation

19. Auditing to Sufficient DepthConfiguration Management Some points an auditor should investigate: • Is the process defined/documented? • Has a CM plan been developed? • Is there a methodology to accomplish CSA? • Are the steps being accomplished? • How are the audits being documented? • Are audit findings being acted upon? • Have roles and responsibilities been defined? • What happens when CM is not maintained?

20. Auditing to Sufficient DepthConfiguration Management Some points an auditor should investigate (con’t): • How does configuration management ripple throughout the organization (i.e. purchasing, production, repair)? • What impact does a change in supplier or source of a material/component have on CM activities? • Is the CM system comprehensive enough for the product and/or contract flow down requirements? • How are drawings controlled, both while in engineering and after release to the program? • How do engineering changes interface with the configuration management process? • At what point does the CM function take control of the configuration from the design function? • How does this transfer take place?

21. Auditing to Sufficient DepthConfiguration Management Other questions on configuration management?

22. Auditing to Sufficient DepthProcurement Addresses supplier control, purchasing and verification of purchased product (10 ANAB NCRs in this area in the last three years)

23. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1: The organization shall evaluate and select suppliers based on their ability to supply product in accordance with the organization’s requirements. Criteria for selection, evaluation and re-evaluation shall be established. Does a supplier having an ISO 9001 or AS9100 certification automatically satisfy this requirement?

24. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1: The organization shall be responsible for the quality of all products purchased from suppliers, including customer-designated sources. • What should an organization do if a customer directed supplier is not performing as needed? • How should an organization proceed if a customer directs use of a supplier that the organization would not otherwise use?

25. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1: The organization shall: • Maintain a register of approved suppliers that includes the scope of approval How is scope of approval determined? • An example

26. Auditing to Sufficient DepthProcurement - Supplier Control To identify scope, we divide our suppliers by commodities. The commodities we use are:

27. Auditing to Sufficient DepthProcurement - Supplier Control Hardware and Software

28. Auditing to Sufficient DepthProcurement - Supplier Control How can scope of approval be cited? • Specific part numbers • Unique item numbers • Commodities • Let’s look at commodities

29. Auditing to Sufficient DepthProcurement - Supplier Control Commodity List AC – Adhesives and Coatings CB - Circuit Boards CH – Chemicals CS – Consumables DI – Distributor ML - Metals MP – Manufactured Parts PL – Plastics TP – Turned Parts Is this list sufficient?

30. Auditing to Sufficient DepthProcurement - Supplier Control Commodity List AC – Adhesives and Coatings Adhesives – pressure sensitive adhesive, composite adhesive Coatings- metal coatings, optical coatings CH – Chemicals Dye, photo resist, cleaning solvents CS – Consumables What does this mean? DI – Distributor How are distributors handled? ML – Metals Round stock, bar stock, sheet stock, aluminum, steel, gold, platinum MP – Manufactured Parts Lathe, extrusion, mold Is this list sufficient?

31. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1 b Periodically review supplier performance; records of these reviews shall be used as a basis for establishing the level of controls to be implemented How does this apply to customer directed suppliers? What are some examples of controls used for suppliers?

32. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1 c Define the necessary actions to take when dealing with suppliers that do not meet requirements How does this apply to customer directed suppliers? What are some examples of actions that can be taken?

33. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1 d Ensure where required that both the organization and all suppliers use customer-approved special process sources How can we tell when this is required? How can this requirement be met without flowing it down to suppliers?

34. Auditing to Sufficient DepthProcurement - Supplier Control 7.4.1 e Ensure that the function having responsibility for approving supplier quality systems has the authority to disapprove the use of sources What must be in place before we can ensure this requirement is met?

35. Auditing to Sufficient DepthProcurement - Purchasing 7.4.2 d. Include positive identification of product, applicable specs, drawings, process requirements, inspection instructions & other relevant technical data e. Include requirements for design, test, examination and related instructions for organizational acceptance f. Include requirements for test specimens (production method, number, storage conditions) for design approval, inspection, investigation or audit g. Include NCM (nonconforming material) notification requirements (supplier notification to organization and arrangements for organization approval of supplier NMC)

36. Auditing to Sufficient DepthProcurement - Purchasing 7.4.2 h. Include supplier notification to organization of product or process definition change and obtain organizational approval if required i. Include right of access by organization, customer, regulatory authorities to all involved facilities and applicable records j. Include supplier requirements flow down to sub-tier - applicable requirements, key characteristics

37. Auditing to Sufficient DepthProcurement – Verifying Purchased Product 7.4.3 Verification of purchased product • Positive verification must occur before supplied product is used - or release under positive recall • When verification is delegated to the supplier, requirements will be defined and a register of delegations maintained • Customer access to suppler & organization’s premises when required to verify subcontracted product meets requirements • Customer verification is not evidence of good supplier control, does not relieve organization’s responsibility for supplying acceptable product & does not preclude customer rejection

38. Auditing to Sufficient DepthProcurement Questions or Comments on Procurement?

39. Auditing to Sufficient DepthSummary • What should auditors do to be effective? • Understand the requirements to which the audit must be conducted • Research/get training in topics they do not understand • Understand the processes and interactions and involvement of all stakeholders • Understand what objective evidence is required for an auditee to show conformance with a requirement • Accomplish fair, unbiased, detailed audits to the established criteria • Avoid “soft” auditing or “glazing over” topics • Continue to gather information/data until a conclusion can be reached • Review their actions and results to ensure effectiveness

40. Auditing to Sufficient DepthSummary • Existence • Adequacy • Conformance • Effectiveness

41. Auditing to Sufficient DepthProduct Acceptance Software Product Acceptance Software

42. CMSC Overview Product Acceptance Software (PAS) and Quality System Requirements - Taking Ownership of One's PAS  

43. Agenda • Product Acceptance Software Defined • Processes Affected • PAS Requirements/Guidelines • Mathematical Computation • Validation Methods • Validation Summary • What can you do and why • Questions

44. Product Acceptance Software Defined • What is Product Acceptance Software (PAS) • PAS is considered software that performs product or tooling acceptance without subsequent inspection • Software used to manage, store, or translate the sole-source authority copies of digital type design, and is able to impact the integrity of the type design • PAS may be commercial off the shelf software

45. Processes Affected • Common PAS applications include • CMM software • PCMS applications (Laser Trackers, Laser Radar, Laser Scanners, Photogrammetry, PCMM’s) • Optical Laser Templates • CAD data translators • CAD Analysis Software • Post processors

46. PAS Requirements • FAA AC21-36 (1993) Quality assurance controls for product acceptance software. (FAA Req’mnt) • AS9100 Quality management systems – Aerospace requirements • Section 7.6 • D6-51991 Quality Assurance Standard for Digital Product Definition at Boeing Suppliers • Section3.0

47. PAS Guidelines • ISO 10012 Requirements for measurement processes and measuring equipment (Guidelines) • Section 6.2.2 • ASME B89.4.10 Methods for performance evaluation of coordinate measuring system software (Guidelines) • ARP9005 Aerospace Guidance for Non-Deliverable Software

48. PAS Requirements • FAA AC21-36 (1993) Quality assurance controls for product acceptance software. (FAA Req’mnt) • Provides information and guidance concerning control of software • “This AC addresses only those sections of FAR Part 21, Subparts F,G,X and O where information on CAM/CAI/CAT software would be helpful”.

49. PAS Requirements • AS 9100_rev B_Sect 7.6f – When used in the monitoring and measurement of specified requirements, the ability of computer software to satisfy the intended application shall be confirmed. This shall be undertaken prior to initial use and reconfirmed as necessary. NOTE: See ISO 10012-1 and ISO 10012-2 for guidance. (Replaced by ISO 10012 rev 2003)

50. PAS Requirements • D6-51991 section 3.1.4 (Boeing) • Supplier PAS must be verified prior to product acceptance use. • The supplier will establish and maintain a procedure independent of the software developer to determine that the software, and subsequent revisions, accomplishes its intended function.