Risk and Credibility Assessments for Computational Modeling of Medical Devices

1. Risk and Credibility Assessments for Computational Modeling of Medical Devices Tina Morrison, PhD tina.morrison@fda.hhs.gov Advisor of Computational Modeling Office of Device Evaluation, FDA Vice Chair ASME V&V40 Subcommittee Member MDIC CM&S Steering Committee

2. If computational models are to be increasingly relied upon in the development and evaluation of medical devices, the consistent application of V&V must be applied to establish model credibility. Need to establish … If the model is correct and credible Demonstrated predictive capabilities to justify use beyond domain of validation Predictive confidence is commensurate with model risk Role of V&V for Computational Models of Medical Devices

3. ASME Subcommittee on V&V • Standards Subcommittee • Provide procedures for assessing and quantifying the accuracy and credibility of computational modeling and simulation

4. Verification & Validation in Computational Modeling of Medical Devices • V&V-40 Charter • Provide procedures to standardize verification and validation for computational modeling of medical devices • Charter approved in January 2011 • Medical device focus • Regulated industry with limited ability to validate clinically • Want increased emphasis on modeling to support device safety and/or efficacy • Use of modeling is hindered by lack of V&V guidance and expectations within medical device community

5. Guide for Verification and Validation for Computational Models of Medical Devices • Regulated industry with limited ability to validate clinically • Want increased emphasis on modeling to support device safety and/or efficacy • Use of modeling is hindered by lack of V&V guidance and expectations within medical device community • Focus of the Guide • Instead of focusing on how to perform V&V (established elsewhere) … • We developed a common V&V framework to standardize definitions, processes, and documentation requirements between industry, researchers, software developers and regulators.

6. Overall V&V Flow Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Define COU Purpose Is the plan achievable? NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. YES Execute pre-defined M&S and V&V plan Is the CM&S Credible for COU? NO YES Document M&S and VV Plan and Findings

7. Overall V&V Flow Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Define COU Purpose Risk Assessment Matrix Is the plan achievable? NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. YES Execute pre-defined M&S and V&V plan Is the CM&S Credible for COU? NO YES Document M&S and VV Plan and Findings

8. Risk Assessment Matrix (RAM) • Establish Context of Use • Model Risk: combination of decision influence and consequence • Decision Influence: contribution of the model outcome to the decision being made • Consequence: impact if the model outcomes prove incorrect • Model risk assessment • Directs/guides V&V activities • Defines model credibility requirements HIGH MEDIUM INFLUENCE LOW CONSEQUENCE

9. Overall V&V Flow Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Define COU Purpose Credibility Assessment Matrix Is the plan achievable? NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. YES Execute pre-defined M&S and V&V plan Is the CM&S Credible for COU? NO YES Document M&S and VV Plan and Findings

10. Credibility Assessment Matrix (CAM)

11. Credibility Assessment Matrix (CAM) ● ● ● ● ● ● ● ● ● ● ● ● ● Establish Target Credibility Requirements based on the Context of Use

12. Overall V&V Flow Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Define COU Purpose Is the plan achievable? NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. YES Execute pre-defined M&S and V&V plan Is the CM&S Credible for COU? NO YES Document M&S and VV Plan and Findings

13. Credibility Level Determination √ √ √ √ √ √ √ √ √ √ √ √ √

14. Example Jeff Bischoff Mehul Dharia Zimmer, Inc.

15. Force on tibial spine of a knee implant Posterior Tibial Spine Force in Deep Flexion May Create Posterior Liftoff Anterior Tibial Spine Force in Hyperextension May Create Anterior Liftoff Locking mechanism between the (metal) tibial tray and (polyethylene) articular surface is intended to prevent disassembly (poly lift-off) of the modular tibial component during activities of daily living

16. Anterior Lift-off Test Context of use of a test for anterior lift-off Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than expected physiological loading, and therefore demonstrate that the new (locking mechanism) design sufficiently mitigates that risk.

17. Contexts of use of a model for anterior lift-off: • Determine the size of component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test relative to a predicate. FEA followed by Physical Test, Comparison to Predicate • Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than that required for a clinically successful predicate, and therefore demonstrate that the new (locking mechanism and/or geometry) design sufficiently mitigates that risk. FEA only, Comparison to Predicate • Determine the size of a component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test without reference to predicate device. FEA followed by Physical Test, No Predicate • Demonstrate through analysis alone that the worst case size can sustain physiological loading without liftoff, without reference to a predicate device. FEA only, No Predicate

18. Risk Assessment Matrix Model influence LOW: Results from the computational model are a negligible factor in the decision associated with the question being answered. MEDIUM: Results from the computational model are a moderate factor in the decision associated with the question being answered. HIGH: Results from the computational model are a significant factor in the decision associated with the question being answered. Patient consequence LOW: A poor decision would not adversely affect patient safety or health, but might result in nuisance to the physician or has other negligible impacts. MEDIUM: A poor decision would result in minor patient injury and potentially requiring physician intervention or has other moderate impacts. HIGH: A poor decision would result in severe patient injury or death or has other significant impacts.

19. Risk Assessment Matrix

20. Risk Assessment Matrix COU4 COU1: Worst case determination COU2: Absolute evaluation COU3 COU1,2

21. CAM – Elements of Computational Models Verification • Code (Column B) – 4 • Used commercially available validated FEA software • Solution (Column C) – 4 • Mesh convergence study was performed • Numerical effects are determined to be small on all important quantity of interests at conditions/ geometries directly relevant to the context of use • All inputs and outputs based on independently reputable source

22. CAM – Elements of Computational Models Validation: Computational Model • System Configuration (Column D) – 2 • Used mean/nominal geometry (no LMC/MMC) • Major and minor features captured • Two sizes considered • Governing Equations (Column E) – 4 • Used nonlinear material (constitutive) model for UHMWPE • Key physics (press-fit, resistance against force) was captured • Material model did not need re-calibration/tuning • System Properties (Column F) – 1 • Nominal physical properties that are representative of the comparator from literature • Sensitivity analysis on material properties was not performed • Boundary Conditions (Column G) – 3 • Load applied through assumed contact patch on spine, rather than directly modeling the femoral component - Representative but simplified BCs with non-quantified effect on QOI

23. CAM – How Well Is The Comparator Understood? Validation: Evidence-Based Comparator • System Configuration (Column H) – 3 • Prescribed location • Geometries matched to machine tolerance (production parts) • Signal to noise ratio is high • System Properties (Column I) – 3 • Off-the-shelf parts were tested • Environmental effects on the material are known (testing speed was modified, environment was kept the same for both groups: in air). • Boundary Conditions (Column J) – 3 • No sensitivity analysis was performed. • Known (recorded) loading (perturbations) was applied and boundary condition variability (e.g. posterior slope) is known. • Sample Size (Column K) – 3 • Statistically relevant sample size (n = 5) • Component size, a key parameter for lift-off, variation was considered.

24. CAM – How Appropriate is CM to Comparator? Validation: Model-to-Comparator • Discrepancy (Column L) – 4 • Equivalent input parameters, equivalent quantity of interest

25. CAM – How Appropriate is CM to Comparator? Validation: Model-to-Comparator • Discrepancy (Column L) – 4 • Equivalent input parameters, equivalent quantity of interest CAM – How Rigorously Are Outputs Compared? Validation: Qualitative or Quantitative • Comparison (Column M) – 3 • Quantitative comparison, with single set of input parameters, without predictive accuracy or uncertainties available • No quantitative comparison with broad range of cases

26. CAM – How V&V activities relates to COU? Validation: V&V to COU • Applicability (Column N) – 3 • Validation activities embody relevant characteristics of the CoU sufficient overlap between the validation domain and the CoU space)

27. What can we conclude? COU4 COU3 COU1,2

28. Overall V&V Flow Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Define COU Purpose Is the plan achievable? NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. YES Execute pre-defined M&S and V&V plan Is the CM&S Credible for COU? NO YES Document M&S and VV Plan and Findings

29. Public Meeting - FDA/NIH/NSF Workshop on Computer Models and Validation for Medical Devices, June 11-12, 2013 http://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm346375.htm Additional resources on RAM and CAM

30. For More Information Please Contact: • Tina Morrison, PhD tina.morrison@fda.hhs.gov • Advisor of Computational Modeling • Office of Device Evaluation, FDA • OR • Michael Liebschner, PhD Liebschner@bcm.edu • Pre-ORS Symposium Chair • Baylor College of Medicine; Exponent Failure Analysis