Practical Session on Metabolites Part II: Goodness of Fit and Decision Making

1. Practical session metabolites Part II: goodness of fit and decision making Claude Beigel, PhD. Exposure Assessment Senior Scientist Research Triangle Park, USA

2. Recommended Tools for Assessing Goodness of Fit • Same recommended approach as for parent substance • Combination of visual assessment and statistical tests • Visual assessment, although bringing some level of subjectivity, is necessary to discern between normal data variability (scattering) and systematic model deviation, this is not done by the statistical tests

3. Recommended Tools for Assessing Goodness of FitVisual Assessment • Visual check of model description of measured data and distribution of residuals (plot of residuals, Predicted - Observed) • Systematic deviation indicates kinetic model may not be appropriate (unless deviation can be attributed to experimental artifacts) Example 8.2 of report, SFO-SFO fit

4. Recommended Tools for Assessing Goodness of FitVisual Assessment • Residuals should be randomly distributed on vertical axis Example 8.2 of report, SFO-FOMC fit

5. Recommended Tools for Assessing Goodness of FitStatistical Indices • Chi2 (2) statistical test • Minimum error percentage to pass 2 test at a 5% significance level • Needs to be performed for each substance individually, to avoid that good fit of the main substances (parent and/or major metabolite) overshadows goodness of fit of more minor substances (weighting issue) • Calculated from fitted versus observed substance data (use of average values for replicates is recommended) • Degrees of freedom for the substance defined as number of substance data points used in 2 test minus number of estimated parameters for the substance • Do not count replicates if averages used, excludes data points set to 0 (metabolite at time-0) or not counted (<LOD/LOQ) • Metabolite parameters defined as metabolite formation fraction and degradation rate parameters (dependent of kinetic model used)

6. Recommended Tools for Assessing Goodness of FitStatistical Indices • One-sided t-test for evaluating uncertainty of rate constant parameters • To determine whether rate is significantly different from 0 • If p < 0.05, parameter is considered significantly different than zero • If p between 0.05 and 0.1, weight of evidence should be considered • Especially important for metabolites that do not show a clear decline • Because parameters in parent + metabolite fits (formation and degradation parameters) can be highly correlated, the t-test is performed at final step (all parameters fitted together) • Degrees of freedom defined as number of data points (including replicates) minus number of fitted parameters

7. Recommended Tools for Assessing Goodness of FitData Handling / Methodology • Basic data handling • Paste ModelMaker output (integration table) in Excel spreadsheet • Extract fitted values corresponding to measured times for each substance • Average replicates if necessary (an automated Excel spreadsheet may be created for that purpose, but not available yet) • Minimum 2 error % for metabolites may be calculated using Parent degradation kinetics.xls file • Paste measured Vs. fitted values in Chi2 all models worksheet, update number of parameters cell and click calculate

8. Recommended Tools for Assessing Goodness of FitData Handling / Methodology • Residuals may be plotted in Parent degradation kinetics.xls file • Valid only for 1- or 2-replicate data sets (if more, needs to be done manually) • Paste measured Vs. fitted values (all replicates) in SFO no-reps or SFO 2-reps worksheet • t-test for rate constant parameters may be performed using provided t-test.xls file • For each rate constant parameter, enter parameter estimate and standard error, number of data points and number of parameters estimated

9. Hands-on Example 1 • Exercise 1 • Open ModelMaker file for example 1 • From result table, extract fitted value for each sampling time, write down in output tables for parent, metabolite1 and metabolite2 • Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically

10. Hands-on Example 1 • Visual assessment • Check ModelMaker plot of fit and answer following questions for each substance • Does fitted line adequately describe data, are there obvious over- or under-predictions (including day-0) • Plot residuals for each substance and answer following questions • Do residuals show distinct pattern, are most of the points above or below 0-line, what is the magnitude? • Statistical indices • Calculate minimum 2 error percentage for each substance • Perform t-test for all rate constant parameters (parent and metabolites) and record P-value and conclusion

11. Hands-on Example 1Visual Assessment

12. Hands-on Example 1Statistical Indices

13. Hands-on Example 2, Parent FOMC • Exercise 2 • Open ModelMaker file for example 2, parent FOMC • From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite • Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically • Perform visual assessment and calculate statistical indices

14. Hands-on Example 2, parent FOMCVisual Assessment

15. Hands-on Example 2, parent FOMCStatistical Indices

16. Hands-on Example 2, parent DFOP • Exercise 3 • Open ModelMaker file for example 2, parent DFOP • From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite • Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically • Perform visual assessment and calculate statistical indices

17. Hands-on Example 2, parent DFOPVisual Assessment

18. Hands-on Example 2, parent DFOPStatistical Indices

19. Hands-on Example 2, Metabolite Decline • Exercise 4 • Open ModelMaker file for example 2, metabolite decline • From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite • Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically • Perform visual assessment and calculate statistical indices

20. Hands-on Example 2, Metabolite DeclineVisual & Statistical Assessment

21. Decision MakingTrigger Endpoints • Based on visual assessment and statistical indices, is SFO model acceptable for the metabolite (in combination with best-fit model for parent)? • Yes  use SFO DT50/90 endpoints • Noand clear decline of metabolite, use FOMC model for metabolite (in combination with best-fit model for parent) • If FOMC acceptable based on visual assessment and statistical indices, use FOMC DT50/90 endpoints • If not, model decline of metabolite with best-fit model and use decline DT50/90 as conservative endpoints • No and no apparent decline of metabolite • Assess relevance of study with regard to metabolite • Check other studies • Study with metabolite may be needed

22. Decision MakingModeling Endpoints • Based on visual assessment and statistical indices, is SFO model acceptable for the metabolite (in combination with appropriate model for parent)? • Yes  use modeling endpoints for metabolite • No and clear decline • If formation fraction estimate is reliable, use with decline rate constant as conservative endpoints • If not, use formation fraction of 1 with decline rate constant as conservative endpoints • If metabolite biphasic, use appropriate higher-Tier approach (e.g. DFOP, PEARL) • If terminal metabolite and biphasic, use FOMC DT90/3.32 as half-life • No and no apparent decline of metabolite • Assess relevance of study with regard to metabolite • Check other studies • Study with metabolite may be needed

23. Hands-on Examples • Determine appropriate trigger and modeling endpoints for Example 1 metabolites 1 and 2 and Example 2 metabolite