Solomon Iyasu, M.D., M.P.H. Medical Team Leader DPDD/OCTAP/CDER Food and Drug Administration

1. Audit of the Columbia Suicidality Classification Project September 13, 2004Meeting of Psychopharmacological Drugs Advisory Committee andPediatric Advisory Committee Solomon Iyasu, M.D., M.P.H.Medical Team Leader DPDD/OCTAP/CDERFood and Drug Administration

2. Outline • Background • Objectives of the FDA audit • FDA audit method, process • Results of the audit • Limitations/strengths • Conclusion

3. Objective of the FDA Audit • Assess the reproducibility and reliability of the Columbia suicidality classification methodology and process

4. The FDA Audit Methods and Process:Event Narratives • Columbia reviewed and classified ALL sponsor-submitted event narratives • FDA audit team received a computerized line listing of these event narratives from the Div. Neuro-pharmacological Drug Products • Drew a sample of events for review

5. The FDA Audit Methods and Process • Grouped event narratives into 4 predefined strata • Selected event narratives for review via a stratified simple random sampling strategy • Over-sampled difficult to classify and reclassified events

6. The FDA Audit Methods and Process:Strata and Definitions • Stratum 1: Events reclassified by Columbia to non-suicidal or other events (n=2) • Stratum 2: Events newly identified and classified by Columbia as possibly suicide related or other categories (n=29) • Stratum 3: Events that were difficult to classify defined as events with discordant initial independent ratings by Columbia reviewers (n=56) • Stratum 4: Events that are straight forward cases defined as events with concordant initial ratings by Columbia reviewers (n=336)

7. The FDA Audit Methods and Process:Sample Size and Selection • Sample size: 64 of the 423 event narratives selected for review (15%) • Stratum 1: All selected ( n=2) • Stratum 2: 1/3rd selected (n=10)* • Stratum 3: 1/3rd selected (n=19) • Stratum 4: 1/10th selected (n=33) • Events from stratum 2 – 4 were selected by a simple random technique using a random number table * Four of the sampled records were also ‘difficult to classify”

8. Planning Group Div. of Pediatric Drug Development, OCTAP, CDER Solomon Iyasu, MD, M.P.H., Audit Team Leader Susan Cummins, M.D., M.P.H. Rosemary Addy, Project Manager Kristin Phucas, Project Manager Div. of Neuro-pharmacological Drug Products, CDER Thomas Laughren, M.D. OND Armando Oliva, M.D. Clinical Reviewers Div. of Pediatric Drug Development, OCTAP, CDER Hari Sachs, M..D., (Pediatrician) ShaAvhree Buckman, M..D., Ph.D. (Pediatrician) Div. of Neuro-pharmacological Drug Products, CDER Cara Alfaro, PharmD. (Pharmacist) Robert Levin, M..D. (Psychiatrist) Consensus meeting facilitator Div of Scientific Investigations Robert Stasko, M.D. (psychiatrist), Composition of the FDA Audit Team None of the clinical reviewers had previously reviewed the event narratives

9. FDA Audit Team Training • FDA Audit Team received a 2-hour teleconference training by Kelly Posner about the suicidality classification scale, method and process including • A review of the classification scale, categories and construct • A review of examples of case narratives for each of the classification categories • Independent classification of test event narratives by participants to evaluate the training

10. The FDA Audit Methods and Process:Review Assignments • Each sampled event randomly assigned to three of four reviewers for independent and blinded review • Each reviewed and rated 48 events (total of 192 reviews) • Reviewers blinded to treatment assignment, sponsor, diagnosis and to the Columbia ratings

11. The FDA Audit Methods and ProcessReview Procedures/Instructions • A memo outlining the procedures of the audit was prepared and provided to the audit team • Reviewers were not allowed to discuss the events among themselves or with colleagues during the independent review • Allowed reviewers to call Kelly Posner, Columbia University to obtain clarification on the classification scale but no discussion of the specifics of any case was allowed • Required reviewers to record on rating form if they consulted with Columbia during the review process • Modified and pre-coded rating form used

13. The FDA Audit Methods and Process • Reviewers returned completed rating forms in sealed envelopes • Ratings double key entered into an Excel database • Discordant ratings identified and discussed during a consensus meeting facilitated • by a board certified child and adolescent psychiatrist external to DNDP • previously not involved in the review of these records or had knowledge of the Columbia ratings score • Final consensus ratings entered into an Excel database and compared to the final Columbia ratings. • Discordant ratings between FDA and Columbia discussed with Columbia to better understand the reason for the differences

14. Criteria for Evaluation of Concordant Ratings • Columbia Definition of Concordance • Categories 1, 2, 3, 6, and 10 required exact match • Categories 4, 5 or 11 all describe self-injurious behavior with no suicidal intent and considered equivalent rating • Categories 7, 8, 9 or 12 all describe “other” (non-suicidal and non-self-injurious behavior) and are considered equivalent rating

15. Results:Concordance among FDA reviewers • 47 of 64 initial ratings were concordant • 17 of 64 initial ratings were discordant • Event narratives discussed during the FDA reviewer’s consensus meeting • Final consensus ratings reached for all 17 events

16. Results:Comparison of Final FDA and Columbia Ratings • Concordance on 57 of 64 final event ratings • Percent agreement of 89% ( k=0.84) • Discordance analysis (n=7) • Severity Hierarchy • (1 or 2) > 6 > 3 > (4 or 5 or 11) > 10 • Compared to Columbia, the FDA audit team classified • six events with higher severity (3 of these events were classified as ‘not enough information’ vs. ‘other’) • one event with lower severity

17. Limitations • Neither the quality of the narratives nor the clinical source material for the narratives were evaluated • Validity of the Columbia classification method not assessed • there is no gold standard

18. Strengths • Achieved a high level of concordance between the two independent review teams despite • Differences in expertise and experience • Short FDA timeline for training and review • Intentional over-sampling of difficult to classify events for review

19. Conclusion • The Columbia Suicidality classification methodology is robust and reproducible.

20. Acknowledgments • OCTAP/CDER: Hari Sachs, ShaAvhree Buckman, Rosemary Addy, Kristin Phucas, Susan Cummins • DNDP/CDER: Robert Levin, Cara Alfaro, Thomas Laughren, Alice Hughes, Tarek Hammad • DSI: Robert Stasko • OND: Armando Oliva • Columbia University: Kelly Posner

21. Thank you!