The Examination for Professional Practice in Psychology EPPP: Separating Myth from Reality

1. The Examination for Professional Practice in Psychology (EPPP):Separating Myth from Reality Dr. Ian R. Nicholson London Health Sciences Centre Dr. Carole Sinclair The Hincks-Dellcrest Treatment Centre Dr. John Hunsley University of Ottawa

2. Primary Aim of the Presentation This conversation hour will attempt to dispel the many myths that have developed over the years about the EPPP by providing accurate information about various aspects of the test.

3. Background of Your Speakers Dr. Ian Nicholson Member, CPA Committee on Ethics Member, ASPPB Exam Development Committee Dr. Carole Sinclair Chair, CPA Committee on Ethics Member, ASPPB EPPP Exam Committee Dr. John Hunsley Previous Member, ASPPB EPPP Exam Committee

4. Myths we have Heard There are easy and hard versions (a.k.a �the beast�) of the exam. (It is unfair if you get the hard version.) A majority of people fail the exam. All candidates, even Canadians, need to know the APA Ethics Code and US laws (e.g., HIPAA) for the exam.

5. Myths (II) The items on the exam are not related to what we do as psychologists. The exam is filled with trivia questions. The exam contains a specific number of Industrial-Organizational questions. The exam is filled with badly worded and unclear / trick questions.

6. Myths (III) There is no validity to the exam. The Association of State and Provincial Psychology Boards prepare the study materials, therefore: It is unfair that the test questions and practice exams differ from the materials. It is unfair when topic areas not in the study guide are on the exam.

7. Other Myths? What have you heard that we haven�t heard?

8. History of the Exam The Association of State and Provincial Psychology Boards (ASPPB) [originally named the American Association of State Psychology Boards] was founded in 1961 with one of its missions to create a national licensure exam. 1n 1965, the AASPB created Form 1 of the Exam. It was administered to 27 candidates in 8 states. Since that time, the EPPP has been a multiple choice exam where candidates are instructed to pick the single best answer to each question.

9. History (II) At first, a new exam was created every 2 years, and the number of questions varied between 150 and 200. By 1978, approximately 4,000 candidates a year were being tested in 53 states and provinces. In the 1980s and 1990s, the exam standard was 200 questions with 2 exam sittings per year (April and October). By 1998, over 122,000 candidates had been tested.

10. History of Exam (III) In 2001, the exam became computerized, and able to be taken at any time during the year. Overall score on the exam moved from the number of items answered correctly (with the most commonly accepted pass point of 140 items or 70%) to scaled scores (with pass point of 500). Exams are now equated.

11. History of Exam (IV) Almost from the beginning of its history, ASPPB has worked with a non-profit test construction company called Professional Examination Service. Located in New York, PES helps construct licensing exams for over 25 disciplines, including psychology.

12. Purpose of the Exam The EPPP is provided to state and provincial boards to assist them in their evaluation of the qualifications of candidates for licensure and certification. The EPPP is intended to evaluate the foundational knowledge that should be acquired by any candidate who is seeking licensure to practice psychology.

13. Purpose (II) Such candidates are expected to have a broad general knowledge of psychology, regardless of their individual areas of entry-level practice. The candidate�s knowledge is assessed through the candidate�s responses to objective, multiple-choice questions representative of the field at large.

14. Current Exam 225 items, of which 25 items are being pretested and are not scored. Candidates have 4 hours and 15 minutes to complete the exam. Administered at Prometric sites across the US and Canada. 2 forms of the exam are rotated into 4 operating forms each year.

15. Current Exam: Content Outline 1) Biological bases of behaviour (11%) 2) Cognitive-affective bases of behaviour (13%) 3) Social and cultural bases of behaviour (12%) 4) Growth and lifespan development (13%) 5) Assessment and diagnosis (14%) 6) Treatment, intervention, and prevention (15%) 7) Research methods and statistics (7%) 8) Ethical, legal, and professional issues (15%)

16. Current Exam: Roles and Responsibilities Covered 1) Psychological services (74%) 2) Consultation, outreach, and policy making (9%) 3) Academic preparation and professional development (10%) 4) Research, evaluation, and scholarship (7%)

17. Development of Test Specifications 1982 � Role delineation study with 2 �blue ribbon� panels that defined practice-relevant domains and roles, and the knowledge required for generic entry level practice. 1983 � Job analysis determined the dimensions of psychological practice among licensed practitioners representing clinical, counselling, I/O, and school psychology.

18. Development of Test Specifications (II) 1996 � Survey of over 3,900 psychologists, who rated knowledge statements (derived from the previous studies) on their importance, criticality, and appropriateness for entry-level practice.

19. Development of Test Specifications (III) 2003 � Update to job analysis completed. Empirical evidence gathered through critical incident interviews, focus panels, and a validation survey that was sent to 1,000 licensed psychologists. On survey, psychologists rated the test components on importance, criticality, frequency, and point of acquisition. Based on results, ASPPB approved content areas and knowledge statements, and revised the weights of two of the 8 content areas.

20. Development of New Items Subject matter experts are asked to participate in an Item Writing Workshop to develop questions in particular area(s) of the examination where questions have been identified as being needed (e.g., insufficient number of items). The subject matter experts write draft items and bring them to the Item Writing Workshop.

21. Development of New Items (II) Each draft item is reviewed and validated by subject matter teams, and evaluated for accuracy, relevance, professional level of mastery, contribution to public protection, freedom from bias, and relevance to both Canada and the US. Additional criteria include: Avoiding items that test rote memory or knowledge of simple definitions. Ensuring items on historical issues test critical thinking. Ensuring items are �entry level.�

22. Development of New Items (IV) Items that pass the above are then edited and brought into compliance with PES style guidelines for EPPP items. The new items then are placed into the EPPP Pretest Item Bank. Once pretested and found to be statistically sound, the item is then moved to the Calibrated Item Bank for possible use in future exams.

23. Exam Construction Items for a draft EPPP exam are chosen by Professional Examination Service (PES) staff psychologist in accordance with established test specifications. The draft exam is reviewed by the ASPPB Examination Committee on an item-by-item basis. Specific items may be removed, with substitute items chosen from the Calibrated Item Bank.

24. Exam Construction (II) Possible pretest items are chosen by Professional Examination Service (PES) staff psychologist The selected pretest items are reviewed by the ASPPB Examination Committee on an item-by-item basis. Pretest items may be revised or edited before being included on new draft exam.

25. Exam Construction (III) The new exam is finalized by the ASPPB Examination Committee, based on expert judgment, item statistics from previous uses, a specific reference for each item, item classification data, and multiple reviews of successive drafts. The ASPPB Examination Committee includes 10 psychologists from across the US and Canada with expertise in various areas of psychology.

26. Reliability Data Reliability generally has been found to be between 0.88 and 0.93. For example, Form 48, 1998: number of candidates -- 2615 mean score -- 144.79 standard deviation -- 20.27 Range -- 58-189 Spearman-Brown Split Half -- 0.92 Kuder-Richardson 20 -- 0.92 --the data on this slide are generally representative of data from current forms of the EPPP --since moving to a computerized format, in addition to traditional reliability information, Item Response Theory (IRT) statistics are used to evaluate the adequacy of exam items --both traditional and IRT statistics are used to inform decisions about selection of items for an exam--the data on this slide are generally representative of data from current forms of the EPPP --since moving to a computerized format, in addition to traditional reliability information, Item Response Theory (IRT) statistics are used to evaluate the adequacy of exam items --both traditional and IRT statistics are used to inform decisions about selection of items for an exam

27. Reliability Data (II) Since moving to a computerized format, use of Item Response Theory (IRT) has been added to check the adequacy of exam items. Both traditional and IRT statistics are used to guide decisions about exam items.

28. Validity Important to remember the purpose of the EPPP (assessment of foundational knowledge) within the broader context of professional registration. In most jurisdictions, the EPPP is only one of several methods used to determine readiness for autonomous practice. Other methods may include academic qualifications, evaluations of supervised practice, letters of reference, jurisprudence exams specific to the jurisdiction, oral exam, etc.

29. Validity Data (II) With respect to the EPPP itself: People with more and better education in psychology obtain higher scores on the EPPP. Patterns of academic professional competence in graduate school correspond to patterns of performance on the EPPP. --in considering the validity of the EPPP, it is important to remember the purpose of the exam within the larger context of professional registration --in most jurisdictions, the EPPP is only one of several components used in determining readiness for autonomous practice (the other components will include elements such as academic qualifications, letters of reference, evaluations of supervisors during a supervised period of practice, and may include elements such as an examination of knowledge of professional ethics & jurisprudence and an oral examination) --all jurisdictions evaluate, in some fashion, both the knowledge and skills required for autonomous practice --the purpose of the EPPP is to provide a standardized evaluation of one�s general psychological knowledge that is relevant to professional practice --more specifically, the EPPP is designed to assess basic knowledge that is necessary and expected for someone at the entry level to the profession--in considering the validity of the EPPP, it is important to remember the purpose of the exam within the larger context of professional registration --in most jurisdictions, the EPPP is only one of several components used in determining readiness for autonomous practice (the other components will include elements such as academic qualifications, letters of reference, evaluations of supervisors during a supervised period of practice, and may include elements such as an examination of knowledge of professional ethics & jurisprudence and an oral examination) --all jurisdictions evaluate, in some fashion, both the knowledge and skills required for autonomous practice --the purpose of the EPPP is to provide a standardized evaluation of one�s general psychological knowledge that is relevant to professional practice --more specifically, the EPPP is designed to assess basic knowledge that is necessary and expected for someone at the entry level to the profession

30. Validity Data (III) Content validity analysis has supported the validity of the EPPP and been used to enhance the test through practice analyses. As with other �high-stakes� examinations for licensure in other professions, predictive validity analysis is not possible as one cannot follow the practice of those who fail the examination.

31. Factors Affecting Scores The data on the next two slides were taken from information given by approximately 4,000 individuals who took the EPPP during 2003-2004. The data suggest that: Although each individual must determine how much preparation is appropriate, there is likely to be a diminishing return beyond 200 hours of study. Taking the EPPP within 3 years of graduation is a good strategy.

32. Factors Affecting Scores: Study Time --the data on this and the next slide were taken from approximately 4,000 individuals who took the EPPP during 2003-2004 --although each individual must determine how much preparation is appropriate, these data suggest that, beyond a certain number of hours of preparation, there is likely to be diminishing return in terms the effects of additional hours of preparation on exam performance --the data on this and the next slide were taken from approximately 4,000 individuals who took the EPPP during 2003-2004 --although each individual must determine how much preparation is appropriate, these data suggest that, beyond a certain number of hours of preparation, there is likely to be diminishing return in terms the effects of additional hours of preparation on exam performance

33. Factors Affecting Scores:Time Since Graduation --clearly these data suggest that taking the EPPP shortly after graduation is a good strategy--clearly these data suggest that taking the EPPP shortly after graduation is a good strategy

34. Canadian EPPP Scores Based on 1997 � 2005 mean scores (pass = 140) for Canadian clinical psychology PhD programs: Mean for Canadian programs: 162.1 University of Victoria: 172.1 (2nd highest) Relatively narrow range of scores These data do not support the concern that the EPPP may not be fair to Canadian candidates. --in Canada, there is a concern among some people preparing for the EPPP that the exam may not be fair to Canadian candidates�the data do not support this concern --it is also important to note that questions on professional ethics make reference to the ethical codes of both the American and Canadian Psychological Associations--in Canada, there is a concern among some people preparing for the EPPP that the exam may not be fair to Canadian candidates�the data do not support this concern --it is also important to note that questions on professional ethics make reference to the ethical codes of both the American and Canadian Psychological Associations

35. Ethics and Laws All EPPP items regarding ethics and jurisprudence are written and selected on the basis of whether or not they can be answered correctly based on a study of either the APA or CPA codes of ethics, or regardless of specific laws in the many jurisdictions of North America. There is no need to study both ethics codes to write the exam, nor to study both US and Canadian law relevant to psychological practice.

36. Study Materials There are a number of commercial companies that offer EPPP study materials, practice tests, and workshops. PES and ASPPB have NO CONNECTION TO THESE COMPANIES. PES and ASPPB do NOT sanction any of them or provide them with items. --the point cannot be made strongly enough�there is no relation between the preparation companies and Professional Examination Service or between the preparation companies and any part of the ASPPB --that being said, the content areas covered in the EPPP are presented on the ASPPB website, so it is hardly surprising that the preparation companies know, in general terms, the areas to address in preparing for the EPPP --some candidates have remarked on the similarity between some EPPP items and items they had in preparation materials --it is important to remember that, prior to taking the EPPP, each candidate agrees not to disclose information about the EPPP items themselves --nevertheless, over the years, some individuals broken the applicable laws by posting their recollections of some EPPP on websites or list serves, and it is possible that some of the items included in the preparation materials have been informed by the posting of such recollections --the point cannot be made strongly enough�there is no relation between the preparation companies and Professional Examination Service or between the preparation companies and any part of the ASPPB --that being said, the content areas covered in the EPPP are presented on the ASPPB website, so it is hardly surprising that the preparation companies know, in general terms, the areas to address in preparing for the EPPP --some candidates have remarked on the similarity between some EPPP items and items they had in preparation materials --it is important to remember that, prior to taking the EPPP, each candidate agrees not to disclose information about the EPPP items themselves --nevertheless, over the years, some individuals broken the applicable laws by posting their recollections of some EPPP on websites or list serves, and it is possible that some of the items included in the preparation materials have been informed by the posting of such recollections

37. Study Materials (II) Candidates sometimes have remarked on the close similarity between some practice test items in the commercial materials and actual items that have appeared on the EPPP exam. Sometimes the similarity is due simply to the EPPP content areas being available on the ASPPB website.

38. Study Materials (III) However, over the years, a few candidates who have taken the EPPP have disclosed items by posting their recollections of them on websites or listserves, thereby making them public. It is important to remember that, prior to taking the EPPP, each candidate agrees not to disclose information about the actual EPPP items. Disclosing items is UNETHICAL.

39. Study Materials (IV) To help candidates prepare for the exam, ASPPB provides on its Web site the following two documents: �Information for Candidates� �Items from Previous Examinations: A study tool for the examination for professional practice in psychology� --perhaps the best way to begin your EPPP preparations is to familiarize yourself with the content areas of the exam, as outlined in documents on the ASPPB website --by doing so, a candidate can then self-evaluate to determine areas of weakness that may require special attention during the preparation process --candidates will also see that the focus of the exam is on foundational knowledge relevant to professional practice --indeed, based on the information provided about the exam, it is clear that over half of the exam will cover foundational aspects of psychology and that the �traditional� practice activities of assessment, treatment, prevention, and consultation will be addressed by a minority of the exam items --the �Items from Previous Examinations� is a great resource, as it provides examples of actual items that were used on a number of exams prior to being �retired�--perhaps the best way to begin your EPPP preparations is to familiarize yourself with the content areas of the exam, as outlined in documents on the ASPPB website --by doing so, a candidate can then self-evaluate to determine areas of weakness that may require special attention during the preparation process --candidates will also see that the focus of the exam is on foundational knowledge relevant to professional practice --indeed, based on the information provided about the exam, it is clear that over half of the exam will cover foundational aspects of psychology and that the �traditional� practice activities of assessment, treatment, prevention, and consultation will be addressed by a minority of the exam items --the �Items from Previous Examinations� is a great resource, as it provides examples of actual items that were used on a number of exams prior to being �retired�

40. Overall Summary Much has been done to ensure that the EPPP: meets industry and legal standards regarding test appropriateness, usage, and fairness; is reliable; and meets accepted criteria for content validity. Test items that appear on the EPPP go through a lengthy process of review and testing before they become scoreable items.

41. Overall Summary (II) Each exam is carefully constructed, with multiple reviews by content experts. The majority of candidates, including the majority of Canadians, pass the exam the first time they take it. ASPPB and PES have nothing to do with the commercial companies that sell test preparation materials. We�re not sure why so many myths have developed!