A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS

1. A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS Dale Curry, Kent State University Basil Qaqish, University of North Carolina-Greensboro

2. The NACP Competencies Organization of Competencies • Domains • Sub-Domains • Categories • Competencies

3. Domains I. Professionalism II. Cultural & Human Diversity III. Applied Human Development IV. Relationship & Communication V. Developmental Practice Methods

4. ExamplefromNACP Competencies Document

5. I. PROFESSIONALISM 6. Advocacy – MEDIUM WEIGHT a. demonstrate knowledge and skills in use of advocacy -SUPERVISOR ASSESSMENTb. access information on the rights of children, youth and families - PORTFOLIO c. describe the rights of children youth and families in relevant setting/s and systems – EXAMINATIONd. advocate for the rights of children, youth, and families in relevant setting/s and systems - PORTFOLIO e. describe safeguards for protection from abuse including institutional abuse - EXAMINATION f. advocate for safeguards for protection from abuse including institutional abuse - SUPERVISOR ASSESSMENT g. ensure that children, youth and family views are heard and considered during the decision making processes which directly affect them -EXAMINATION

6. Situational Judgment Items • Can be used to place competencies in a context of relationships, events, and settings • Can help establish face validity of test if they are based on real case studies collected from the field • Moves beyond the simple recitation of learned knowledge

7. Situational Judgment Items • Case studies of real incidents were collected from child and youth care practitioners in a variety of practice settings • Used as the basis for the exam questions.

8. Example ofSituational Judgment Item

9. Example of Situational Judgment Item Competencies assessed by examination: I.6.c. Describe the rights of children youth and families in relevant setting/s and systems I.6.e. Describe safeguards for protection from abuse including institutional abuse

10. Item 8: Read this case study of an incident and answer the questions at the end: Lisa (a girl in care) became angry during individual counseling session with Social Service Manager (MH) and cursed at the staff member. Lisa was escorted to “off dorm” time out due to disrupting milieu. While in time out, Lisa continued to use profanity toward staff members, urinated on the floor, and said “maybe I should pimp again and make some money…

11. After giving her several expectations and after Lisa refused to calm down, staff member (JT) began to physically escort Lisa to seclusion when Lisa became physically assaultive, hitting staff member. Staff members JT and SR placed Lisa in a therapeutic hold and eventually removed her to seclusion room. By 8:15 P.M., Lisa regained physical control of herself and agreed to a plan to maintain the safety of Lisa and the staff members. Lisa cleaned seclusion room and time out room.

12. Which of the following statements are true? [Circle True or False for each statement] T/F Lisa’s rights were properly protected in this situation. T/F The physical restraint, as explained in this case study was justified and proper. T/F An alternative intervention such as having the staff member (JT) switch off with another staff member could have been more effective.

13. Pilot Testing

14. STUDY GOALS • Administer exam and get feedback regarding face validity & ways to improve • Conduct item analysis • Examine relationship between test scores and supervisory assessment of worker performance (concurrent validity) • Explore differential performance results • Determine a cut score (pass/fail)

15. METHOD • Assessment Work Group developed 100 item situational judgment exam. • Exam was administered to 775 child and youth care workers in 29 sites. • Sites located in six states & two provinces.

16. METHOD • Scores were statistically compared with supervisor’s rating of competence. • Supervisor ratings of worker competence (six items) administered. • Examinee feedback on exam and face validity solicited through questionnaire

17. PROCEDURES • Item analysis • Correlation of exam total score with supervisor composite ratings • Modified Angoff procedure to determine cut point (pass/fail)

18. ITEM ANALYSIS • Reliability analysis • Difficulty analysis • Discrimination analysis • Distracter analysis • Differential Item Analysis (DIF)

19. RELIABILITY ANALYSIS • Excellent internal reliability (Chronbach’s alpha=.90) • 18 items identified for possible deletion or revision

20. DIFFICULTY ANALYSIS • Difficulty scores obtained for each item (acceptable difficulty score range from .10-.80) • Also examined top and bottom 20% • Most items were of moderate difficulty

21. DISCRIMINATION ANALYSIS • Point Bi-serial correlation between the item and total test score • Also compared top 20% with bottom 20% • .15 minimum discrimination score

22. DISTRACTER ANALYSIS Figure 1

23. DIFFERENTIAL ITEM FUNCTIONING (DIF) • Exploring differences for race and gender for those with similar abilities on the test • Use of computer program SIBTEST and Mantel Haenszel procedure • 7 items identified by either SIBTEST or Mantel Haenszel for race and 1 item for gender (must give these items more scrutiny)

24. FACE VALIDITY AND OTHER FEEDBACK FROM EXAMINEES • 90% indicate that the items in the exam accurately assess important aspects of child and youth care work • 90% indicate that the case studies provide realistic samples of child and youth care work • 80% indicate that the exam content is similar to their job duties • 59% indicate that their performance on the exam is an excellent indicator of their job performance (34% neither agreed or disagreed)

25. OTHER FEEDBACK • Test is consistent with real situations-overall a good and fair test • Motivating and thought provoking • Test too long – perhaps give a break or do it in two parts • Not familiar with some of the terminology • Some of the case studies were hard to follow

26. CANADIAN RESULTS • Mean total exam score - 75.7 (U.S. 59.3) • Mean competency rating (composite of 6 items, 5 point scale) – 28.4 (U.S. 26.2) • Mean face validity rating (composite of 4 items, 5 point scale) – 15.4 (no significant difference with U.S. examinees)

27. COMMENTS CANADIAN EXAMINEES • Exam was too long, a break was needed • Well-thought questions with very realistic, engaging scenarios but a mix in style of question type (e.g. true/false) may have been a welcome • The exam did not reflect the cultural aspects of British Columbia • Knew about N.A. Code but only familiar with Ontario Code of Ethics

28. POST PILOT ACTIVITIES • Explored items for possible deletion to improve reliability (18 items) • Examined content of items identified by DIF for gender/racial bias • Examined examinee feedback pertaining to each item and regarding the test in general • Completed process of setting cut-off (passing scores) for exam

29. What remains to be done? • Continue to collect and analyze data to improve the exam and accumulate evidence of the exam’s validity • Continue to develop and research new exam items • Continue to explore reliability and validity of exam in future as it is implemented

30. More Information North American Certification Project www.acycp.org