BEI Certification Overview Lauri Metcalf

1. BEI Certification Overview Lauri Metcalf L. Metcalf 2017 Copy only with permission

2. How it all began! • In 1994, Texas Commission for the Deaf and Hard of Hearing and the Board for Evaluation of Interpreters brought in for the first time, outside experts in the field of Sign Language Interpreting and certification. • From 1994 to 2001, the BEI worked with numerous people including Daniel Burch, Gary Sanderson, Kathy Friedenreich and Dr. Carol Patrie to establish the need for a new BEI certification exam. L. Metcalf 2017 Copy only with permission

3. BEI Test Development Activities • Spring 2001 - The Texas Commission for the Deaf and Hard of Hearing entered into a contract with the University of Arizona’s National Center for Interpretation to develop new testing materials for the agency’s interpreter certification program. • In June, 2001 - The first test development meeting was held in Austin to discuss the process and the initial work on the job analysis began. • Spring of 2001 to Summer of 2006 – The BEI Panel of Experts worked closely with the University of Arizona’s National Center for Interpretation to develop the new BEI Certification Exam. l metcalf 2013 Copy only with permission L. Metcalf 2017 Copy only with permission

4. Test Development Activities, Cont. • January 2007 – DHHS began administering new testing system. • March 2009 – DHHS entered into Licensing Agreement with Michigan. • January- September 2009 - Development of parallel scripts and final approval of scripts. • April 2010 DHHS entered into Licensing Agreement with Illinois. • July 2014 DHHS entered into Licensing Agreement with Missouri. L. Metcalf 2017 Copy only with permission

5. Test Development Activities Cont. • April 2010 – DHHS entered into Licensing Agreement with Illinois to establish Illinois BEI • June 2010 - Illinois BEI Rater Training • July 2014 - Missouri BEI Rater Training L. Metcalf 2017 Copy only with permission

6. University of ArizonaNational Center for Interpretation • Dr. Roseann Gonzalez was the key expert in the development of the Federal Court Interpreter Certification Exam • This exam has served as a model for high-stakes spoken language interpreting proficiency exams • This is the model that the BEI adopted for the Sign Language Interpreting Exam L. Metcalf 2017 Copy only with permission

7. University of ArizonaNational Center for Interpretation • Their examination process conforms to universally accepted standards of constructing, administering and evaluation of employment tests as prescribed by: • American Education Research Association • American Psychological Association • National Council on Measurement in Education (Standards of Educational and Psychological Testing) L. Metcalf 2017 Copy only with permission

8. University of ArizonaNational Center for Interpretation • From all the experts we learned how important it was to develop an evaluation that was valid, reliable, and legally defensible. L. Metcalf 2017 Copy only with permission

9. The purpose of the BEI Assessment is to answer an important question. • Does this person have the requisite interpreting skills to carry out the responsibilities of a BEI-certified interpreter at this level? • When we certify candidates, we are saying that at this point in time they possess the minimal level of proficiency required at their level of certification – not the maximum or even average level of ability when compared to their peers. L. Metcalf 2017 Copy only with permission

10. Overarching Goals of the BEI Interpreter Certification Process • Ensure that consumers have equal access to rights, services, and education. • Produce tests that are empirically grounded, fair, reliable, valid, and legally defensible. • Identify proficient interpreters at different credential levels. L. Metcalf 2017 Copy only with permission

11. Developing the Evaluation was a Three-Phase Process 1. Job/Language Analysis to document specifications for construct and content validity 2. Develop Test of Language Proficiency 3. Develop Interpreter Performance Examinations L. Metcalf 2017 Copy only with permission

12. Survey Structure Questions we wanted answered: • Who is currently interpreting? • Where are they working? • What tasks are they performing and how often? • What knowledge, skills, and abilities must interpreters possess to perform their tasks? • How might the certification process be improved to ensure that interpreters possess these skills? L. Metcalf 2017 Copy only with permission

13. Survey Sample • Survey was mailed to 588 incumbents. • All Level II through V interpreters and 10% of Level I who had been certified over 5 years. • Returned surveys evaluated for any bias with no correlation found. • Job analysis completed Fall of 2002. L. Metcalf 2017 Copy only with permission

14. Example of Findings • Question: In what settings are interpreters being most used and what level of certification do they hold? • Question: What skills do you need in order to be successful at your job? (1,632 pieces of data per questionnaire - 306,816 pieces of data total) L. Metcalf 2017 Copy only with permission

15. Implications for Test Development • Expert Panel opinion and Job Analysis data concurred that a three-level certification system was justified: Basic, Advanced, and Master. • Basic: Emphasis on educational settings • Advanced: Emphasis on routine medical, routine legal, government, and routine mental health settings. • Master: Emphasis on complex medical, complex legal, and complex mental health settings. L. Metcalf 2017 Copy only with permission

16. Implications for Test Development • In addition, Sight Translation is introduced as a new component of the BEI Performance Exam. L. Metcalf 2017 Copy only with permission

17. Implications for Test Development Based on the University of Arizona’s testing model, the job analysis and input from the Expert panel a two-stage process was decided upon: • Written Test of English Proficiency (TEP) • Interpreting Performance Exam L. Metcalf 2017 Copy only with permission

18. Ethics and Protocol Testing • Knowledge of the Standards of Ethical Behavior and Deaf culture and history are essential to the responsible fulfillment of interpreter duties. • It was recommended that these areas should be incorporated into training and continuing education programs, not into the certification exam. L. Metcalf 2017 Copy only with permission

19. Justification for this decision • The type of knowledge tested does not directly relate to the ability to accurately interpret. • They reflect a fairly narrow scope of knowledge, are easily learned and memorized, and may skew test results so that deficiencies in other areas, such as English proficiency, are camouflaged. L. Metcalf 2017 Copy only with permission

20. Test of English Proficiency • Screening tool • Measure of essential KSATS • Allows for the assessment of a broader and more representative sample of a candidates language proficiency • The 5 sections of the test were designed to assess candidates proficiency at the lexical, syntactical and discourse levels • Test items were written to assess the level of English proficiency at the 11th and 12th grade level as identified in the job/linguistic analysis L. Metcalf 2017 Copy only with permission

21. Phase 2: Development of TEP • Do candidates posses the minimum level of English proficiency required to advance to the performance examination? • 16 items in each of 5 subsections • 80 items on the final Test of English Proficiency • Test items are multiple choice format L. Metcalf 2017 Copy only with permission

22. Development of Pilot Test • A valid written proficiency test requires pilot testing to select items that are at the appropriate level of difficulty and are able to discern between acceptable and unacceptable levels of proficiency. L. Metcalf 2017 Copy only with permission

23. Development of Pilot Test • BEI Expert Panel and UA NCITRP staff collaborated in the development and revision of 161 total initial items for piloting. After statistical analysis, the final 80 items were chosen and divided into five subsections. L. Metcalf 2017 Copy only with permission

24. Five Subsections of the TEP • Reading Comprehension • Synonyms • Grammar and Usage • Sentence Completion • Antonyms L. Metcalf 2017 Copy only with permission

25. Phase 3 Developing the Performance Exams • Establishing Exam Format • Scripting Process • Filming of Performance Exam • Scoring Methodology • Pilot Testing L. Metcalf 2017 Copy only with permission

26. Introduction to Interpreter Performance Test Three certification Levels: • Basic • Advanced • Master L. Metcalf 2017 Copy only with permission

27. Performance Test Format Factors Differentiating the Exam Levels: • Complexity of Language • Complexity of Topics/Settings • Speed of Speaker/Signer L. Metcalf 2017 Copy only with permission

28. Performance Test Format Factors Differentiating the Exam Levels: • Basic ~ 110 wpm • (Click to hear a sample.) • Advanced ~ 120 wpm • Master ~ 130 wpm L. Metcalf 2017 Copy only with permission

29. Scripting Process • Scripting involved attending to a host of issues to ensure that the resulting exams contain a representative sample of the relevant aspects of languages and language abilities required of interpreters. • Begins with Job Analysis, Review of Current Exams, and BEI Expert Panel • Hundreds of hours and dozens of drafts L. Metcalf 2017 Copy only with permission

30. “Loading” the Scripts Register Variation Authentic Language ASL & English Breadth and Depth of Vocabulary Specialized Terminology Grammatical & Linguistic Elements Consistent Density Colloquial & Idiomatic Speech Fingerspelling & Numbers Proper Nouns Affect . . . and so on L. Metcalf 2017 Copy only with permission

31. Scoring System • Purpose: to provide a replicable, fair, and valid device for assessing the interpreting proficiency of candidates for certification. • The function of this system is to assess an interpreter’s ability to transform meaning from the source language and accurately conveythe same meaningin the target language. L. Metcalf 2017 Copy only with permission

32. Two-Part System Objective scoringused specifically to determine candidates’ interpreting proficiency by evaluating underlined scoring units. Subjectivescoringwill supplement this by holistically evaluating candidates’ Delivery, Adaptability, and Pronunciation/Fluency. L. Metcalf 2017 Copy only with permission

33. What the Exams Measure Objective Scoring System • Interpreting Proficiency The ability to meaningfully and accurately understand, produce, and transform ASL to and from English in a culturally appropriate way. L. Metcalf 2017 Copy only with permission

34. Objective Assessment • The objective assessment of a candidate’s level of interpreting proficiency will be determined by how many Objective Scoring Unitsthe candidate renders appropriately. • ObjectiveScoring Unitsare contained in every part of each exam. They represent significant words, phrases, and clauses that are critical to communication. L. Metcalf 2017 Copy only with permission

35. Examples of Objective Assessment • Specialized Terminology • Register Variation • Affect/ Other Rhetorical features • General vocabulary • Grammatical structures L. Metcalf 2017 Copy only with permission

36. Examples of Objective Assessment • Idiomatic language • Use of classifiers and non-manual markers • Accuracy of fingerspelling • Use of sign space and grammatical space L. Metcalf 2017 Copy only with permission

37. Sample Script from the DARS Study Guide L. Metcalf 2017 Copy only with permission

38. Subjective Scoring System • Holistic Scoring • Three Dimensions: Delivery—is the ability to maintain appropriate delivery, pacing, coherence, and composure consistency throughout the interpretation Adaptability—is the level of resourcefulness the candidate display in adapting to changes, patterns, and challenges in the text Subjective Scoring Scale 3 = Exceeds Expectations 2 =Meets Expectations 1 = Fails to Meet Expectations L. Metcalf 2017 Copy only with permission

39. Pronunciation/Fluency • Scored holistically for candidate feedback only • Pronunciation—is the ability to produce spoken language, including accurate English phonology, and to produce clear signs, as well as the appropriate use of rhythm, stress, and intonation, without interfering with meaning or undermining comprehensibility • Fluency—is the ease with which a candidate can produce native-like language, including the degree of hesitation and the clarity of signs L. Metcalf 2017 Copy only with permission

40. Parts of the Performance Exam Expressive Interpreting In this section, candidates will be asked to watch a video recording of spoken English and render it into ASL. L. Metcalf 2017 Copy only with permission

41. Parts of the Performance Exam Expressive Transliterating In this section, candidates will be asked to watch a video recording of spoken English and render it into signed English. L. Metcalf 2017 Copy only with permission

42. Comparison of Interpreting vs. Transliterating • Click to launch YouTube • Remember, grammatical perfection is not the goal • Expectation for grammatical mastery increases with the Advanced and Master Evaluations L. Metcalf 2017 Copy only with permission

43. Parts of the Performance Exam Receptive Only the Basic Level test has a section entitled, “Receptive.” In this section of the Basic Level test, candidates will be asked to watch a video recording of sign (some combination of ASL and signed English) and render it into spoken English. L. Metcalf 2017 Copy only with permission

44. Parts of the Performance Exam Receptive Interpreting The Advanced and Master Level tests both include a section entitled, “Receptive Interpreting.” In this section, candidates will be asked to watch a video recording of a presentation in ASL and render it into spoken English. L. Metcalf 2017 Copy only with permission

45. Parts of the Performance Exam Receptive Transliterating • The Advanced and Master Level tests both include a section entitled, “Receptive Transliterating.” In this section, candidates will be asked to watch a video recording of a presentation in signed English and render it into spoken English. L. Metcalf 2017Copy only with permission

46. Parts of the Performance Exam Sight Translation The Sight Translation part of each performance test differs from the other parts of the tests in that it does not include a pre-recorded video stimulus. At the beginning of this section, the candidate will be instructed to sight translate a short, written English document into ASL. L. Metcalf 2017 Copy only with permission

47. Basic Level Performance Test Content • Focuses primarily on the language found in routine educational and social service settings, as determined by the empirical job analysis of incumbent Texas interpreters. • Examples of possible topics include: K-12 classroom presentations by students or teachers, special school assemblies, homework assignments, school memos, informational meetings or orientations. L. Metcalf 2017 Copy only with permission

48. Performance Test Video Instructions INTRODUCTION: This is the Expressive Interpreting part of the exam. This part consists of a pre-recorded simulated lecture. In a moment, you will hear a pre-recorded “warm-up” that will introduce the lecture. You may interpret it if you wish, but the warm-up will not be scored. During the warm-up you may adjust the volume to your level of comfort. Following the warm-up, you will have one minute to prepare for this part of the examination, after which you will be prompted by the words, “Begin Interpreting Now”, which mark the beginning of this part of the exam. Any questions? L. Metcalf 2017Copy only with permission

49. Sight Translation Instructions INTRODUCTION: This is the Sight Translation part of the exam. You have been asked to sight translate a written English document into sign. You will have a total of seven (7) minutes both to prepare and to deliver your translation. You may start your translation when you wish, but if you have not started in two (2) minutes, the technician will instruct you to begin. Each introduction will also contain a sentence describing the setting in which the document is being used, and for whom you are interpreting. For example, the sentence could read, “You have been asked to sight translate a PTA flyer for the deaf mother of an elementary school student.” L. Metcalf 2017 Copy only with permission

50. Advanced Level Performance Test Content • Focuses primarily on the language found in a variety of high stakes settings, such as routine medical, social services, higher education, routine mental health, and routine quasi-legal, as determined by the empirical job analysis of incumbent interpreters. • Examples of possible topics include: post secondary student and professor classroom presentations, professional development seminars, doctor or dentist visits, application for services, employment forms, counseling sessions. L. Metcalf 2017 Copy only with permission