STANAG 6001 Testing Workshop Skopje, Macedonia 5 – 7 September 2017

1. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA SETTING THE STAGE FOR TESTER SUCCESS: THE MACEDONIAN EXPERIENCE STANAG 6001 Testing Workshop Skopje, Macedonia5 – 7 September 2017

2. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA English language proficiency testing IAW STANAG 6001 Three periods: • Early Age • Middle Age • Anno Domini

3. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Early Age (2003-2010) 2003 • Edition 1 adopted • Three English language teachers authorized 2008 • Edition 2 adopted 2009 • Rules and Procedures for Administration of STANAG 6001 Proficiency Testing

4. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Early Age (2003-2010) • Challenges? Yes • teaching and testing • ad-hoc testing • translations • testers leaving the team • Working group formed • MA, HRD/MODorTRADOC – which one satisfies the testing team needs?

5. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Middle Age (2010-2014) • transfer of the team in HRDstructure • 3 full-time testers • unfamiliar with testing practices • recognized good practices: • 2 testing sessions annually • 50 test takers per session • 2-year valid certificates • pre-testing of test items abroad

6. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Middle Age (2010-2014) • communication with BILC • seminars in GarmischPartenkirchen • annual testing workshops • bilateral cooperation • tailored workshop with BILC experts • documentation: • STANAG 6001 Test Specification • STANAG 6001 Handbook for Test Takers

7. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Middle Age (2010-2014) • More challenges? Yes • Hiring freeze • New testers • Selection? Not an option • prerequisites: • experienced English language teacher • bachelor’s degree in English language and literature • high level of English • high level of enthusiasm and motivation

8. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Anno Domini (2014 - )

9. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA First day as a tester After a week as a tester First day as a tester After a week as a tester

10. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Job qualifications for new testers – higher position Advisor – STANAG 6001 tester • Bachelor’s degree in English language and literature • At least 3 years of work experience in the field • Knowledgeable in MS Office • ability for objective reasoning and development of new solutions • ability to work in a team • excellent organizational and interpersonal skills • STANAG 6001 SLP 3333 or C1

11. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA • Work assignments: • Participate in compiling a manual for proficiency testing IAW STANAG 6001 • Prepare procedures for each segment of the testing process (item writing, item moderation, test administration, …) • Write test items based on discourse analysis • Modify test items based on qualitative and quantitative analysis • Rate productive skills • Prepare analyses, reports and various kinds of information regarding the testing process • Propose measures for improving the process of implementation of STANAG 6001

12. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Job qualifications for new testers – lower position Associate – STANAG 6001 tester • Bachelor’s degree in English language and literature • At least 1 year of work experience in the field • Knowledgeable in MS Office • ability for objective reasoning and development of new solutions for certain issues • ability to work in a team • excellent organizational and interpersonal skills • STANAG 6001 SLP 3333 or C1

13. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA • Work assignments: • Write test items based on discourse analysis • Modify test items based on qualitative and quantitative analysis • Rate productive skills based on discourse analysis • Prepare, organize and coordinate test administration • Invigilates test administration • Gather information and help in the preparation of analysis and reports • Update the test item bank • Conduct statistical analysis

14. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Major features of Anno Domini • Formal training • Review of documentation and procedures “Manual for Working Practices of Test Specialists in the Development and Administration of English Language Proficiency Tests IAW STANAG 6001” • Development of standardized procedures • Test Administration Manual • Familiarization with the testing process

15. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Draft Test Administration Manual Contents • Test Administrator Responsibilities • Preparations for Test Administration • Review and Signing of Test Administrator Security Oath • Preparation of the Testing Environment • Preparation of Test Materials for Distribution • Test Administration • Ensure Proper Testing Procedures • Breaks • Answer Students’ Questions • Collect Test Materials • Return Test Materials • Special situations with sensitivity • Test Administration Instructions for Writing Section • Test Administration Instructions for Reading Section • Test Administration Instructions for Listening Section • Test Administration Instructions for Speaking Section

16. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Draft Test Administration Manual Extract from the writing section: “Read aloud WORD FOR WORD the material printed in bold type and preceded by the word “SAY.” You may repeat the directions as many times as needed. The material in italics is information meant for you and should not be read aloud to students. SAY:I kindly ask you to place all cell phones and personal electronic devices on the appointed spot during test administration. Cell phones can be on silent mode (no vibrations).

17. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Draft Test Administration Manual Extract from the writing section (cont.): After all test takers are seated (after leaving the devices), tell them to prepare the ID. SAY: Prepare your ID with a photo. Leave the ID on the desk in front of you. Check IDs. SAY:No additional reference materials are allowed during testing (dictionaries, grammar books, etc.)All desks must be cleared of books and other materials that are not required for the test.On the desks in front of you, you should only have a writing kit. “

18. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Familiarization with the testing process • Familiarization of teachers and students • Familiarization course for STANAG 6001 on Moodle platform • Lecture on “Implementation of STANAG 6001”

19. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA

20. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Division of responsibilities (e.g. responsibilities of Tester #1) • needs analysis of the reading items: • review the item bank, • determine how many items have good values, • how many are outdated, • how many need revision, • how many have been overused and will be transferred to the bank that includes items to be exposed for practice • coordination of writing reading items: • text selection • plenary text analysis • item writing • exchange of items and individual moderation • plenary moderation and finalization of items • update the item bank for the reading skill

21. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Current features of STANAG 6001 testing system in MKD • 3 full-time testers • 2 testing sessions per year + one upon request • 50 test takers per session • Bi-level tests: • Levels 1 & 2 pre-screened with ALCPT (min. 35 points) • Levels 2 & 3 pre-screened with ALCPT (min. 75 points) • 3-year valid certificate • Pretesting abroad

22. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Towards a better future • Include a variety of item types • Operationalize STANAG 6001 familiarization course on Moodle platform • New Handbook on STANAG 6001 test • Computer assisted testing

23. MINISTRY OD DEFENCE REPUBLIC OF MACEDONIA Thank you for your attention!!!