Prepared by the CTAG Exam and Manual Workgroup Presented by: Tim M. Drake, Jr., Clemson University

1. A Flexible Process for Developing Effective Integrated Licensure Tests and Pesticide Training Manuals Prepared by the CTAG Exam and Manual Workgroup Presented by: Tim M. Drake, Jr., Clemson University Margaret Tucker, Washington Department of Agriculture

2. Manual and Test Development Tool • Was developed by CTAG (Certification and Training Assessment Group) in 2006-07. • Was written in response to the need for a national guidance tool that outlines an appropriate method of manual and test development. • Is a proactive step taken by CTAG to assist agencies in the development of new manuals and certification examinations, and in the revision of existing ones.

3. Manual and Test Development Tool • Was brought about by the proposed US Environmental Protection Agency (EPA) Certification and Training (C&T) regulation changes. • Is intended only as a guidance document and not a mandatory method for manual / exam development.

4. Manual and Test Development Tool • There is no single “best” method to use in the creation of content-relevant examinations and training manuals. • CTAG does not advocate a particular process or suggest that EPA mandate states to adopt any particular method.

5. Manual and Test Development Tool • If not already implemented, states must set into motion a process that creates defensible tests which support sound decisions about who should or should not be certified as pesticide applicators. • CTAG feels that a “flexible” process is the most logical method to use.

6. Flexible Process • Allows state lead agencies (SLA), tribes, and pesticide safety education programs (PSEP) to build on what they already have in place instead of starting over with a new process. • Is less costly to agencies than the requirement to adopt a new rigid process that would likely necessitate an increase in expertise, resources, staffing, and time commitments.

7. Purpose • To provide the basic components of a flexible process for developing effective licensure tests and training manuals. • To encourage the implementation of professionally credible testing practices. • To ensure that training curricula align with licensure tests.

8. Test and Manual Development • For licensure tests to be effective, their development needs to occur in unison with corresponding study material. • Well-written manuals can change applicator behavior and provide program accountability. • The specific order of development is not as important as the incorporation of each of the components of test and manual development.

9. Using a flexible process ensures that: • Test and manual content relate directly to a job analysis of a certified applicator. • Tests are good measures for determining who should and should not become a certified applicator. • Manuals are designed to be useful both for learners preparing for licensure tests, and as basic educational tools to help new applicators properly handle and apply pesticides.

10. Part A: Licensure Testing • Licensure tests should focus on job knowledge and skills critical to protecting the public and the environment. • Testing standards should be sufficiently rigorous, but not so strict that they limit the rights of a qualified person to engage in professional practice.

11. Six Steps of Licensure Test Development • Job Analysis • Test Plan Development • Item Writing • Test Assembly • Item Analysis • Standard Setting

12. Job Analysis • Provides the proper basis for establishing licensure test and manual content. • Is a systematic means of collecting and organizing information about what jobholders do and which qualities are necessary to perform a job safely and successfully.

13. Job Analysis • Should provide the means to ensure that test and manual content reflect necessary knowledge and skills. • Should lead to an examination that is properly balanced for the test taker.

14. Job Analysis • Must define the target population • Must gather information on tasks (work-related behaviors) and competencies (knowledge, skills, abilities that support task performance). • Typically uses a two-phase inductive job analysis method.

15. Job Analysis • Inductive Job analysis consists of two phases: 1) collection and organization, by qualitative means, of job information, and 2) corroboration and quantification of the gathered information. • Qualitative data collection includes jobholder observation, jobholder interview, subject matter expert discussion, and document analysis.

16. Job Analysis • Quantification may be performed by questionnaire or discussion among a small panel of subject matter experts. • Using subject matter experts is appropriate when working with small and/or homogeneous populations of jobholders , testing at the state or local level, or when revising an existing examination.

17. A Typical Subject Matter Expert

18. Test Plan Development • A committee of subject matter experts should formalize a test plan. • Content emphasis is determined by ranking of job information associated with each content area.

19. Test Plan Development Test Plan should include: • Total number of test items • Item format • Number of items per content area • Administration procedures • Scoring rationale

20. Item Writing • The test plan should serve as an item writing guide. • Subject matter experts review items for technical accuracy and editors review items for grammar, punctuation, consistency, and format.

21. Test Assembly • In test assembly, items are selected for inclusion on the test according to content and number, so as to reflect the test plan.

22. Item Analysis • Pilot testing should be performed to identify the poorly functioning items. • Statistical analyses should be performed. • These analyses should include p-values, discrimination indices, and a reliability estimate.

23. Standard Setting • Establishes a passing threshold • Should be a subject matter expert committee activity. • Regulatory agencies may elect to perform an item analysis after the first test administration if a review of item statistics show that there are poorly functioning items. This will allow any flawed test items to be repaired before the next test administration.

24. Part B: Training Manual Development Training Manuals May Serve As: • Study Guides • Basic Pesticide Users’ Guide • Outreach Document (by SLA and Educators) • Comprehensive Resource (providing advanced technical and use information for new and experienced pesticide applicators).

25. Training Manual Development Through study of a well-constructed training manual, the learner should develop a thorough understanding of safe pesticide use, and also be made aware of their obligation to ensure security and to protect human health and the environment.

26. Training Manual Development Training manuals should be developed with two primary goals in mind: • Answers to all test items must be found in or inferred from the manual. • The manual must facilitate learning and test preparation as well as provide necessary content identified by the job analysis.

27. Training Manual Development Incorporation of the following elements is essential for the development of good training manuals: * Learning Objectives * Learner Orientation * Short, Topic-Specific Chapters * Review by Subject Matter Experts * Readability Measures * Cooperation Between SLA and PSEP

28. Learning Objectives • Reshape the job analysis into a set of specific learning objectives. • Align manual content to these objectives. • Let each learning objective represent only a single concept. • Express each learning objective as an action phrase.

29. Learning Objectives Each Objective Must : • Relate to knowledge or skills that you want the applicator to obtain. • Use an action verb to tell the person what is expected of them. • Be measurable

30. Learner Orientation • Learning objectives are effective only if their meaning and importance are conveyed to the learner. • This can be accomplished in the manual’s introduction. • Learning objectives must indicate what the individual is expected to learn and what they are expected to demonstrate on the licensure test.

31. Learner Orientation • Tell the user how to best use other components (appendices, glossary, etc.) of the manual. • Include several sample items that illustrate the test format. • Provide a copy of the answer sheet style used for the licensure test. Explain how the sheet is filled in and discuss any common problems with it.

32. Learner Orientation The more the user is familiarized with the test, the less anxiety they will experience, and the more likely it is that the test score will be a true reflection of their knowledge and skills.

33. Short, Topic-Specific Chapters • Using the learning objectives as a starting point, develop an outline of the manual’s content with a hierarchy of chapters. • Learning will be enhanced because the user can focus on a small set of concepts and then proceed to the next in a logical sequence. • Keep each chapter short. • Organization must be from the learners’ point of view.

34. Review of Manual • Review by subject matter experts is needed to ensure that the manual content is pertinent, accurate, and complete. • This should occur early in the manual development process. • The manual should be reviewed more than once.

35. Readability Measures Factors that affect the physical act of reading are: • Font Size and Type • Line Length • Justification • Line Spacing • Use of White Space

36. Readability Measures Factors that affect a reader’s ability to comprehend text include: • Reading level of the text • Effective use of headings and subheadings • Active vs. passive voice • Vocabulary • A clear and concise writing style

37. SLA and PSEP Cooperation • Manual development should require cooperation between the SLA and PSEP programs. • SLA and PSEP should work from a common job analysis and reach agreement on the scope and wording of learning objectives. • This cooperation also is important during the SME review phases.

38. SLA and PSEP Cooperation • Having at least one SLA representative involved as a SME helps ensure that the training manual targets enforcement issues. • At least one representative from both the SLA and PSEP programs should review the final draft of any licensure test to ensure consistency between the manual and each item on the test.

39. Other Considerations • Manual / Test development is a time-consuming and expensive process when it is done correctly. • Good manuals use research-based techniques. • Considerable time, effort, and expense can be saved in the construction of a quality test when starting with a well-designed manual.

40. Other Considerations • Cost efficiency may be enhanced by adapting an existing manual from another state, or by working cooperatively to develop manuals. • Multi-state manuals reduce the cost for any one state and create a larger pool of content expertise. • This may be more difficult to accomplish at the regional or national level because relationships are more distant and less interdependent.

41. Other Considerations • One option would be to share, nationally or regionally, a category job analysis because the vast majority of the work performed by the applicators is identical. • States could use their SMEs to adapt the job analysis to the state’s unique legal, climatic, and pest issues when they revise or create new manuals and exams.

42. Licensure test and training manual development have been discussed separately for clarity. However, the best practice is to develop these together, basing the content of each on a common job analysis and linking the job analysis, training manual, and licensure test via learning objectives.

43. CTAG Exam and Manual Workgroup • Michelle DeVaux US EPA • Tim Drake Clemson University • Ron Gardner Cornell University • Dean Herzfeld University of Minnesota • Drew Martin Purdue University • Clyde Ogg University of Nebraska • Carol Ramsay Washington State University • John Scott Colorado Dept. of Agriculture • Margaret Tucker Washington Dept. of Agriculture • Dan Wixted Cornell University

44. Questions?