Robert Feyen, Ph.D., CPE Asst. Professor/MEHS Program Director, University of Minnesota Duluth

1. Academic Program Evaluation Using BCSP CriteriaA Case Study at the University of Minnesota Duluth Robert Feyen, Ph.D., CPE Asst. Professor/MEHS Program Director, University of Minnesota Duluth T.W. Loushine, Ph.D., P.E., CSP, CIH Asst. Professor, University of Wisconsin - Whitewater

2. UMD MEHS program Background • Started in 1976 with roughly 750 graduates to date • Practitioner-based program • Designed primarily to prepare students as generalists in EHS, but allow room for some specialization Mission statement The Master in Environmental Health and Safety program at the University of Minnesota Duluth will produce highly-regarded and sought-after graduates who have the requisite skills and knowledge to practice environmental health and safety effectively in a diverse range of occupations and will do so in a competent, professional and ethical manner.

3. Curriculum: Content (2008) Core (32 credits) • SAFE 6002—Regulatory Standards and Hazard Control (3 cr) • SAFE 6011—System Safety and Loss Control Techniques (3 cr) • SAFE 6012—Risk Management and Workers’ Comp (2 cr) • SAFE 6051—Construction Safety Management (3 cr) • SAFE 6101—Principles of Industrial Hygiene (3 cr) • SAFE 6111—Industrial Noise and Ventilation Control (3 cr) • SAFE 6301—Occupational Biomechanics (2 cr) • SAFE 6302—Occupational Ergonomics (3 cr) • SAFE 6401—Environmental Safety and Legal Implications (2 cr) • SAFE 6802—Leadership, Teamwork, Behavior in EHS (3 cr) • SAFE 6821—Organization and Administration of Safety Programs (2 cr) • SAFE 6997—Cooperative Internship Electives (7 credits) • SAFE 6102—Advanced Industrial Hygiene (2 cr) • SAFE 6121—Epidemiology and Industrial Toxicology (2 cr) • SAFE 6201—Fire Prevention and Emergency Preparedness (2 cr) • SAFE 6211—Transportation Safety (2 cr)

4. Curriculum: Subjective Feedback • Internal (University-level) administration • Economic constraints (e.g., coordination of resources) • Academic “consistency” with other graduate level programs • EHS professionals (formal and informal) • Alumni • Employers • Practitioners • Peer graduate programs in safety and health • Funding agencies (e.g., NIOSH) • External reviewers

5. Curriculum: Objective Assessment • Literature • Graduate Research Council reports (e.g., student demographics) • ASSE guidelines for ABET accreditation • Academic journals (e.g., Journal of Safety Science) • ASEE and ASSE proceedings papers on curriculum assessment • BCSP outcome assessment program • Developed an assessment approach based on Brauer(2006) • Utilized the BCSP subject matter tables (2008) • However, no other literature could be found regarding this approach or its use • The (the devil) details • Took a few tries, several modifications and a GRA to obtain data on each course • Took even more tries to figure out what to do with the data…

6. Method • Instructors identify knowledge items from the BCSP subject matter tables to be used in evaluating each course taught • Develop assessment matrix of courses and relevant knowledge items • Conduct interviews with students and instructors to rank emphasis and rate coverage of each item • Enter item ratings and rankings for each course into spreadsheet • Analyze and interpret data • Aggregated over all knowledge items (n=249) • For each subject area, use ratings to determine percent of items covered within overall program and individual courses • Within subject area, identify knowledge items not covered thoroughly or requiring additional evaluation

7. Method: Subject Matter (example) Measurement and Monitoring • Knowledge (Foundation Level) • Methods and techniques for measurement, sampling, and analysis (D2- T1) • Uses and limitations of monitoring equipment (D2-T1) • Knowledge (Advanced Level) • Electronic data logging and monitoring equipment (D1-T2) • Electronic data transfer methods and data storageoptions (D1-T2) • Methods and techniques for measurement, sampling, and analysis (D1-T1, D2-T1)

8. Methods: Measures • Objectives: • Identify knowledge items for evaluation within a course • Rate evidence of coverage/assessment for knowledge items

9. Results: Initial Assessment • 3 knowledge items not identified for rating

10. Results: Master Dataset • 14 courses (horizontal) • 4 columns each • “rated”, “primary”, “secondary”, and “tertiary” • 249 items (vertical) • Group by subject area • Color-coded • Green, yellow, red • Blue – no rating

11. Results: Aggregate by Rank & Rating

12. Results: By Subject Areas

13. Results: Subject Area by Coverage Blue = .80-1.00; Green = .60-.79; Yellow = .40-.59; Orange = .20-.39

14. Results: Subject Area by Non-Coverage

15. Results: By Course Coverage • For Business Management Principles (n=28) • 39% with rating of “5” • 71% with rating of “3 or greater”

16. Discussion: Cons • Requires time and effort commitment • Process requires motivated participants • Up to two hours per course • Faculty to provide data on the courses they teach • Subset of students to provide data on courses they have taken • Three to four hours per course • One person to setup assessment spreadsheet, coordinate data collection, enter and interpret data • Terminology from BCSP blueprints is often unclear and redundant • Requires reaching consensus on definitions • May be misinterpreted

17. Discussion: Pros • Effective benchmarking tool • Provide baseline measurement of topic coverage • Individual courses • Overall program • Assess effectiveness of topic coverage • Compare faculty expectations to student perceptions of topic coverage • Evaluate how program prepares graduates for either the ASP or the CSP • Evaluate impact of modifications • Course alterations • Program restructuring*

18. Discussion: Example of Simple Improvement

19. Discussion: Pros • BCSP foundation provides objectivity and flexibility • Externally validated criteria • Provides objective assessment of curriculum • Easily adapted to any EHS program* • Other degree programs with a certification or licensing body could apply the approach*

20. Summary • The UMD MEHS program needed an objective means to assess curriculum design • Modified Brauer’s proposed approach for using the BCSP blueprints to evaluate curriculum • Using a combined ranking and rating system for each course in the program provided an effective multi-purpose benchmarking tool • Questions?