430 likes | 506 Views
Assessing Faculty. By “Sitting Beside” Larry A. Braskamp Loyola University of Chicago Lbraska@luc.edu. Assess ( e -ses’) v.t. …To take stock of; evaluate; to assess the situation … L assidere sit by (as a judge in court)…
E N D
Assessing Faculty By “Sitting Beside” Larry A. Braskamp Loyola University of Chicago Lbraska@luc.edu
Assess ( e-ses’) v.t. • …To take stock of; evaluate; to assess the situation • … L assidere sit by (as a judge in court)… • Source. Funk and Wagnalls New International Dictionary of the English Language, 1993
To Assess is to • Understand • Judge • Act (decide, assist, help develop)
Challenge • Design and implement a faculty assessment program that simultaneously fosters individual faculty development and fulfills the institutional mission
Assessing by “Sitting Beside” Setting Expectations Using Evidence “To Sit Beside” Collecting Evidence • Source. Braskamp, Larry A. & Ory, John C. (1994) Assessing Faculty Work. San Fransisco: Jossey-Bass
“To Sit Beside” as an image • To humanize the process • To understand as well as to judge • To enhance role of colleagues • To build community • To increase respect for diversity • To demonstrate individual accountability • To promote mutual and collective accountability
Assessing by “Sitting Beside” Setting Expectations Using Evidence “To Sit Beside” Collecting Evidence
Setting Expectations • Division of faculty work • Teaching • Research and Creative Activity • Outreach/Professional Practice/Engagement • Citizenship • Individual and Institutional • Responsibilities vs. quality • Quality/value/impact/influence • Merit • Worth
Setting Expectations • To expand the variety of faculty work (roles and responsibilities) to be recognized and rewarded (e.g., teaching; research and creative activity; outreach; citizenship)
The Work of Teaching • Instructing • Advising, Supervising, Guiding, and Mentoring Students • Developing Learning Activities • Developing as a Teacher
The Work of Research and Creative Activities • Conducting Research • Producing Creative Works • Editing and Managing Creative Works • Leading and Managing Funded Research and Creative Projects
The Work of Practice and Professional Service • Conducting Applied Research and Evaluation • Disseminating Knowledge • Developing New Products, Practices, Clinical Procedures • Participating in Partnerships with Other Agencies • Performing Clinical Service
Characteristics of Public Service • They contribute to the public welfare or the common good • They call upon faculty members’ academic and/or professional expertise • They directly address or respond to real-world problems, issues, interests, or concerns
The Work of Citizenship • Contributing to the Local Campus • Contributing to Disciplinary and Professional Associations and Societies • Contributing to civic, political, religious, and other communities
Setting Expectations • To distinguish between expectations for (institutional) and expectations of (individual) faculty
Setting Expectations • To distinguish between what one does and how well one does it, i.e., workload/ activities/ effort/ roles/ responsibilities vs. quality/ effectiveness/ influence/ impact/value/ merit/worth/excellence
Setting Expectations • To distinguish between merit (“quality according to standards of the profession”) and worth (“value of work that is a benefit to the institution”) M. Scriven. • Source. Scriven, Michael (1978) Avalue versus Merit. Evaluation News (8), 1-2
Assessing By “Sitting Beside” Setting Expectations Using Evidence “To Sit Beside” Collecting Evidence
Collecting and Organizing Evidence • Multiple Perspectives • Sources • Methods • Credibility of Evidence • Trustworthiness of Evidence • Validity • Reliability • Fairness • Consequences • Portrayal of Faculty Work • Building a Case
Collecting and Organizing Evidence • Everything counts, but not everything needs counting • To think in terms of “building a case” • To emphasize a multiple perspective approach to collecting evidence from multiple sources using multiple methods
Collecting and Organizing Evidence • Evidence must be credible • Evidence must be trustworthy (i.e., valid, reliable, fair, consequential) • Faculty need to be able to portray their work for others to review, critique, and judge
Multiple Sources • Oneself • Faculty colleagues • Campus administrators • Faculty development professionals • Students • Parents • Participants • Alumni • Citizens and community groups • Public officials • Professional and disciplinary colleagues • Accreditation officials • Board members • Consultants • Experts • Customers
Multiple Methods • Rating Scales • Observations • Interviews • Written appraisals • Measures of outcomes and achievements • Documentation and records review • Measures of eminence, quality, and impact • Video and audio tapes • Simulations
Types of Evidencefor Describing and Judging Teaching • Descriptions of Teaching Activities • Summary of responsibilities and activities • Analyses of student learning and challenges • Audio and videotapes • Samples of teaching • Participation in improvement activities • Outcomes • Student learning and achievements • Student development
Types of Evidence for Describing and Judging Teaching • Judgments About Teaching • Ratings from various sources • Written appraisals from various sources • Eminence Measures • Honors and awards • Invited presentations • Self-Reflections and Appraisal • Personal journals and logs • Public self-appraisals
Types of Evidence for Describing and Judging Research • Descriptions of Research and Creative Activity • Summary of responsibilities and activities • Analyses of research and creative problems • Participation in improvement activities • Outcomes • Publications in journals • Papers presented at professional meetings • Books (authored and edited) • Chapters in books • Monographs • Grants and external funding • Unpublished papers and reports
Types of Evidence for Describing and Judging Research • Judgments about Research • Evaluations from faculty peers • Evaluations from departmental chairs, deans • Evaluations from experts (curators, critics) • Eminence Measures • Referee or editor of journal • Honors and awards from profession • Officer of national professional association • Invited papers and guest lectures • Invited exhibitions and performances • Citation rate of published work • Self-reflection and Appraisal • Personal journals and logs • Public self-appraisals
Types of Evidence for Describing and Judging Practice • Descriptions of Practice Activities • Analyses of contemporary problems • Audio and videotapes • Samples of work • Participations in improvement activities • Outcomes • Client feedback on progress • Client behavioral outcomes • Degree social problem addressed is understood • Policy changes linked to work of faculty • Influence on research and teaching within profession • Influence on teaching and research within institution • Inventions, improved clinical practices and procedures
Types of Evidence for Describing and Judging Practice • Judgments about Practice • Evaluations from participants, clients, patients • Evaluations from sponsoring organizations • Evaluations of and letters of appreciation • Evaluations from faculty colleagues and experts • Eminence Measures • Honors and awards from profession • Officer of professional association • Invited exhibitions and performances • Self Reflection and Appraisal • Personal journals and logs • Public self-appraisals
Types of Evidence for Describing and Judging Citizenship • Descriptions of Activities • Attendance records of committee work • Representation at functions for institutional advancement • Support of campus activities (cultural and sporting events) • Degree of involvement in professional organizations • Degree of participation in religious/public/civic affairs • Outcomes • Changes in policies in governance of campus and professional associations
Types of Evidence for Describing andJudging Citizenship • Judgments About Citizenship • Ratings of effectiveness by faculty peers and administrators • Evaluation by fellow committee members and chair • Modeling behavior as judged by colleagues and students • Evaluation from participants of community programs, public officials • Eminence Measures • Reappointment or reelection to public office • Reelection or reappointment to leadership positions • Self-reflection and Appraisal • Personal journals and logs • Public self-appraisals
Administration Influencing Student Ratings of the Instructor or Course Student anonymity -- Signed ratings are more positive than anonymous ratings. Instructor in classroom -- Ratings are more positive if the instructor remains in the room. Directions -- Ratings are more positive if the stated use is for promotion. Timing -- Ratings administered during final exam are generally lower than those given during class. Midterm -- Ratings are less reliable if the student raters can be identified.
Nature of Course Influencing Student Ratings Required/elective -- Ratings in elective courses are higher than in required courses. Course level -- Ratings in higher-level courses tend to be higher that in lower-level courses. Class size -- Smaller classes ten to receive higher ratings, yet low Correlations between class size and student ratings suggest class size is not a serious source of bias. Discipline -- In descending order, lower ratings are given to courses in arts and humanities, biological and social sciences, business, computer science, math, engineering, and physical sciences.
Instructor Characteristics Influencing Student Ratings Rank -- Professors receive higher ratings than teaching assistants. Gender of instructor -- No significant relationship exists between gender of instructor and his or her overall evaluation, although ratings do slightly favor women instructors. Personality -- Warmth and enthusiasm are generally related to ratings of overall teaching competence. Years teaching -- Rank, age, and years of experience are generally unrelated to student ratings. Research productivity -- Research productivity is positively but minimally correlated with student ratings.
Student Characteristics Influencing Student Ratings • Expected grade--Students expecting high grades in a course give higher ratings than do students expecting low grades. • Prior interest in subject matter---Similar to elective courses, students with prior interest give somewhat higher ratings. • Major or minor---Majors tend to rate instructors more positively than nonmajors. • Gender---Gender of student and overall evaluations of instructors are not related although students tend to rate same sex instructors slightly higher. • Personality characteristics---No meaningful and consistent relationships exist between the personality characteristics of the students and their ratings.
Instrumentation Influencing Student Ratings • Placement of items---Placing specific items before or after global items on the rating form has insignificant effect on the global ratings. • Number of scale points--Using six-point scales yields slightly more varied responses and higher reliabilities than five-point scales. • Negative wording of Items--Overall ratings of the course and instructor are not significantly influenced by the number of negatively worded items in the rating scale. • Labeling all scale points versus only end-points---Labeling only end-points yields slightly higher average ratings
Assessing by “Sitting Beside” Setting Expectations Using Evidence “To Sit Beside” Collecting Evidence
Using Evidence • To distinguish between individual and institutional uses of evidence • “Assessment is everybody's business, but not everybody else's business” R. Stake • To promote both individual and mutual and collective accountability
Using Evidence for Self Development • Emphasize the informational rather than controlling use • Design assessment feedback so that it is intrinsic to the task itself • Rely on specific, diagnostic, descriptive information that focuses on faculty work • Encourage feedback on work in progress • Develop mentoring relationships among faculty so that discussions of work are encouraged
Using Evidence for Accountability • Rely on various types of descriptive and judgmental evidence collected from multiple sources to develop a composite portrayal • Interpret evidence in a way that is consistent with institutional goals • Develop profiles of faculty over time • Closely link assessment with both faculty and institutional development
A Portrayal of Faculty Work • Statement of personal goals, roles and institutional expectations • Teaching • Responsibilities and Activities • Assessment and its Use • Research and Creative Activities • Practice • Citizenship • Honors and Recognition • Activities to Improve Faculty Work
Creating a Campus Culture of Assessment • Do you collect evidence about your work solely for your own personal and professional use? • Do you share this evidence with colleagues so that you can discuss your own effectiveness, enhance your own career development, and meet institutional expectations?
A Possible Reorganization • Does your current organizational structure most effectively: • Foster development of your faculty and staff? • Fulfill the mission of your institution/college/school/department