EVALUATOR TIPS FOR REVIEW AND COMMENT WRITING

1. EVALUATOR TIPS FOR REVIEW AND COMMENT WRITING The following slides were excerpted from an evaluator training session presented as part of the June 2011 CBFA Conference. Special thanks to Dr. Annette Craven for providing resources she received after participating in a recent national Baldrige examiner training.

2. Use of Standards & Criteria The Standards and Criteria serve as the basis for: • Providing feedback to applicants in the form of Process & Results • Strength comments • OFI comments

3. Process Maturity

4. Individual Review

5. Individual Review Process Read Overview Read Standards One - Six

6. Evaluation Dimensions • Approach • Deployment • Results • Improvement (Learning)

7. Approach • Appropriateness of methods • Effectiveness of the methods • The degree to which the approach is repeatable and based on reliable data and information (i.e., systematic)

8. Systematic • Documented • Repeated • Evaluated • Improved

9. Deployment (Implementation) • the approach applied in addressing the requirements is relevant and important to the organization • the approach is applied consistently • the approach is used by all appropriate work units

10. Results • Current performance • Rate and breadth of your performance improvements • Performance relative to appropriate comparisons and/or benchmarks • Linking results measures to important customer, market, process, and action plan performance requirements identified in the Overview and in Approach-Deployment Items.

11. Where to Look for Results • Results are found throughout the ACBSP Standards, for example: • Student/Stakeholder Satisfaction (Standard 3) • Student Learning Outcomes (Standard 4) • Faculty Deployment & Qualifications (Standard 5) • Business Process Results (Standard 6) • Performance is generally presented in one of three ways • Charts/Graphs • Tables • Narrative

12. What Should You Look For? • Expected Results • Appropriate Comparisons • Important Segmentation

13. Appropriate Comparisons • “Performance is examined relative to competitors and/or other organizations providing similar products and services” • “Include appropriate comparative data”

14. Improvement (Learning) • Refining the approach through cycles of evaluation and improvement • Encouraging breakthrough change to the approach through innovation • Sharing refinements and innovations with other academic departments or disciplines and processes in your school of business or institution

15. Writing Evaluator Observations • Self-study • Standards and Criteria Booklet • ACBSP handouts including evaluator workbook

16. Observation Guidelines Observations should be: • Standards and Criterion based • Relevant • Clear • Concise • Actionable

17. Observation Guidelines (Cont.) Observations should not: • “Parrot” the self-study • Be prescriptive • Be judgmental • Conflict with one another

18. Strength Observations: 3 Components • Nugget • what the business unit does to merit the score you assign to them • Example(s) • are the things the organization does with some level of proficiency • Relevance • Why the observation is important for the business unit (e.g. mission related, etc.)

19. Strength Observation Example • “The development of the budget is an integral part of the strategic planning process. This ensures that the budget reflects the business unit’s strategic priorities. As a result, both long-term plans and shorter range operational activities are focused on the organization’s mission, vision, and values.”

20. OFI’s • Is an opportunity for improvement • Is not necessarily a weakness • Is what prevents an applicant from scoring at a higher level • Is tied to the Standards and Criteria

21. Example of an OFI Observation • the business unit, through its strategic planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

22. So What’s • Explains the significance of the observation so that the business unit doesn’t have to ask, “So what?” • Can be used with both Strengths and OFI’s though more commonly used with OFI’s

23. Where’s the So What? • the business unit, through its strategic planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

24. Where’s the So What? • the business unit, through its strategic planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

25. Guidelines for Results • Start with a subject from the self-study or the Criteria • Include the time frame you are writing about -- such as “in 2009” or “from 2005 to 2010” • Include the actual numbers observed in the levels or trends • Include figure references

26. Results Strength • the business unit operates in a fiscally sound manner as evidenced by its Aaa and AAA bond ratings (Figure 7.3-1) and debt capacity (Figures 7.3-8 and 7.3-11). These results suggest that the business unit makes financial decisions with its long-term financial viability and sustainability in mind.

27. Results OFI • Only 3 of the 13 figures in Standard 4, Criterion 4.3 contain comparative data. Without comparative data it is difficult to evaluate the business unit’s current level of performance or the trends they report in these figures.

28. What’s Wrong with this Observation? • “The standard being addressed is in need of improvement.” The OFI does not provide enough information – how does the business unit respond to the comment and how do the commissioners identify the weak area.

29. What’s Wrong with this Observation? • An opportunity exists to broaden the advanced educational experience of the faculty by hiring and/or sending current faculty to a broader range of institutions than is currently the case. This may result in a broader base of knowledge for students in the Department of Business. • This is a prescriptive comment – the Standards and Criteria do not prescribe which institutions faculty members should be hired from.

30. What’s Wrong with this Observation? • “Although the team was delighted with the depth of the understanding and the commitment to assessment, there were numerous closing the loop issues based on the self study report review. Upon examination and audit during the site visit, numerous closing the loop evidences were discovered.” • “Team was delighted” is judgmental and the issues discussed in the OFI were resolved while on site. This is not an OFI – it can actually be a strength.

31. Common Mistakes • Critique vs. information • “It is not clear” • Not addressing both feedback report customers: Commissioner and school/program • Too much brevity • Lack of clarity; “So what”? • Program Evaluator Self-Study Review not converted to Feedback Report format • Comments limited to “they have a plan…” • Preliminary comments not prepared prior to site visit

32. QUESTIONS TO ASK YOURSELF • WHEN REVIEWING THE SELF-STUDY: • What approach is described? Is it systematic? • How is the approach deployed and to what extent for applicable stakeholder groups? • Are the results presented? • 4. Is there evidence of learning or improvements?

33. QUESTIONS TO ASK YOURSELF • WHEN REVIEWING THE SELF-STUDY: • Are the results presented? • Are there three to five data points? • Are the results segmented? • Are comparisons provided (e.g.,peer, national norm, etc.) ? • 4. Are targets/benchmarks identified?