2012 EXAMINER TRAINING Final Workshop November 15, 2012

1. 2012EXAMINER TRAININGFinal WorkshopNovember 15, 2012 Presented by The Granite State Quality Council and The Northern New England Alliance for Excellence

2. Goal of this Training White River Junction VA Medical Center • Provide each participant with a method and the tools to develop Baldrige-based feedback for Award Applicants in New Hampshire, Vermont, Maine, and other State Baldrige-based Programs. • Provide each of you with a deeper understanding of Baldrige in support of your organization & career

3. Agenda: Examiner Training – Final Workshop • Introductions • Putting faces with voices…. • Quick Review • Independent Review Process • The Consensus Process • Creating Consensus Comments and Rationale • Feedback (Applicant) Ready Comments • Practice, practice, practice • Examiner Roles, Ethics & Conflict of Interest • The Organizational Profile Assessment • After Consensus • Key Themes • Introduction to Site Visit

4. The Baldrige Application Evaluation Process - Stages Goal: Value Added Feedback To the Applicant Site Visit Comments Stage 3 Consensus Comments Today’s Focus Stage 2 Independent Review Stage 1

5. Review: The Application Evaluation Process – Independent Review Evaluate Each Criteria Item Check for Conflict of Interest Assemble Materials and Read Application Key Themes Observations Draft Initial Key Factors Complete Checklist and Assemble Scorebook

6. Independent Review Worksheet A-D-L-I Or Le-T-C-I Supporting Evidence & Examples Main Idea or Nugget Strengths / OFIs Relationship with Key Factors (Importance) Relationship with Criteria (traceability)

7. Independent Review Worksheets – Comments & Score

8. Feedback Ready Comments • Turns Strength/OFI & Evidence into clear and actionable feedback for the applicant • Polite, Professional, Positive • Not: Judgmental or Prescriptive • Concise: one message per comment • Indicates • Criteria-based strength or potential OFI • Potential significance of Strength/OFI

9. Elements of a feedback ready Comment usingN-E-R-DComments should include the following: 1. Nugget: Key point based on Criteria 2. Examples for clarity & context 3. Relevance to this applicant Done! No need to add additional information what won’t add value to the applicant

10. Creating a Feedback Ready Comment: 5.1 Strength b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace health, safety, and security. These methods include a system-wide Safety Committee that meets monthly to review progress toward goals, unexpected events, and best practices. Daily rounding by Collaborative Care Teams helps identify potential safety hazards.

11. b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace health, safety, and security. These methods include a system-wide Safety Committeethat meets monthly to review progress toward goals, unexpected events, and best practices. Daily rounding by Collaborative Care Teamshelps identify potential safety hazards. Judgmental Nugget: Key point based on Criteria Polite? Examples for clarity & context Professional? Prescriptive Indicates Relevance to this applicant Positive?

12. Creating a Feedback Ready Comment: 5.1 OFI a, b It is unclear how the applicant manages volunteers, physicians, and students to fully support its work. For example, it is unclear how volunteers are trained and managed and how approaches are deployed to precepted students and credentialed physicians. Approaches for managing these workforce groups may help the applicant deliver high quality care and services and maintain a safe, secure environment.

13. a, b It is unclear how the applicant manages volunteers, physicians, and students to fully support its work. For example, it is unclear how volunteers are trained and managed and how approaches are deployed to precepted students and credentialed physicians.Approaches for managing these workforce groups may help the applicant deliver high quality care and services and maintain a safe, secure environment. Nugget: Key point based on Criteria Polite? Examples for clarity & context Professional? Indicates Relevance to this applicant Positive? Judgmental Prescriptive Avoid starting OFI with the “While” or “Although” phrase

14. Agenda Review • Quick Review • Independent Review Process • The Consensus Process • Creating Consensus Comments and Rationale • Feedback (Applicant) Ready Comments • Practice, practice, practice • The Organizational Profile Assessment • Examiner Roles, Ethics & Conflict of Interest • After Consensus • Key Themes • Introduction to Site Visit

15. The Consensus Process Incorporating Team Feedback into Comments

17. The Consensus Process Creating Consensus Comments & Rationale

18. What is Consensus? The Gospel according to Baldrige states… Consensus is a form of agreement reached by a group of people resulting from discussion and consideration of everyone’s ideas, ultimately bringing the group to agreement on a particular decision all can live with.

19. Why have Consensus? • Resolve diversity in individual evaluation • Key Factors • Strengths and OFIs • Score • Key Themes • Pooling of information adding value to overall knowledge: • All members exposed to pool of knowledge • Achieving basic agreement on Strengths & OFIs • Weigh overall findings to assign score

20. Why have Consensus? (cont.) • Consensus comments used • As basis for site visits • For feedback report if no site visit • Used by Judges in some cases to determine site visit or site visit purpose

21. What Changes in Consensus? • Item Leads: • ~3 Criteria Items lead by each Examiner • Incorporate the input of other team members • 4-6 Key Factors • Most important ideas/evidence • Output • Feedback Ready Comments & Rationale • Score based on Consensus Comments

22. Creating Consensus Feedback Examiner A Independent Review Consensus Comments Rationale Examiner B Independent Review Examiner C Item Lead Examiner C Independent Review Examiner D Independent Review

23. Consensus Review Feedback Report Independent Review The Consensus Process Consensus Key Factors Examiners “send” Independent Review to Item Lead Item Lead & Back-up Identified Item Lead creates V2 Draft Back-up Review & Feedback Item Lead creates V1 draft (V3 Draft) Consensus Meeting Consensus Output Consolidated Team Review & Feedback “Virtual” Consensus

24. SN

25. “Virtual” Consensus ProcessV1 Draft • Re-familiarize yourself with Criteria Requirements for assigned item • Develop consensus 4-6 Key Factors • Re-familiarize yourself with Applicant Response • Aggregate all team member ideas/evidence • Determine most important for feedback • Draft ~6 Comments for each Item • Include Rationale for each comment • Propose Score based on Comments

26. “Virtual” Consensus ProcessV2 & V3 • Receive input from teammates • V2: Backup (and Team Leader) • V3: Full Team • Review Criteria Requirements • Verify/update 4-6 Key Factors • Review Applicant Response • Make Changes, Refine for Clarity • Read from the Applicant perspective • Update Rationale • Verify/update Score Would you be comfortable giving that feedback to the applicant?

27. Consensus Meeting • Entire Examiner Team participates • Discuss all items • Focus on topics Virtual Consensus did not close out • Key Factors • Key Themes • Strengths & OFIs by Item • Scores • Team Leader sets agenda • Item “Owner” leads discussion

28. Agenda Review • Quick Review • Independent Review Process • The Consensus Process • Creating Consensus Comments and Rationale • Feedback (Applicant) Ready Comments • Practice, practice, practice • The Organizational Profile Assessment • Examiner Roles, Ethics & Conflict of Interest • After Consensus • Key Themes • Introduction to Site Visit

30. Same content as in Independent Review Item lead reviews input of all Independent Reviews and decides on Consensus Key Factors

31. Significant ? “Double” Criteria Reference Feedback Ready Comment • Notes: • Unused Strengths/OFIs & Rationale • Enough detail to make a Strength/OFI • Rationale: Why is this a Strength/OFI? • Examiners having same/similar Strength/OFI • Examiners having conflicting Strength/OFI • Indication of ADLI/LeTCI • Notes to team members • Not provided to Applicant Capture all ideas from all Independent Reviews on Consensus worksheets

32. Score Range: Which of the 6 Scoring Bands best reflects feedback? Rationale: Use ADLI or LeTCI to explain Range selected Score Value: % Score within the Band – 5% intervals

33. Round 1 Practice: Consensus(Items 1.1, 1.2, 7.4) As a TEAM: • Review Criteria Requirements • Verify/update 4-6 Key Factors • Review Applicant Response • Identify ~6 Total Strengths & OFIs • Use Blank Item Worksheet • Develop Rationale for each • Verify/update Score Summarize your findings and share with group

34. Results vs. Process Items Process Items: Strengths and gaps in “How” • Approach, Deployment, Learning, Integration Results: charts, graphs, tables, and texton outcomes/outputs of processes • Importance to the applicant, alignment with key factors • Beneficial vs. adverse trends, inconclusive pattern • Performance relative to comparisons • Breadth of results: address key customer, market, and process requirements; address appropriate segmentation for the item. Comment on what results tell you that can be of benefit to applicant’s improvement • Levels, Trends, Comparisons, Segmentation, Importance • What is missing??

35. Practice One - ProcessInstructions • Using a round robin approach, capture all the strengths and opportunities that you noted or recall from your reading. • Test each observation for its relationship to major ideas in the Criteria, the key factors you identified, the value added content, and its potential impact on the applicant’s desired results. • Draft around six strength and OFI “nuggets”. • Prioritize your strengths according to their importance to the applicant; repeat with your opportunities. • Remember to take a holistic view of the applicant’s responses.

36. Agenda Review • Quick Review • Independent Review Process • The Consensus Process • Creating Consensus Comments and Rationale • Feedback (Applicant) Ready Comments • Practice, practice, practice • The Organizational Profile Assessment • Examiner Roles, Ethics & Conflict of Interest • After Consensus • Key Themes • Introduction to Site Visit

37. Writing Value-Added Comments “Beyond Criteria” ORGANIZATION ENVIRONMENT “Parroting” “Prescriptive” CRITERIA “Value-Added Comments

38. Comment Fundamentals • Comment Purpose: Communication • Clearly identify most important Strengths and Opportunities for Improvement (OFIs) • Provide feedback in a non-prescriptive, non-judgmental manner • Comments are based on what the Criteria is asking and what is important to the Applicant (Hint: Key Factors) • Comment Components: clearly & concisely state • What is Strength or Opportunity For Improvement? • Why is Strength or OFI important to the Applicant?

39. NEW: Comment Components Applicant Requirements • Traceability back to Criteria • Criteria references now listed in feedback • A concise opening sentence • Expresses a single thought—the “nugget” (or essence, or main point) • Some examples • The relevance or importance of the nugget to the application No prescribed order Can Applicant use each part of my comment?

40. Writing a Comment Application Organizational Profile Item Responses Criteria Requirements ADLI LeTCI your best judgment Call to Action Comment

41. Strengths and OFIs defined • A strength is a process, approach, or result that supports the applicant’s achievement of its desired results. • An OFI is a process, approach, or result (or lack of one of these) that creates a vulnerability in achieving the applicant’s desired results. Give the applicant the benefit of the doubt when appropriate (Your best judgment)

42. Comment Components:The “Nugget” Application Organizational Profile Item Responses Criteria Requirements your best judgment The characteristics of a well-written comment Precise Concise Shows insights

43. Sample “Nuggets”Item 5.2 • a(3) The alignment of individual performance goals with strategic goals contributes to the achievement of those goals. • b(1) The applicant’s approach to assessing workforce engagement is aligned with performance measurement and strategic planning. • a, b, c It is unclear how the applicant applies workforce performance management, engagement, and professional development processes to volunteers, students, and credentialed physicians/nurse practitioners. • c(2) The applicant’s evaluation of the learning and development system, which appears to be limited to a review of posteducation performance, may not allow the organization to fully leverage its investment in learning.

44. Comment Components:Examples Application Organizational Profile Item Responses Criteria Requirements your best judgment

45. Sample “Examples”Strengths Item 5.2 • a(3) The alignment of individual performance goals with strategic goals contributes to the achievement of those goals. The applicant uses resident outcome data to drive high workforce performance by linking outcomes to the performance evaluation and compensation structure, which is unique in the industry. • b(1) The applicant’s approach to assessing workforce engagement is aligned with performance measurement and strategic planning. APEX scorecard results that are considered indicators of workforce engagement are reviewed regularly, and the organization addresses key workforce engagement findings within strategic planning.

46. Sample “Examples”OFIs Item 2.1 • a, b Tillingate Living’s strategic planning and objectives do not appear to address all strategic challenges or balance all stakeholder needs. For example, it is not clear how the strategic objectives (Figure 2.1-3) address the challenge to integrate existing practices with ACOs or how physician partners and suppliers are systematically included in the SPP. Such gaps may prevent Tillingate Living from being a top choice for care. • a(1) It is not clear how Tillingate Living systematically identifies and refines its core competencies. For example, how the core competency of designing and delivering rehabilitation services was systematically determined during step 3 of the SPP is unclear. Without a defined approach in this area, Tillingate Living may miss the opportunity to leverage these strategically important capabilities for future success.

47. 5.2 Workforce Engagement – Tillingate OFIs • a, b, c It is unclear how the applicant applies workforce performance management, engagement, and professional development processes to volunteers, students, and credentialed physicians/nurse practitioners. Without engaging the entire workforce, the applicant may miss opportunities to maintain its excellent reputation, especially related to the key family requirement of an attentive staff. • c(2) The applicant’s evaluation of the learning and development system, which appears to be limited to a review of posteducation performance, may not allow the organization to fully leverage its investment in learning. A systematic approach in this area may help the applicant maintain its core competency of developing clinical and service competencies for a caring and exceptional staff. OFI Examples help to clarify exactly what the gap is

48. Comment Components:Relevance/Importance Application Organizational Profile Item Responses Criteria Requirements your best judgment Comment Relevance demonstrates how well we know the applicant

49. Sample CommentsItem 2.1 • a(1) The five-month, 14-step strategic planning process (SPP) enables Tillingate Living to learn about residents’ needs and expectations and determine its strategic challenges and advantages. The SPP considers resident needs and minimizes blind spots through an environmental scan and SWOT analysis (Figure 2.1-2). The short- and near-term horizons allow responses to senior living trends and corporate/facility performance. Resident and Family Councils now give input to the process. Note: ADLI is not called out but evident Relevance/Importance

50. Sample CommentsItem 2.1 • a, b Tillingate Living’s strategic planning and objectives do not appear to address all strategic challenges or balance all stakeholder needs. For example, it is not clear how the strategic objectives (Figure 2.1-3) address the challenge to integrate existing practices with ACOs or how physician partners and suppliers are systematically included in the SPP. Such gaps may prevent Tillingate Living from being a top choice for care. How can we make this relevance more motivating to act on??