Auditing Feedback Interventions: Development and Validation of a Measure

1. Brandon L. Young Dissertation Proposal Auditing Feedback Interventions: Development and Validation of a Measure

2. Overview • Performance management systems • Feedback and feedback Intervention Defined • What we know about feedback • What we do not know about feedback • Problems with current practice and research • Purpose of Current Study • Measure Developmentand validation process • Contributions

3. Performance Management • “a continuous process of identifying, measuring, and developing the performance of individuals and teams and aligning performance with the strategic goals of the organization” (Aguinis, 2009; p.3) • Impacts profits, returns on investment, and stock prices (Huselid, 1995). • Firms with strong performance management systems more likely to outperform their competitors (Bernthal, Rogers, & Smith, 2003)

4. Purpose of PMS • Motivate and develop employees by generating and delivering performance feedback (Cascio& Aguinis, 2011) • Communicate and negotiate performance expectations • Inform employees as to how well they are meeting those expectations

5. What is Feedback? • Information or data regarding performance (Latham & Locke, 1991) • Individual resource used for monitoring and inquiry (Ashford, 1986; Ashford & Cummings, 1983).

6. Feedback  interventions (FIs) • “actions taken by (an) external agent(s) to provide information regarding some aspect(s) of one's task performance” - Kluger& DiNisi, 1996; p. 255 • Embedded in performance appraisal, assessment centers, training, selection

7. Why Feedback? Consensus among practitioners that, “feedback is an essential feature of all stages of the performance review process.” -Performance Management: American National Standard published by the Society for Human Resource Management (2012, p. 20)

8. Feedback Intervention Effectiveness • Objective reviews • Consistent or mixed effects in 99% of reviewed applications (Alvero et al.,2001) • Meta-analyses • Improved performance by .40 SD (Kluger & DeNisi, 1996) • Improved productivity by .50 SD (Guzzo et al., 1985) • 68 out of 70 primary studies showed productivity gains (MBO; Rodgers & Hunter, 1991) • Productivity under ProMES feedback 1.16 SD higher than baseline (Pritchard et al., 2008)

9. Here’s the thing… • Feedback does not uniformly improve performance (Alvero et al., 2001) • More than one-third of the 607 calculated effect sizes indicated negative performance effects (Kluger & DeNisi, 1996) • Effect sizes ranged from -2.53 to 5.37 (Pritchard et al., 2008)

10. What don’t we know? “in spite of the large and varied literature, generalizations about the effects of feedback on individuals are few.” - Ilgen, Fisher, and Taylor (1979)

11. Why don’t we know what we don’t know? • Research focused on • psychometric properties of performance appraisal rating systems(Tziner & Latham, 1989), • accuracy and rater errors (Bretz, Milkovich, & Reed, 1992; Saal, Downey, Lahey, 1980) and • rater bias during the appraisal process (Landy & Farr, 1980). • Clouds the more important goal of performance management; giving FEEDBACK to guide PERFORMANCE IMPROVEMENT

12. Where should we focus? • Feedback strategies are only as effective as the user’s ability and willingness to use them(Lizzio, Wilson, and MacKay, 2008) • Research support that reactions to feedback interventions are critical to effectiveness (e.g., Dipboye & de Pontbriand, 1981; Bernardin & Beatty, 1984; Murphy & Cleveland, 1995) • Reaction criteria are of more interest to practitioners than the psychometric properties of performance appraisals (Thomas & Bretz, 1995; Murphy & Cleveland, 1995; Keeping & Levy, 2000).

13. What’s the problem? • Attitudes toward PMS are generally unfavorable. • 13% of managers and 6% of CEOs believed that their organization’s current performance management system is useful (Leadership IQ, 2005). • More recent survey results • 34% of participants felt they received useful feedback from managers during performance reviews. • 26% of participants indicated the receipt of regular performance feedback would motivate them to stay in their current positions. (CornrstoneOnDemand, 2013)

14. Problem with Current Approaches • Complex interactions between characteristics of feedback interventions • Content • Processes • Contexts • Left with ill means of measurement

16. Purpose of Current Study • Comprehensive, parsimonious means to systematically audit feedback interventions • Perceptions regarding the properties of; • Performance measurement • Feedback content • Feedback delivery • Source • Organizational Support

17. Measurement Characteristics Measurement Performance Standards • Observation • Judgment

18. Feedback Characteristics

19. Contextual Factors

20. How will this be accomplished? • Scale development and validation • item generation and SME review • establishing factor structure & internal consistency • examining convergent and discriminant validity • testing the empirical validity of the instrument • examining relationships with reactions to feedback

21. Item Generation • Items refer to characteristics of system components • Performance standards… • The feedback I receive… • Feedback is delivered… • My supervisor… • The organization… • At least two items per characteristic • 7-point agreement scale

22. Item Review • At least 5 SMEs will review instructions and items for clarity • Flesch-Kincaid Grade Level test • At least 5 SMEs will review item content • Retranslation process

23. Scale Development • Revised items will be administered • Item total correlations and alphas will be calculated • Principle axis factor analysis, promax rotation • Items will be revised or eliminated based on analyses and domain coverage

24. Factor Structure • Revised items will be administered • Four models will be tested with CFA • One-factor model • Two-factor model (i.e., System, Contextual) • Four-factor model (i.e., Measurement, Feedback, Source, Organizational Support) • Five-factor model (i.e., Measurement, Feedback Content, Feedback Delivery Process, Source, Organizational Support)

25. Convergent & Discriminant Validity • Convergent Validity • Landy et al. (1978) • Burke et al. (1978) • Dipboye & de Pontbriand (1981) • Discriminant Validity • Brayfieldand Rothe (1951) • Michigan Organizational Assessment Questionnaire (Cammann, Fichman, Jenkins, & Klesh, 1979)

26. Reaction Mediated Model

27. Feedback Acceptance • Cognitive Reactions • Accuracy (5 items; adapted from several sources) • Fairness (4 items; adapted from Kendharnath et al., 2010) • Utility (6 items; adapted from Jawahar, 2010) • Achievability (3 items; adapted from Kendharnath et al., 2010)

28. Feedback Acceptance • Affective reactions • Satisfaction with feedback discussion (4 items; adapted from Giles & Mossholder, 1990 & Dobbins et al., 1990) • Satisfaction with feedback intervention (3 items; adapted from Giles & Mossholder, 1990) • Satisfaction with feedback source (3 items; adapted from multiple sources) • Affect toward the feedback intervention (12 adjectives developed by Zuwerink & Devine, 1996)

29. Behavioral Reactions • Motivation to use feedback (8 items; adapted from Dorfman et al., 1986; Kendharnath et al., 2010) • Motivation (Pritchard, 2006) • Effort – amount of energy exerted (3 items) • Direction – how effectively effort is applied (8 items)

30. Control Variables • Demographics (i.e., age, gender, race, tenure) • Recalled task performance • Job Satisfaction (8 items; Brayfieldand Rothe, 1951; Cammann, Fichman, Jenkins, & Klesh, 1979)

31. Contributions • Theoretical • Holistic approach to exploring feedback intervention characteristics • Mediated Model • Practical • Diagnostic tool for auditing feedback intervention