Unit 5

1. Unit 5 Schedule Lecture: Monday, 10/15 and Wednesday, 10/24 Exam: Monday, 10/29 No class, fall break: Wednesday 10/17 ME1: Monday, 10/22 Last day to withdraw w/o academic penalty: 10/29 Feedback, cont. and Goal Setting

2. Schedule revisited by date • Monday, 10/15: Lecture, U5 • Wednesday, 10/17: No class, fall break • Monday, 10/22: ME 1 • Wednesday, 10/25: Lecture U5 • Monday, 10/29: E5 • Monday, 10/29: Last day to withdraw w/o penalty

3. NFE: Crowell et al. • My main reason for including this article was its implications for task clarification and feedback • Feedback includes task clarification, so task clarification was examined first • If the only function of feedback was as an antecedent, one would not expect additional increases in performance when it was implemented after task clarification • Second reason (related to the first) was because of the authors’analysis of whether feedback functioned as an antecedent, a consequence, or both (rarely done, astute analysis) • Third reason: Demonstrates the importance of evaluative feedback in an applied study (per Johnson, 2013) (pick up with feedback)

4. Crowell et al: Important Results 1. Task clarification improved performance 2. “Objective” feedback improved it further but 3. Praise improved it again (evaluative feedback) Task clarification alone only results in modest increases; Objective feedback should be combined with evaluative feedback/consequences (note that these results re objective feedback are consistent with the Johnson article from Last unit and thus emphasizes that the difference between objective and evaluative feedback is important)

5. NFE: Crowell et al. overview • 6 bank tellers • 11 customer service behaviors defined • Quality points assigned to each depending upon importance • 100 total points possible, with 85 as the minimum acceptable • Task Clarification • Memo from management identifying the target behaviors and quality points (sos on your own)

6. NFE: Crowell et al. overview • Feedback • Posted individual charts (graphs) with individual point scores, daily, codes for names • Fake scores for any teller that was absent to protect confidentiality • Some “evaluative” feedback was used in the sense that the minimum acceptable point score, “85”, was highlighted on the graphs • Supervisor met with each teller individually when the teller came to work, but “descriptive” not “evaluative” • Praise added • Supervisors praised tellers if the point score was above 85 or below 85, but higher than the preceding score

7. NFE: Crowell et al. results last 6 sessions, > 85 only phase mean > 85 Above standard performance was obtained and sustained only when both feedback and praise were added to task clarification (20-day suspension before baseline 2)

8. NFE: Social validity not definitive, but actual hard datafacts, not survey data – cool! • Dollars on deposit in bank • 24 to 42 million, 75% growth • Customer complaints • 2-3 per month, dropped to near-zero level • Compliments increased (no customer input, so no certainty customer perceptions and behaviors were affected by the intervention, but …; results continued on next slide)

9. NFE: Social validity not definitive, but actual hard datafacts, not survey data – cool! • Bank managers rolled-out the program • Streamlined and implemented it in 6 other branches • Program was being implemented in the remaining 5 branches at the time the study ended • Cost • Annual cost of program as implemented: $6,000.00 • Streamlined version in other 11 branches, an additional $16,000 • This included the cost of a dedicated full-time program administrator/observer (streamlining next slide)

10. NFE: Social validity not definitive, but actual hard datafacts, not survey data – cool! • Streamlined version • Reduced teller observations from daily to 3X a week • Reduced the number of transactions recorded per teller in each session from six to three • Once praise was introduced, a maintenance procedure was suggested in which the frequency of recording be further reduced to one session per week (reduce labor intensiveness and number of observers required; ok last slide on this, moving on to Gatani & Johnson)

11. Intro: Gaetani & Johnson, cash shortages • Purpose: Decrease cash and inventory shortages in a retail beverage chain (employee theft?) • Assessed the effects of data plotting (self-recording), praise, and state lottery tickets • I’m including it because of the comparison of self-recording/praise/lottery tickets and as another example of a lottery system; not expensive but effective • Unusual lottery system in that each mgr who met the criterion received 2 state lottery tickets (only $1.00 a piece!) • Most lottery systems employees who meet the criterion have a chance to win a prize, but same basic principle (also including it – tie in with the Johnson study from last unit – address that in a moment)

12. Intro: Gaetani & Johnson, cash shortages • Participants: 12 store managers • DV: Efficiency estimate (SO4) • Very nice measure because it equated high sales volume and low sales volume stores • Also a very nice measure because it was computed from data the stores were already collecting – BIG PLUS because (include the following points in your answer): • No additional data collection system was required • Baseline measures could be calculated using archival data, so there wasn’t any delay in implementing the intervention (one of few studies that has examined supervisory praise alone – without specific graphic feedback – weekly phone conversations – this is the tie-in with the Johnson study from last week; praise alone without objective feedback)

13. Intro: Gaetani & Johnson, cash shortages • IVs • Data plotting alone (closed circles on graph) • Praise alone (open triangles on graph) • Data plotting and praise (closed triangles on graph) • Data plotting, praise, and lottery tickets (closed squares on graph)

14. SO7: Results, least effective and most effective IVs • Least effective • Data plotting alone • Supervisory praise alone • Most effective • Data plotting, praise, and lottery tickets • Group A stores, 123% above baseline • Group B stores, 108% above baseline • Group C stores, 40% above baseline • Group C, no prior exposure to self-recording or praise as Group A and Group B managers, so prior exposure may well have affected the results (in behavior analysis, we sometimes forget that the sequencing of interventions may affect results: carry-over) (skipping to SO7– careful with wording; data plotting + praise better)

15. NFE, but: remember your consequences! • Self-monitoring alone does not seem to be a powerful intervention • Richman et al. study, look at in U7 • Gaetani et al. (1983): The study I recommended but did not include in course pack • Supervisory praise (evaluative feedback) by itself does not seem to be a powerful intervention (need objective feedback as well) • Objective feedback (self-monitoring) itself may not be powerful w/o praise or some type of evaluation (also per Crowell study) • And once again, the combination of feedback, praise, and tangible rewards appears to be the most powerful (note again, the consistency of these results with Johnson- that’s why I like that article so much; it ties in and Explains the results of some of these applied studies – more controlled study; moving to the next study)

16. Intro: Aljadeff-Abergel et al. • First study, that I know of, to demonstrate that when training new skills, providing feedback before performance improves performance more than providing feedback after performance • (Should we change the term to feedforward rather than feedback when it is given before the performance? Perhaps, but there would be no feedforward unless it was also feedback…) • As the authors point out, in many (most) applied settings, it is difficult to determine whether feedback improves performance because it follows the performance or because it precedes the performance (acts as an antecedent prompt) • The main reason for that is because with ongoing performance (making widgets, interacting with customers, etc.), feedback follows one instance of performance but precedes the next instance of performance • And, an added complication, not mentioned by the authors is that there is often a time delay between (a) the performance and feedback and (b) between the feedback as an antecedent and the next instance of the performance (Exquisite study)

17. Intro:Aljadeff-Abergel et al. • Participants: 4 undergraduate practicum students who worked with 2-4 children in a group • Setting: Pre-school for typically developing children • Material: Direct Instruction lessons • DVs • Percent of correct error corrections • Rate of specific praise • IVs: Feedback before or after 30—min DI sessions • Design: Adapted alternating treatment design • Baseline: 3 sessions • FDB Before for one DV and FDB After for the other, alternating sessions

18. (these are sweet data! Clearly show FDB before was much more effective)

19. SO15: Tosti (1978, 1986) in the Land of Ago,When to use feedback before and after • Formative feedback • Feedback to teach new skills or change the quality of current performance, by correcting, guiding and improving it • Should be delivered immediately before performance to have the most impact • Summative feedback • Feedback to maintain or increase quantity of performance • Should be delivered immediately after performance I have always liked that distinction and believe it is on target, although there may be some qualifying factors, assuming we are dealing with highly verbal adults. For example, it may not matter if feedback is provided before or after (a) if the skill to be learned is not very complex and can be easily learned or (b) if the performance is already fluent (Wrap up: Tosti was an early behavior analyst, originally focused on programmed instruction and educational material – he was one of the early NSPI guys – defined/classified two types of feedback/language is from program evaluation)

20. NFE: This distinction is also consistent with basic programmed instruction and instructional design principles • Prompt new performance and gradually fade the prompts as the learner becomes more and more proficient • Use evaluative feedback after the performance to reinforce it and strengthen it (remember, a lot of the early pioneers in our field of OBM got their start in programmed instruction, so I would venture to say, most, if not all of the early generation OBMers, and those of us taught by them would agree with this distinction, but somehow it may have gotten lost…moving on to Crawley et al.)

21. Intro: Crawley et al. • One reason I included this article is because it is the best one I have ever seen with respect to improving sales behaviors and I would wager that most many of the behaviors identified in their exquisite analysis would generalize to other sales positions • Another reason I included it is because it is an excellent applied case study that includes feedback (prompts) before performance to train/improve performance (per Aljadeff-Abergel) and evaluative feedback delivered as soon as possible after the critical performance • Study was conducted by one of Ed Feeney’s consultants, Bill Crawley (I didn’t stress Feeney’s accomplishments in U1, but I recommend that you go back and read the Dickinson article for an historical perspective) • (NFE) Note the analysis at the beginning that was designed to determine the best opportunities for intervention, based on both the potential for improving performance (exemplar performer vs. average performer) and the economic pay-off of intervening on the performance (old study)

22. SO16: What approach was not successful in identifying what made sales reps effective? • Surveys were sent to the top sales representatives in the country asking what is was that they did that made them so effective • This approach did not work because sales representatives could not describe the behaviors that made them successful • I am friendly • It’s genetic - my parents were sales representatives • You need to be “up” • You need to be aggressive • General point Even though workers are exemplary workers, they often can’t tell you what they do that makes them exemplary workers. Those behaviors are often contingency-shaped (controlled by direct-acting contingencies) and employees never have had to describe them (describing what you do and doing what you do are different behavioral repertoires) • Automobile mechanics • Construction workers

23. SO17 (NFE): Exquisite specificity of the targeted behaviors • To determine the behaviors • They observed 65 top performers • Over a four month period of time • For 1,000 hours • Both inside the store and at in-home sales calls • Recorded the stimulus-response sequences • That is, what were the antecedents that prompted a response by the sales representative, and how did the sales representatives respond to those antecedents • Also interviewed customers for 50 hours • Pilot tested the entire intervention in two stores • First with the consultant as the coach • Then with the store manager as the coach (fidelity - did they create an intervention that could be carried out by employees)

24. SO17 (NFE): Exquisite specificity of the targeted areas and behaviors, cont. • Eight areas with 5-11 behaviors in each area • Customer greeted • Customer needs identified • Needs matched to store product and service benefits • Objections identified and overcome • Decision maker identified • Close made • Results of sales contact • Follow-up action taken (48 behaviors in addition to smiling, eye contact, natural voice, and use of customer’s name in each area)

25. SO17 (NFE): Exquisite specificity of the targeted areas and behaviors, cont. • Examples of behaviors in the Customer Greeted area • Customers should be approached within 120 seconds after entering the store • The sales representative should stand within 3-4 feet of the customer, smile and maintain eye contact • The sales representative should approach the customer at a normal pace and maintain a natural and relaxed posture • Introduce self using first and last names and identify his/her position • Obtain the customer’s name and use it throughout the sales interaction

26. SO19: Two reasons why commissions did not function as effective rewards Sales representatives received sales commissions monthly which most would assume would be sufficient to maintain high levels of performance • Commission payments were delayed, often by as many as 3 months, weakening the relation between sales and the amount of money earned • Commissions earned in January would not be received until March or April • Commissions were based on sales, an accomplishment measure, and sales representatives did not know the behaviors required to improve sales • The initial survey that failed to identify the critical target behaviors showed that sales representatives did not know what behaviors led to improved sales (skippingSO20 – on your own; Tom Gilbert, Human comp., 1978, accomplishments v. behaviors; daniels’ concern, systems v. PM)

27. SO19: I want to head off a problem and potential misconception In previous years, based on the point I just made about the fact that the commissions did not improve performance because they were based on accomplishments rather than behaviors, many students stated that incentives should never be based on accomplishments, but rather should always be based on behaviors. That is not what I meant/intended to say. I strongly believe it is preferable to incent/reward accomplishments because the accomplishments are what affect the “bottom line” of an organization; not the many behaviors that may be required to produce an accomplishment. However, if employees do not know how to perform the behaviors or do not know what behaviors are required to produce the accomplishment (as in Crawley et al.), then you cannot improve performance by incenting/rewarding accomplishments and need to “drop” back to behaviors. (next slide on this as well)

28. SO19: I want to head off a problem and potential misconception You have two options if you need to “drop” back to behaviors • Incent/reward the behaviors • Train the employees so that they are able to emit the behaviors required to produce the accomplishment, and incent/reward the accomplishment • If you are using monetary incentives, I much prefer this approach because (a) it is very difficult and too labor intensive to measure behaviors and (b) again, accomplishments are usually more directly tied to business goals/needs (next slide on this as well)

29. SO19: I want to head off a problem and potential misconception Unfortunately, for those of you working in human service settings and schools, your interventions often must target the behaviors of staff rather than accomplishments because the “accomplishments” (consumer progress, meeting goals), are often not under the control of the behavioral technician, particularly if consumers are severely impaired. Therefore, the only way to set up a fair incentive/reward program is to directly monitor/observe the behavior of staff when implementing the protocol: Did they correctly implement the protocol, regardless of whether the consumer made progress? I will talk about this again in U7, the unit on OBM in human service settings. (last slide on this)

30. SO20: Why is it important to compare data to records for the same months in the preceding year? • As part of the analysis to determine the effectiveness of the program, they compared the sales data to sales records for the same month the preceding year. Why? • Sales fluctuate seasonally and monthly • February is traditionally a big sales month while December is traditionally a low sales month • In behavior analysis, we often use time series data (AB design) to determine the effectiveness of our interventions but • If you compared sales in February and it had increased in comparison to Dec and Jan, you may conclude that your program was successful when it was not • Alternatively, if you compared Dec data with Nov data (with traditionally higher sales), you may conclude your program was not successful, when indeed it was (including this just so you don’t just say “due to seasonal fluctuations” but add an explanation)

31. Goal-Setting

32. SO22: SMART goals (NFE) • Locke & Latham developed “goal setting” theory and have done some stellar work in the area (and are the acknowledged experts in traditional goal-setting theory • While originally, Locke did not believe feedback was important, over the years he adjusted his opinion about that based on his research • Thus, while we would not agree with Locke’s conceptual analyses (and, in turn, he is adamantly opposed to behavior analysis), once again, as with expectancy theory, we do agree about practical implementation • SMART goals (Rubin, 2002) • Specific, Measurable, Attainable, Relevant, Time-bound • I would add “accompany with feedback and consequences” Locke & Latham (Eds.) (2013). New developments in goal setting and task performance.

33. SO23A: Specific goals are better than general goals (Locke & Latham). Why from a behavioral perspective? OBM position: Goals affect performance only because of the consequences that follow behaviors that result in goal attainment. • When goals are specific • They specify the performance requirements • They specify the criterion for reinforcement/reward • Thus, both employees and managers can easily discriminate successful from unsuccessful performance • Goals function like task clarification in the sense that the employee knows exactly what good performance consists of • They also provide an explicit “evaluative” component (the performance criterion for reinforcement/reward) which, as I have indicated earlier, appears to be necessary for feedback to consistently and effectively influence performance (material is from an analysis by Fellner & Sulzer-Azaroff, 1984, jOBM. Time to update the literature review, do your best goals next; evaluation component does NOT have to be goals - could be achieved a number of ways, but goals “work” )

34. SO23B. What are the problems with “do your best” goals? • What about “do your best goals?” • They preclude objective assessment because no performance criteria are stated • Employees may set lower goals than the supervisor and anticipate rewards that they then don’t receive • Remember, most employees evaluate themselves better than their supervisor evaluates them (Research indicates that Do your best goals are no better than no goals at all)

35. SO24: Although difficult goals may lead to higher levels of performance, be careful! Goals should be realistic and challenging but attainable From a behavioral perspective. Why? There are 3 problems. 1. Goal: R (work hard) ––––> Sp (supervisory criticism) 2. Goal: R (work hard) ––––> Ext (don’t meet goal) 3. Goal: R (work hard) ––––> Sp (signs of failure) Explanation of #3 above. In our culture, signs of failure tend to be conditioned punishers. Think about it. Regardless of the criticism you get from a supervisor (or professor), when you fail to meet a goal or standard, how do you “feel?” Signs of failure tend to be “automatically punishing” - which punishes the behavior of working harder. (example in book, students who were failing set goals to get a 4.0; stretch goals, #2 Daniels’ Oops – 13 mangt practices that waste time & money operationally defined as those that are attained less than 10% of the time; students often get the first two, miss the third; 1st EOM programs)

36. NFE: How do you set “difficult, but attainable goals?” • In behavior analysis, we do not have a rule of thumb for setting goals • And, granted, it may depend highly on the task • Setting goals at a group level is even more difficult than setting goals at an individual level • A particular goal may be too high for some employees, just right for some employees, but too low for other employees (lab study, interesting study, interesting results)

37. NFE: Traditional Goal-setting theorists’ definition of difficult goals • Difficult goals, also referred to as stretch goals: goals that 10% of employees can meet • Note that this means that 90% of employees will repeatedly fail to meet the goal! • Challenging, but achievable goals: goals that 20%-50% of employees can meet • Note the large range • Also, note that even with this definition, 80%-50% of employees will fail to meet the goal! (Daniels calls a stretch goals – which, by the way, are very popular in business and industry))

38. NFE: Two behavioral studies show that goals that are too high hurt performance • Roose & Williams (JOBM, 2018) • A medium goal (150% of baseline performance) improved performance significantly more than a very difficult goal (175% of baseline performance) and a “do your best goal” • The difficult goal resulted in the same performance levels as the “do your best goal” • 7 of the 8 participants in the medium goal condition met the goal (which could account for the increase) • Only 1 of 8 participants in the difficult goal condition met the goal (which could account for the lack of increase) (very, very nice study; lab study!! Punishing or extinction effects….)

39. NFE: Two behavioral studies show that goals that are too high hurt performance • Jeffrey et al. (2012) Showed that ability-based goals were better than a high “one goal for all” • Determined performance levels for the task based on an extensive pilot study • Based on an initial performance assessment, divided participants into low performers, middle performers and high performers Complicated study: based on pilot study; decoding task – 3-digit number that they had to convert into a letter of the alphabet – were given the key; divided Ps into three groups based on initial session)

40. Jeffrey et al. (2012): nfe • Two conditions • One goal for low, medium, and high performers: 20% of all performers had met the goal in the pilot study • Ability-based goals: different goals for low, medium, and high performers • Low performers: 20% of low performers had met the goal • Middle performers: 20% of middle performers had met the goal • High performers: 20% of all performers had met the goal

41. Jeffrey et al. study (2012): nfe • Ability-based goals were more effective than one goal for all for low and middle performers • Low performers who were given the “low” goal performed significantly better than their counterparts who were given the higher goal based on the performance of all performers • Low and middle performers who received the one goal decreased their performance across 3 experimental sessions • 0% of the low and middle performers in the one goal for all condition met the goal (interpreted to mean, once again, punishing/extinction effects of goals that are too high ~35% of high performers did)

42. NFE: Goal-setting • The literature supports the notion that the individuals who are most likely to meet a goal are the ones who are performing below a goal but not way below the goal (difficult, but achievable) • But, again, how do we set goals that are difficult, yet achievable when workers are performing at different levels? • Tiered goals for the same behavior/accomplishment? That is, giving employees multiple goals for the same performance. • Successively increase goals? Start low and successively increase goals, so that performers are reinforced for meeting the goal(s) yet end with a difficult terminal goal? (Daniels’ position) (not typically feasible or fair in a work setting to set different goals for different performers)

43. NFE: Goal-setting • The literature supports the notion that the individuals who are most likely to meet a goal are the ones who are performing below a goal but not way below the goal (difficult, but achievable) • But, again, how do we set goals that are difficult, yet achievable when workers are performing at different levels? • Tiered goals for the same behavior/accomplishment? That is, giving employees multiple goals for the same performance? (not typically feasible or fair in a work setting to set different goals for different performers)

44. Raw Behaviors or 4 5 6 7 8 9 10 11 12 Weight Score Points Results Appropriate Greeting 65% 85% 90% 98% 80% 59% 60% 70% 75% 4 7 8 5 6 100 Sub-goals Total Pts. Baseline Goal Performance NFE: A behavioral example of tiered goals? The Performance Matrix Sample Matrix: Postal Worker Number of Bins Sorted 9 12 10 3 91% 93% 100% 96% 97% 98% 99% 92% Till Correct Number of Complaints 5 4 6 3 2 1 0 (not typically perceived this way, but conceptually it is. No research re its effectiveness versus just assigning the terminal goal in column 10 itself; Abernathy used this with his systems-wide incentive system; 12 orgs, 33% increase first year)

45. SO25: Practical dos and don’ts • When possible do use tiered (multiple goals) with successively increasing rewards for meeting each higher goal • Performance Matrix is a great tool for this • Do not have different goals for different individuals with the same tangible/monetary rewards – disaster (organizations typically can’t set individual goals – too labor intensive; but it may be possible 2-3 tiers; Pampino et al. did that in study in U2; Dan’s dissertation; last slide on this)

46. SO27: Assigned vs participative goals • The research indicates that participating in goal setting does not increase performance when compared to assigned goals • Three meta-analysis studies now support this conclusion, first one in 1986 • The key issue, thus, is not how a goal is set, but whether a goal is set (Back to the Sos: this is a very common misconception; issue keeps coming up at ABAI, with a student -from another program-arguing and maintaining that participative goals were better; just this summer I was asked about this by a consultant who works for a behavioral firm)

47. SO28A: Goals – what is the best combination? • We know that goals combined with feedback are more effective than either alone • Feedback enhances the effectiveness of goals • Goals enhance the effectiveness of feedback • Studies suggest that graphic feedback is the most effective type of feedback to use with goals: better than vocal or written • We know that goals and feedback are much more effective when consequences are provided • Not definitive, but it appears that monetary incentives/rewards are more effective than nonmonetary incentives/rewards THUS………. (next slide)

48. SO28A: What is the best combination of OBM interventions? (answer) • Goals • Graphic feedback that displays performance over time, preferably at least once a week • Some type of performance consequence, preferably monetary incentives (the same recommendation from Balcazar et al – just add goals when at all possible)

49. SO28B: Group goals • When using group goals, what factor should be taken into account? Group size • Group goals are more effective with small groups than large ones • However, we don’t know what the “critical” group size is • This would be a very interesting and valuable study to conduct • Goals, graphic feedback, consequences, while manipulating group size (alone, not combining them with individual goals; group size is an issue in ALL group contingencies.; recent meta-analysis with incentives..)

50. SO29 NFE: Possible behavioral functions of goals • Summarized the prevalent ones in SO30 • Good summaries and starting points if you want to pursue this: Tammemagi et al. (2013) Roose & Williams (2018)