Human Service Settings

1. Schedule: Lecture: Monday, 11/12 and Wednesday, 11/14 Exam: Monday, 11/19 No Class, Thanksgiving Recess: Wednesday, 11/21 PSY 6450, Unit 7 Human Service Settings

2. Human Service Settings, into • Articles • DiGennaro Reed & Henley reports the results of a survey conducted with 382 BCBAs or aspiring BCBAs about training and staff management procedures where they are employed; the article documents why OBM is so needed in HSS! • Richman et al. has (a) a very nice simple measurement system, (b) demonstrates that in-service training is not effective, and (c) self-monitoring alone is not consistently effective over time • Carbone staff incentive system – only on ppt; this is a description of an excellent monetary incentive program at a center for autism; only example I know of but there may be more (ba students work in Hss, asked if I could include low-cost interventions; usually target client training, paperwork is very important)

3. Human Service Settings, into • Parsons et al. is the model for a large scale PM and systems intervention in residential treatment facilities • Gil & Carter is an example of a very successful large scale group feedback and goal setting intervention in 13 homes with 200 DCS; it’s unique features include a group feedback procedure, group social comparison feedback, and no monetary or cash-value incentives/rewards • Green et al. increases supervisory staff satisfaction with their most disliked task through a simple, but effective intervention

4. Staff Management,intro Staff management, while similar to OBM interventions in business and industry, offers some unique challenges • Few professionals in human services are trained in staff management; rather they are trained to develop effective training and behavior management programs for their consumers • Most individuals who obtain graduate degrees to work in human services end up as supervisors or managers – they do not implement the programs with the consumers themselves, rather they supervise those that do 4

5. Staff Management, intr0 • Also, many professionals in HS manage several different units or programs and several have started their own organizations, schools, and consulting businesses but again they have no or little training in staff management or organizational systems analysis • Yet, it is quite clear that no matter how well designed a training or behavior management program is, unless it is implemented correctly by staff, the consumers will not benefit Important Caveat: The relatively new requirement that BCaBAs and BCBAs who supervise prospective certificants must have a certain number of hours in supervisory training may help solve this problem but most people in HSS whom I have talked to said it won’t because training is focused on what the BACB requires in order to be approved as a supervisor. 5

6. Staff Management, intro • In business and industry, it’s understood (usually) that supervisors/managers need to know how to supervise their employees and organizations promote, hire, or train employees based on that; that is not typically the case in HSS • In HSS, supervisors and managers are promoted/hired because of their clinical skills • It is a given that technical knowledge does not a good manager make (remember Komaki from U2) • All mid- to large-size business organizations hire experts in training, performance management, and organizational systems analysis, usually in the human resources dept.; HSS have not done that • A qualification: four of our OBM graduates have recently been hired into OBM jobs in HSS but that is not the norm (At WMU, over the years, no idea how many of our graduates in human services have told me that they wished they had taken all of our OBM courses while in graduate school here) 6

7. Challenges for the organization • Direct care staff are often not well trained when they enter the organization • The relatively new Registered Behavior Technician credential being offered by the BACB may help solve this problem – there is more training being offered now, particularly through third-party vendors • WMU’s Autism Center of Excellence has now developed RBT training, partnered with Foxy Learning • Interestingly, my colleagues who are supervisors/managers/human resource directors have been telling me that the direct care staff now often lack basic job skills (dress, attendance, getting along with others) • High turnover (often 60% or higher annually) • Imagine the organizational resources needed to hire and train new employees when you need to replace 60% of your staff each year! (recent development; )

8. Challenges for the organization • Critical measures are often the behaviors of the staff or clients, rather than accomplishments • Implementing training procedures correctly and when scheduled, correctly responding to client inappropriate behavior • Very labor-intensive as a result • Large number of staff • Residential facilities require 24/7 staffing • In day training programs (centers for autistic children) still need intensive one-on-one training for consumers (i.e., behavior challenges, verbal behavior training, functional living skills training) • Adds to the labor intensiveness for supervisors doing direct observations, performance assessments, and feedback (behavior is effervescent unlike accomplishments; need direct observation)

9. Challenges for the organization • The accomplishment of the staff really is the improvement and/or engagement of the client • Some studies measure both the behaviors of the staff and the client, but is this really feasible for an organization to do? • Can you hold a staff member accountable for the progress of a particular client when each client is likely to have different individual needs? • Reason why HSS tend to measure the behaviors of staff – which is a very reasonable thing: should not hold an employee accountable for something that is outside his/her control • In this unit, Richman et al. and Green et al. measured staff behavior while Parsons et al. measured consumer behavior • One question for managers/supervisors in HSS is whether to measure staff or consumer behavior (Same issue as behaviors as critical measure; a bit redundant– expanding this a bit; May have to alter protocol, frequency of observations/measurement)

10. Challenges for the direct care staff • Pay is typically low • Sometimes staff get “kicked, bitten, and scratched” • They often have little or no professional training before being hired • Again RBT training may help solve this! • Job responsibilities are often not well defined after hire • Often are excellent at daily care of consumers and scheduled activities (meal preparation, outings, etc.) • Not given adequate training or supervision • Often not given much guidance about what to do when there is “free time” for consumers 10

11. Challenges for the direct care staff What all of this boils down to for staff: Staff need job training and support that management is often not trained to give them. (don’t blame the staff, but you can’t blame the supervisors either!; new supervisory training required for BCaBAs and BCBAs may help solve this) 11

12. SO1: The Results of Lack of Supervisor Skills in Staff Management Research has consistently shown: • Developmentally disabled individuals who live in residential facilities or group homes spend ~65% of their time off-task • That is, not doing any meaningful activities or leisure activities • Direct care staff who work in such facilities spend ~45% of their time off-task • That is, not doing any work-related activities (Give you the references for this in the Sos; one of the most interesting things about these data is how consistent they have been – remained basically the same from the 60s to the 90s – I haven’t seen any more recent data – let’s hope this is changing!) 12

13. What three OBM courses should human services personnel take? • Performance management • PSY 6450, Psychology of Work • Personnel training • PSY 6440, Training • Organizational systems analysis • PSY 6510, Behavioral Systems Analysis 13

14. SO2: State the differences between human service professionals and professionals in business that help account for staff mgt problems • Human service professionals, have little or no training in training, performance management, and organizational systems analysis • There aren’t experts in the organization to help them, unlike in business and industry, who hire experts in training, performance management, and organizational systems analysis in “support” units 14

15. SO3: Why haven’t individuals been trained in staff mgt or OBM? • Failure to recognize the importance of staff mgt • Lack of availability of courses in staff mgt/OBM at the undergraduate and graduate level • Very few graduate training programs in OBM • OBM courses that are offered typically focus on business and industry and are marketed that way, thus students pursuing a career in human services don’t recognize the relevance of these courses 15

16. SOs 4-11: Intro, DiGennaro Reed & Henley • Purpose To document types of training and performance management practices that BACB certificants and aspiring certificants receive in their work settings • Participants • 382 BCBA-Ds, BCBAs, BCaBAs, and those seeking certification • Interestingly, only 52% of the participants indicated that their behavior analysis was their major field of study • Other areas included special education, education, school psychology, counseling, clinical, speech pathology, etc. (Sos are straightforward; and I am only covering them in lecture to emphasize the need for OBM) 16

17. SO5: Initial or Pre-service Training and SO7: Relevance of training • Percentage of respondents indicating that they received initial or pre-service training before working independently? About half!! • Percentage of those who received initial/training who reported that the training prepared them to perform successfully? 66% (that means ½ did not; as the authors state, that is alarming. Probably overestimate, BCBA-Ds; Particularly when combined with the fact that Only 66% who did receive training reported that the training prepared them to perform successfully which means about 1/3 did not) 17

18. SO5: Initial or Pre-service Training and SO7: Relevance of training If my math is correct, that means that 66% of respondents reported that either (a) they did not receive any initial orientation or training before working independently, or (b) the training they did receive did not prepare them to perform successfully on the job! 18

19. SO6: Training methods in initial training or pre-service training • The two most common training methods were verbal and written instructions about how to perform a skill • The two least common methods were practice in a role-play or rehearsal situation and practice with actual clients • Fewer than half who received practice reported that a mastery criterion was required So, the least effective training methods are being used most often and the most effective training methods are being used least often and even then, without a mastery criterion! Take Dr. McGee’s Training and Development course!!!!! 19

20. SO9A&B: Ongoing feedback • 66% of respondents said they received ongoing feedback • Which means that 34% did not! • Of the 66% who said they received ongoing feedback, the majority of respondents said they received it once per month or less • What is the most likely reason that a high percentage of respondents said that (a) they received ongoing feedback and (b) they received it once a month (or less)? 20

21. SO10: Monetary or non-monetary incentives or bonuses • 25% of respondents said they received monetary or non-monetary incentives or bonuses • Which means 75% did not! • Of the 25% who said they received incentives or bonuses about 30% said that they were not contingent on performance • Again, do the math: combining the above two, it means that 80% of respondents are either not receiving incentives or bonuses, or are receiving them non-contingently • 40% of respondents who said they received incentives or bonuses said they received them annually (it’s good that 25% said they received incentives or bonuses, but…or perceiving them to be noncontingent) 21

22. SO11: Supervisory training for supervisors • 75% of respondents said that they supervised staff • Of those, about 65% said that they did not receive any training on effective supervisory practices 22

23. To sum up…. BACB certificants and aspiring certificants are not being given effective training or support through effective performance management practices Why then would we expect them to perform well on the job? We, in OBM and those of you who are in the behavior analysis program and in this class, have a huge opportunity here!! And, for those in the behavior analysis program, I might add, perhaps both a moral and ethical responsibility per the BACB ethical guidelines to change things (sorry, that was a little preachy) (just add that these certificants, for the most part, are working in organizations that provide behavioral services to clients… They are not just your garden variety organization; most are staffed and supervised by other BCBAs and behaviorally trained clinicians! OK, moving on….now what to do about that – examples of good PM!) 23

24. Intro NFE: Richman et al. article • Simple, but effective measurement system • Demonstrates that in-service training is ineffective • Self-monitoring alone may have immediate effects but will not consistently sustain performance over time (redundant – I talked about this when I introduced this unit; study objectives are straightforward – touch on some of the main points)

25. Intro NFE: Richman et al. • Rationale of study To determine whether a self-monitoring procedure, with minimal supervisory involvement, could increase staff adherence to scheduled activities and on-task behavior • Participants 10 staff members in two houses of an intermediate care facility (group home) for the developmentally disabled

26. SO12A: What two general categories of behavior were recorded? • On-schedule behavior • Is the staff member in the assigned area for the scheduled activity according to the posted schedule? • Does the staff member have all of the materials necessary to conduct the activity? • 12B Regardless of whether the staff member was actually implementing the task (that is, the staff member could be off-task in the sense of chatting with another staff member, drinking coffee, or just interacting “generally” with the clients; as long as the person was there as scheduled with right materials) • On-task behavior • Is the staff member engaged in behaviors for any of the three appropriate activities (group, client/house custodial, or one-on-one training) • 12C Regardless of whether the staff member was implementing the specific activity that had been scheduled (in other words, even if the staff member was doing group training when one-on-one training was scheduled; staff member ) (very nice measures of behavior, simple; apologize for the crowded slide – needs to be on one)

27. SO16: Self monitoring increased performance substantially. Why, then, was supervisory feedback added? (First, note lack of effectiveness of in-service – SO15 – Not just a self-monitoring procedure – turned daily schedule cards into the Supv. at the end of the day; implication of fdbk; again; yes, they did get further increases, but the main reason -next slide)

28. SO16: Why add supervisory feedback? • The behavior of 5 of the 10 staff members became variable over time (that’s 50% of the participants) • Supervisory feedback improved both on-schedule and on-task behavior for each of the 5. • Demonstrates the importance of supervisory feedback and evaluation • Also suggests that self-monitoring may be effective on a short-term basis but may not be effective long-term • But, why would we expect self-monitoring to be effective over the long run? • What consequences are there for self-monitoring or for the self-monitored performance? • Revisit U5 on self-monitoring (question: are we doing more harm than good when we publish short term studies that indicate that interventions are Successful, particularly when the results don’t seem to conform to a solid behavior analysis? Carbone next)

29. Carbone staff incentive system • Carbone Clinic, center for autism • 10 instructors, 2 classroom supervisors, 40 learners with autism and other disabilities, age 2-14 years • In 2014, Carbone moved to FL and is no longer the head of the clinic He is continuing his Dubai school and his consulting

30. SO18: Two components that the incentives are based on • Instructors earned a monthly bonus totaling $300a month, $3,600 per year, for exemplary performance • There are two incentive components, independent • $150 based on supervisor observations of training skills • $150 based on accuracy of child’s program/data book • Bonus is publicly announced at the staff meeting that follows the assessments • Checks are given to staff at that meeting (monthly may be an adaptation to HSS because of labor intensiveness; for paperwork – often only based on clinical training, but paperwork is very important)

31. Carbone incentive system, cont. • Names were publicly posted in the staff dining room • Performance scores were referred to in the employee’s annual review • Performance scores contributed heavily in determining the size of annual raises and future promotions components: Observation and feedback, $ incentives, goals/criterion for incentives, supervisory and public recognition – all of the components of an effective mgt system)

32. SO19: What’s so cool about this system? • Everyone who meets criterion can earn the incentive (and there is criterion/goal) • Significant amount of money • Public recognition at the next staff meeting • Separate check – the money doesn’t get “lost” in the person’s regular paycheck • Embedded in the management system – used to determine pay increases and promotions (list any three)

33. Details: Performance observation, this slide, nfe • Following training, unannounced monthly assessments of performance were conducted • Supervisor observed instructional sessions using three to four competency checklists • Natural environment teaching • Discrete trial teaching • Teaching adaptive living skills • Teaching vocal manding • Implementation of behavior reduction protocols (He is willing to give copies of these checklists to individuals who are interested)

34. SO20: Details: Performance observation, criterion & remediation • Supervisors gave vocal feedback to instructors after the observation sessions • Instructors must have scored 90% on each of the 3-4 checklists used, with nocritical errors to earn the incentive • If instructors did not meet the criterion, supervisors coached instructors and repeated an assessment of those competencies approximately one week later

35. Results NFE: Carbone incentive system • When staff were stable (no new hires) and staff were not assigned to new learners (who may have new competency checklists, thus staff is still learning the protocol): • 75% to 85% of staff earned the bonus for teaching/training per month • 25% to 65% of staff earned the bonus for accuracy of the child’s program/data book (relatively old data, 2007; training percentage is considerably higher than accuracy of books; exquisite system and and data: 75-85% of the staff were performing at least 90% of the checklist items correctly With no critical errors – I wonder how many other agencies can say that about their direct care staff )

36. Limitations: Carbone incentive system* • No experimental design to assess the effectiveness • But, replication across new instructors • $$ paid out increased over the years as percentage of instructors who met criterion increased • No outcome data related to changes in learner behaviors (Carbone has developed this list; Parsons next)

37. Parsons et al. article • This is the best study I have seen about a large scale OBM intervention in a human service setting • Complete systems intervention, not just OBM, which makes it different than the next article, Gil & Carter • The study was conducted in five group homes for the developmentally disabled • In the study objectives, I point out some very useful procedures that could be implemented in any human service setting although clearly some of the details of the procedures would have to be modified • Implemented a total system intervention package • Problem? It was very labor intensive

38. Parsons intro, cont. • There are two experiments • I don’t have any SOs over the first one because I wanted to focus on the intervention, but part of the beauty of this work is having the normative data from the first when analyzing the results from the second

39. Overview of Experiments 1 & 2 • Experiment 1 • Benchmarking study on treatment and services • 22 living units in six state residential facilities • 18 were certified as intermediate care facilities under Medicaid (which means services can be reimbursed through Medicaid) • Experiment 2 • Purpose was to develop and implement a comprehensive management system to improve treatment services in five group homes • Group homes were Medicaid certified • Medicaid had reviewed services and the facilities had been given a time-limited mandate to improve services or face de-certification. Improvement was critical - “critical business issue”

40. NFE: Results of E1, benchmarking • On average, what percentage of resident behavior was off-task? 67%!! (range 0-100%) When developmentally disabled clients are in group homes, 2/3 of their time is spent doing things that do not help them. This suggests that residential facilities are not fulfilling their active treatment obligations • On average, what percentage of resident behavior was active treatment? 19% (range 0-40%)

41. Organizational structure, staff, and residents 110 Direct care staff 165 Residents One of my purposes with the SOs is to point out the systems aspects of the program - they implemented monitoring and feedback systems for individuals at EACH level of the organization - we often intervene at the direct care staff level, but who provides PM to the group home supervisors, and to the supervisor of the group home supervisors? We forget to do that, yet are often surprised our interventions don’t last

42. Intervention NFE: Four basic components • Structure (scheduling) and reassignment of staff • Staff training • Monitoring of staff performance • Supervisory feedback

43. SO22: What PM benefits are derived from scheduling and reassignment? • Reassignment and scheduling alone or in combination are common interventions in human service settings • What are the benefits? • Task clarification (specification of what they are supposed to be doing and when) • Decreased conflict with other responsibilities • Individual accountability • Individuals can be identified • Their performance can be measured and evaluated • Their performance can be consequated (this is important – common that staff are not scheduled, everyone is just supposed to pitch in as needed; and know when it is needed)

44. Each staff member initialed the checklist I am pointing this out because this is basically the same procedure used by Wilk & Redmon and it permits the assessment of the integrity of the intervention without observers. Remember this procedure! SO23A&B • 23A How often did each supervisor or assistant supervisor observe each staff person? Once a week • 23B What procedure was used to verify that the supervisor(s) observed and gave feedback to the staff member immediately after the observation? (these are, of course, straightforward, but I want to emphasize b)

45. SO24 NFE: Systems approach, (a) the program director gave feedback to the area director; (b) the area director gave feedback to the group home supervisors, (c) supervisors gave feedback to DCS • The data on resident behavior collected by researchers (independent of the measures related to staff observations) were summarized and graphed, and sent to the program director weekly. • The program director sent the graphs along with comments to the area director, who then sent the appropriate graphs to each group home supervisor • Note two separate and independent measurement systems • Were supervisors observing and giving feedback to the direct care staff • How was the supervisory system affecting resident behavior - was it decreasing resident off-task behavior and increasing active training • Also note that the resident behavior data were collected by: • 8 staff members • Student interns (number wasn’t specified) • Extremely labor intensive (also the systems aspect – everyone in the hierarchy was involved – top to bottom)

46. SOs 26&27: Back to why the normative data from Exp1 was so important SO26: What very nice contribution does the normative data provide when analyzing the results of the study? • Most studies would have reported the improvement in resident behavior in comparison to baseline • During baseline off-task behavior averaged 64%, which decreased to 41% during the PM intervention • That looks like a nice decrease (23% decrease) but residents were still off-task 41% of the time (cont. on next slide)

47. SO26, cont. With the normative data they could also report • Their baseline average was similar to the average off-task behavior in the 22 other group homes (18 of which were Medicaid certified): 64% and 67%, respectively (so maybe they weren’t doing that badly to begin with!) • Not only did off-task resident behavior decrease considerably, but it is now well below the normative average, so… (in business & industry, we often call this benchmarking)

48. SO26, cont. Not only could the administrators and researchers show that these group homes had improved considerably, they could also show that they were doing considerably better than other state residential facilities

49. SO 27: Why is it important to collect normative data from a staff perspective? • Basically, so you know realistically, what good performance is given typical staff-to-resident ratios • The residents were profoundly developmentally disabled, typically nonverbal, and required assistance in self-care routines • The agency can only hire a certain number of direct care staff due to budgetary constraints - and usually these type of organizations are understaffed • Extremely high staffing ratios: 165 residents, total staff of 127 • It is simply unrealistic to assume that it is possible to have 0% off-task resident behavior - so back to the original question - what is good performance?

50. SO29: What is the potential disadvantage of targeting staff behavior in contrast to resident behavior? • As the authors note, and I mentioned briefly earlier, while group home supervisors observed the behaviors of staff and gave feedback to them weekly immediately after the observations, neither staff behavior nor supervisor observation behavior were graphed and fed back to supervisors or staff • Rather, the feedback that was given was feedback on the % of off-task resident behavior and % of time residents were involved in active treatment • To truly determine a functional relationship between staff and supervisor behavior and resident behavior, you would have to measure both (however, I admit I am convinced by the data)