CMOR-MRA Interviewer Assessment

1. CMOR-MRA Interviewer Assessment Performance Evaluation

2. Evaluation Used properly, a meaningful evaluation of employees’ work is one of the best motivational tools at the supervisors’ disposal Benefits: • Positively reinforce proper data collection practices and other work related behavior • Correct improper practices at work “A good job that goes unrecognized goes away. “A poor job that goes unrecognized multiplies.”

3. Evaluation • Looking forward: evaluations establish realistic goals and expectations • Looking back: they measure employees’ quality & quantity of work in relation to the overall goals of the organization

4. Standardized Measures For employees to meet job expectations, they must be communicated in advance, in terms that are SMART • Specific • Measurable • Attainable • Relevant • Time-bound

5. Specific • Specific: having a discreet area of performance to measure. For example: the “percentage of time on shift an interviewer spent dialing” (dialingproductivity, considers time spent on interviewing) is a concrete measure, conversely concepts like “dials a lot of numbers” are vague.

6. Measurable • Measurable: using quantifiable data to determine a score. For example: “dial telephone numbers for at least “x%” of the time you are logged in, and not interviewing” is measurable, while “do your share of the workload” is not.

7. Attainable • Attainable: realistic goals the employee has the skills and resources to reach. For example: “dial for 100% of the time you are logged in, and are not interviewing” is not a realistic goal if the employees need a few moments to learn their daily assignment, log into the project, and change from one project to another. • Setting unattainable goals is discouraging and setting inadequate goals is ineffective.

8. Relevant • Relevant: goals that directly relate to the employees’ job tasks. For example: “dialing productivity” relates to employees’ daily tasks whereas “keeps coworkers abreast of new information” is inappropriate since that is not the employees’ responsibility. Setting relevant goals helps employees motivate themselves on a daily basis. It allows employees to see the relationship between their daily performance, their long-range performance, and the overall success of a project.

9. Time-bound • Time-bound: goals that are expected to be reached by a deadline or within a specified time frame. For example: a performance expectation that “interviewers end of month dialing productivity scores must reach at least 90% to earn a ‘satisfactory’ rating” sets a specific time frame. Concrete time frames, communicated to employees in advance, help employees pace themselves and have some degree of self determination over the measurable aspects of their job.

10. “Time bound” also refers to giving employees adequate time to make needed adjustments. For example: if an employee is not informed until day 27 of the month that his or her dialing productivity is 40% instead of the expected 90%, it is unreasonable to expect the employee to raise their end-of-month productivity to 90% in just three days. Time-bound

11. Performance Measures • While exact performance measures vary across organizations, there are generally five areas examined in most call centers. • Productivity • Quality • Attendance • Punctuality • Soft Skills (e.g. "Plays well with others”)

12. Productivity • In research, there is a school of thought that measuring interviewers’ performance based on a number of completed interviews or telephone calls may encourage falsification of data. • For that reason, many call centers measure other aspects of performance to account for “quantity” of work. One example is measuring the percentage of the shift the interviewer spent dialing.

13. DialingProductivity • For example, in a five hour shift (after subtracting break time and time spent interviewing) there may be 240 minutes in which the interviewer has the potential to dial records. • An interviewer dialing records for 230 out of those 240 potential dialing minutes has 96% “productivity” (230/240=95.8) The fewer minutes spent dialing, the lower the productivity.

14. Production Completes per hour Incidence For Tougher Quotas Refusal Prevention Rate Agree Rate Response Rate Length Performance Measurement What To Evaluate? Productivity • Quality • Effective Intros • Gaining Cooperation • Getting Answers • Controlling the Respondent • Rapport/Phone Presence • Data Entry

15. Productivity Performance Measurement Production Scores • Dials Per Hour • Completes Per Hour • Agrees Per Hour • Initial Refusal Rate • Refusal Conversion Rate • Best Use of Sample (highest number of agrees per numbers dialed) Pick Your Goals!

16. Dialing Productivity Non-Dialing Activities: • There are some legitimate non-dialing activities that interviewers may be required to perform periodically, for example, editing their schedules or reading memoranda. • Supervisors may direct interviewer to log into “non-dialing” account activities in order to reduce the number of “potential dialing minutes” when interviewers have legitimate reasons to not dial. • In other words, an interviewer editing their schedule for five minutes would have 235 potential dialing minutes in our example.

17. Dialing Productivity Non-Dialing Activities: • Because some interviewers may attempt to disguise unproductive activities (like socializing or taking unauthorized breaks) by leaving records open, call centers often limit the amount of time counted toward productivitybased on the type of call. • If “no answer” calls are known to take a minute or less, then an interviewer taking ten minutes to dial a “no answer” only gets credit for the one minute it should have taken.

18. Dialing Productivity • One call center uses the following scores and expectations with about 96-99% of their employees regularly keeping themselves at acceptable productivity levels. • “Quantity” is not officially counted in Month One. Coaching on increasing productivity is given in Month Two of employment if less than “Satisfactory.” This gives new employees a chance to acclimate. • Outstanding: >95% • Above Satisfactory: 94.9%-92.5% • Satisfactory*: • 92.4%-90% • Needs Improvement: 89.9%-87.5% • Unsatisfactory: <87.4%

19. Quality • “Quality” of the interviewers’ work is measured by their monitor sheets. Call centers use their own variations on the areas rated on the monitor sheet in terms of “quality”. • The following is an example for illustrative purposes.

20. Attitude X Courteous polite and professional (3pts) X Sounds interested (2pts) X Correct pronunciation (3pts) X Clear speech (2pts) Probing -- Probes for more accurate information (9pts) -- Identifies all answers that need a probe (3pts) -- Neutral Probes (3pts) -- Effective Probes (3pts Interview Quality Control Interviewer: (357) JOHN DOE Key: Date: 05/27/2007 X - Met Job Standards Session: 357200510271 * - Standards Met but Coaching Given Monitored by: (895) JANE DOE 0 - Did not meet Job Standards Began At: 18:21:34 did not Observe Ended: 18:37:10 Total Time: 00:15:36 Ranking Score: Language: EN Outstanding: >= 99.5 Outcome: Observed Above Satisfactory: >= 98.8 to < 99.5 Satisfactory: >= 98.0 to < 98.8 Needs Improvement: >= 97.1 to < 98.0 Unsatisfactory: 0 to 97.1 Quality Procedures X Follows established procedures (3pts) X Correctly enters responses (9pts) X Gives correct disposition codes (3pts) 0 Lets the phone ring 6 times (2pts) Interviewing Techniques: General -- Moderate pace (3pts) -- Enumerates smoothly and correctly (9pts) -- Reads all scripted questions/screen (9pts) -- Questions read were verbatim; no added, deleted or substituted words (3pts) -- Follows on screen instructions (3pts) -- Competently encourages reluctant R or I (3pts) -- Does not misrepresent org or use of data (9pts) -- Answers I's/R's questions correctly (3pts) Interviewing Techniques: In Survey -- Controls progress of the interview (2pts) -- Proceeds from intro directly into first Q (3pts) -- Listens to I/R (3pts) -- Gives only neutral and varied feedback (3pts

21. Quality • In this layout, the employee begins with a score of 100% and loses points for each category in which they “do not meet” job requirements. • Categories are weighted; the most important tasks, such as reading verbatim and marking correct answers, carry a heavier penalty than, say, ringing a ‘no answer’ ten times instead of six.

22. Quality • Performance Measurement Quality Scores There are two approaches to developing a score for a Interviewer’s performance: • Start at 100 points and deduct as deviations occur. • Start at 0 points and add up as compliance occurs or procedures are followed.

23. Quality Performance Measurement Quality Scores ~ Areas of Evaluation • Accurate Data Entry • Delivery Style • Avoidance of Leading • Adherence to Quotas • Avoidance of Bias • Intro Effectiveness • Reading Verbatim • Probing Where Needed • Degree of Respondent • Difficulty

24. Quality • One call center uses the following scores and expectations with most employees regularly keeping themselves at acceptable quality levels. • “Quality” is not officially counted in Month One. Coaching on how to increase quality is given in Month Two of employment if less than “Satisfactory.” This gives new employees a chance to acclimate. • Outstanding: >99.5% • Above Satisfactory: 99.4%-98.8% • Satisfactory*: 98.7%-98.0% • Needs Improvement: 97.9%-97.1% • Unsatisfactory: <97%

25. Reliability • There are various methods of measuring employee attendance. Some organizations count all missed shifts as “absences” while others only count shifts missed for which the employee did not have enough “leave” (vacation or sick hours) to “cover” the absence. • The method that follows counts any missed shift without 24 hour notice to be an “absence”, regardless of whether they employee had leave to cover the absence.

26. Reliability • Number of unplanned absences versus goal • Number of late arrivals versus goal • Number of times leaving early versus goal • Number of no call/no shows • Number of extra shifts scheduled Performance Measurement Attendance Score

27. Reliability • To determine the “reliability” score, dividing the number of hours the employee was absent from work without24 hours notice by the number of hours the employee was scheduled to work (and subtract 1, & take the absolute value of the result to get the percentage reliable). • One call center uses the following scale with the typical employee keeping themselves within acceptable limits. • Outstanding: >97.5% • Above Satisfactory: 97.4%-95% • Satisfactory: 94.9%-92.5% • Needs Improvement: 92.4%-90% • Unsatisfactory: <89.9%

28. Reliability • Outstanding: • OPoints • Above Satisfactory: • 1-15 Points • Satisfactory: • 16-31 Points • Needs Improvement: • 32-47 Points • Unsatisfactory: • 48-63 Points • Another approach includes allowing interviewers to use their accrued vacation and sick leave according to established personnel policies. Should the employee miss work without properly requesting sick or vacation leave, or they do not have enough leave to cover the whole absence, the employee earns a demerit for each shift (or portion of the shift) missed. • A demerit could take the form of 1 Unauthorized Absence Point for some specified time period (e.g. for every 15 minutes of work). • One call center uses the following scale with 1 Unauthorized Absence Point equaling 15 minutes of work missed with no leave to cover it.

29. Punctuality • There are different schools of thought on measuring employee punctuality. Some organizations count an employee tardy if they clock in for their shift as little as 1 minute late. • Other call centers may be more lenient, may not time employee arrival down to the minute, or may not use time clocks at all. • Some organizations include returning on time from break in measuring “punctuality”.

30. Punctuality • To determine the percentage of late arrivals, divide the number of times the employee was late to work by the number of shifts (subtract 1 to get the percentage on-time,then take absolute value of result). • Outstanding: >98% • Above Satisfactory: 97.9-96% • Satisfactory: 95.9%-94% • Needs Improvement: 93.9%-92% • Unsatisfactory: <92 %

31. Subjective Employee Attributes • Most employers address soft skills such as “initiative and ingenuity”, “relationships with others”, “professional image”, “personal fitness” and so forth. • These are not quantifiable measures, so it is important to retain some sense of neutrality and not allow personal feelings about the employee to determine the score. • In the often fast-paced, production driven environment of many call centers, supervisors may not have enough personal knowledge of employees’ attributes in these areas other than they “appear to be meeting expectations”.

32. Without data to score employees “soft skills”, supervisors have a natural tendency to develop an impression of an employee first, then substantiate that rating with anecdotes. Obvious drawbacks include: Scoring employees higher than they deserve encourages substandard performance- has the potential to unfairly raise overall evaluation score Scoring employees lower than they deserve discourages employees who are doing well in at least some areas- has the potential to unfairly lower overall evaluation score Rating employees on “soft skills” with no clear cut criteria or standardized measures opens the organization up to grievances from employees who believe they are being held to different standards than other employees Subjective Employee Attributes

33. Subjective Employee Attributes Facilitator’s notes: Employees are encouraged to think of a “satisfactory” in these areas as a “within expected range” score. Not like a “C” in school • “Satisfactory” scores in the soft skills categories are not officially counted toward the overall evaluation score if they will lower the overall evaluation; for example pulling an “outstanding” or “above satisfactory” score down. • Less than satisfactory scores in soft skills are counted toward the overall evaluation score. • For these reasons some call centers score all employees as “satisfactory” in soft skills categories except when there are documented instances of the interviewer requiring “corrective counsel” or commendations for “far exceeding” minimum job expectations.

34. Subjective Employee Attributes Performance Measurement Cooperation Score • Stays within allowed break time • Respectful of company property • Follows company procedures/rules • Represents company professionally • Respects co-workers

35. Typical feedback using this scheme: COMMENTS: Evaluation Rating Period = ____/____/____ – ____/____/____. ____________________, the scores for each job category were obtained using the following measures: Quality: Monitor Sheet Scores O>99.5% A 99.4%-98.8% S 98.7%-98.0% N 97.9%-97.1% U<97% Current Calendar Month __________ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ __________ __________ __________ Quantity: Productivity Scores O>95% A 94.9%-92.5% S 92.4%-90% N 89.9%-87.5% U<87.4% Current Calendar Month __________ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ __________ __________ __________ Reliability: % of scheduled shifts worked O>97.5% A 97.4%-95% S 94.9%-92.5% N 92.4%-90% U<89.9% Current Calendar Month __________ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo __________ __________ __________ __________ __________ Punctuality: % on-time to shift/from break O>98% A 97.9-96% S 95.9%-94% N 93.9%-92% U<92 % Current Calendar Month __________ __________ __________ __________ __________ [sh:_____ br:_____] FYI: Cumulative from Hire date to End of Curr Cal Mo __________ __________ __________ __________ __________ [sh:_____ br:_____] The remaining categories, Work Habits, Relationship with People, Initiative, Attitude, and Personal Fitness are considered “satisfactory” unless otherwise mentioned or discussed with a Supervisor. * “Quality” and “Quantity” data are not officially counted in Months 1 and 2 if they are less than satisfactory. ** However, all interviewers must earn at least satisfactory scores in all categories by Months 3 and beyond in order to satisfy basic job requirements. Refer to the feedback from your monitor sheets for the specific commendations and/or instructions for work improvement and adhere to them consistently. General reminders: read verbatim, probe responses to get one clear and final answer and key data correctly. Review your job aids regularly including the Interviewer Handbook (especially Lesson 5) and the Questionnaire Specific Guides. In closing, keep up the good work you are doing in the areas mentioned earlier and/or on the monitor sheets and make any/all necessary improvements noted there as well. If you have questions, please feel free to ask now, and you may also ask your team leader or a shift supervisor for assistance at any time as well. Overall evaluation rating = □ Outstanding, □ Above Satisfactory, □ Satisfactory, □ Needs Improvement, □ Unsatisfactory. INTERVIEWER: ___________________________________________________________ EVALUATOR: ________________________________________________ DATE: ________________________

36. Typical feedback using this scheme: COMMENTS: Evaluation Rating Period = ____/____/____ – ____/____/____. ____________________, the scores for each job category were obtained using the following measures: Quality: Monitor Sheet Scores O>99.5% A 99.4%-98.8% S 98.7%-98.0% N 97.9%-97.1% U<97% Current Calendar Month __________ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ __________ __________ __________ Quantity: Productivity Scores O>95% A 94.9%-92.5% S 92.4%-90% N 89.9%-87.5% U<87.4% Current Calendar Month __________ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ __________ __________ __________ Reliability: % of scheduled shifts worked O>97.5% A 97.4%-95% S 94.9%-92.5% N 92.4%-90% U<89.9% Current Calendar Month __________ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo __________ __________ __________ __________ __________ Punctuality: % on-time to shift/from break O>98% A 97.9-96% S 95.9%-94% N 93.9%-92% U<92 % Current Calendar Month __________ __________ __________ __________ __________ [sh:_____ br:_____] FYI: Cumulative from Hire date to End of Curr Cal Mo __________ __________ __________ __________ __________ [sh:_____ br:_____] The remaining categories, Work Habits, Relationship with People, Initiative, Attitude, and Personal Fitness are considered “satisfactory” unless otherwise mentioned or discussed with a Supervisor. * “Quality” and “Quantity” data are not officially counted in Months 1 and 2 if they are less than satisfactory. ** However, all interviewers must earn at least satisfactory scores in all categories by Months 3 and beyond in order to satisfy basic job requirements. Refer to the feedback from your monitor sheets for the specific commendations and/or instructions for work improvement and adhere to them consistently. General reminders: read verbatim, probe responses to get one clear and final answer and key data correctly. Review your job aids regularly including the Interviewer Handbook (especially Lesson 5) and the Questionnaire Specific Guides. In closing, keep up the good work you are doing in the areas mentioned earlier and/or on the monitor sheets and make any/all necessary improvements noted there as well. If you have questions, please feel free to ask now, and you may also ask your team leader or a shift supervisor for assistance at any time as well. Overall evaluation rating = □ Outstanding, □ Above Satisfactory, □ Satisfactory, □ Needs Improvement, □ Unsatisfactory. INTERVIEWER: ___________________________________________________________ EVALUATOR: ________________________________________________ DATE: ________________________ ◄ Dates & Name ◄Categories & Scores Needed for Each Rating ◄Explanation of Soft Skills ratings ◄Narrative Feedback ◄Overall Rating, Signatures, Date

37. Timely Feedback • Evaluation frequency may vary by organization. • Most employers require an “official” annual evaluation of record at the employee’s anniversary of hire, upon which some form of merit pay increase may be granted. • Most call centers use some end-of-month evaluation in order to give timely reinforcement of proper interviewing techniques.

38. Timely Feedback • Regular, ongoing feedback allows interviewers to make the necessary performance adjustments and improve their data collection habits. This yields a better performance than when feedback is infrequent. • Supervisors use the employees’ monthly evaluation as a time to not only discuss end-of-month scores and specific interviewing techniques, but also to build rapport with employees, answer their questions, and listen to concerns.

39. Feedback • A detailed examination of how to give interviewers proper feedback is included in the other CMOR/MRA Modules. • General concepts include: • Remember the guiding principles of giving feedback: • Positively reinforce proper data collection practices, correct improper data collection practices • “A good job that goes unrecognized goes away. • A poor job that goes unrecognized multiplies.”

40. Feedback • Coach employees to at least a satisfactory performance, increases unit-wide data collection quality and quantity: “We are all in this together”– it is not “us against them“ or a “policing” outlook. • Communication has four parts: intended message, words stated, words heard, message understood.

41. Feedback • Use standardized conventions when writing to reduce the burden of composing fresh feedback for every interviewer when most employees need feedback in similar areas. • Advice must guide future actions, not punish past actions • Watch “tone” and word choice both in and out of monitor sessions • Avoid phrases like, “you handled such and such incorrectly”. Opt for phrases like, “the proper way to deal with a question like such and such is to…” • Avoid the accusatory or condescending tone suggested by words like “you” and “should”. Opt for phrases like, “to gather valid data, all employees must do such and such…” and “we need everyone to do such and such… “

42. Feedback • Using standardized conventions when writing feedback not only saves time, but increases the supervisors’ ability to ensure the language used is fair and legal. • One such convention entails beginning all evaluations with, “Keep up the good work you are doing in the area of___” and fill in each one or more area in which the employee is meeting or exceeding job standards. • If there is an area in which to improve use a standardized lead-in. For example: “in order to collect valid data, the one task we need you to perform more consistently is___” or “one area in which you need to improve is___.” • Come up with standardized ways to explain what constitutes deficient performance in each area, how this impacts the workplace, what the employee must do to improve. For example: “In the area of ___ you are not currently meeting job standards because ___. The impact to the call center is ___. Strive to ___. Your end-of-month scores for ___ must meet or exceed ___.” • The next few examples show a few “boilerplate” evaluations where “[ ]” represents the feedback filled in from a standardized list of comments. Facilitators note: the “[ ]” are illustrative and would not be included in the evaluation.

43. Feedback Examples COMMENTS: Evaluation Rating Period = 01/01/07 – 01/31/07. JDoe______________, the scores for each job category were obtained using the following measures: Quality: Monitor Sheet Scores O>99.5% A 99.4%-98.8% S 98.7%-98.0% N 97.9%-97.1% U<97% Current Calendar Month __________ 99.1______ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ 98.0______ __________ __________ Quantity: Productivity Scores O>95% A 94.9%-92.5% S 92.4%-90% N 89.9%-87.5% U<87.4% Current Calendar Month 95.5______ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* 95.0______ __________ __________ __________ __________ Reliability: % of scheduled shifts worked O>97.5% A 97.4%-95% S 94.9%-92.5% N 92.4%-90% U<89.9% Current Calendar Month __________ __________ __________ __________ 50%______ FYI: Cumulative from Hire date to End of Curr Cal Mo __________ __________ __________ 90%______ __________ Punctuality: % on-time to shift/from break O>98% A 97.9-96% S 95.9%-94% N 93.9%-92% U<92 % Current Calendar Month __________ __________ 94.0______ __________ __________ [sh: 98__ br: 91___] FYI: Cumulative from Hire date to End of Curr Cal Mo __________ __________ 95.0______ __________ __________ [sh: 95__ br: 95___] The remaining categories, Work Habits, Relationship with People, Initiative, Attitude, and Personal Fitness are considered “satisfactory” unless otherwise mentioned or discussed with a Supervisor. Keep up the good work you are doing in the areas of [“quality” and “quantity”]. In the area of [“quality”] your monitor sheets show you are doing well on [all standard interviewing techniques like reading verbatim, using a moderate pace, marking correct answers, and being polite and professional toward the public]. The one task we need you to perform more consistently is [“neutral probing”]. Specifically, [be sure to offer all on screen options (which is “neutral”) and not tapering off after reading only the first couple of items (which is “leading”)]. Good job meeting minimum job expectations for [“punctuality”]; [be careful to not dip into the “needs improvement” range which you one percentage point above right now]. [Strive to arrive on time for all shifts and from breaks]. In the area of [“reliability”][ you are not currently meeting job standards because this month you only worked (half) of the shifts you signed up to work]. Strive toward working all of your scheduled shifts]. Alternately, at least notify us of expected absences one day in advance. [Consider this “corrective counsel” to increase your (reliability) to at least (92.5%) for the whole month of (February)]. Overall evaluation rating = □ Outstanding, □ Above Satisfactory, □ Satisfactory,  Needs Improvement, □ Unsatisfactory. INTERVIEWER: ___________________________________________________________ EVALUATOR: ________________________________________________ DATE: ________________________

44. Feedback Examples COMMENTS: Evaluation Rating Period = 01/01/07 – 01/31/07. PDoe______________, the scores for each job category were obtained using the following measures: Quality: Monitor Sheet Scores O>99.5% A 99.4%-98.8% S 98.7%-98.0% N 97.9%-97.1% U<97% Current Calendar Month __________ __________ __________ 97.5%____ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ 98.0%____ __________ __________ Quantity: Productivity Scores O>95% A 94.9%-92.5% S 92.4%-90% N 89.9%-87.5% U<87.4% Current Calendar Month __________ __________ 90.1%____ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ __________ 89.9%____ __________ Reliability: % of scheduled shifts worked O>97.5% A 97.4%-95% S 94.9%-92.5% N 92.4%-90% U<89.9% Current Calendar Month 100%_____ __________ __________ __________ 50%______ FYI: Cumulative from Hire date to End of Curr Cal Mo 98%______ __________ __________ 90%______ __________ Punctuality: % on-time to shift/from break O>98% A 97.9-96% S 95.9%-94% N 93.9%-92% U<92 % Current Calendar Month 99%______ __________ __________ __________ __________ [sh: 98__ br: 100__] FYI: Cumulative from Hire date to End of Curr Cal Mo 99%______ __________ __________ __________ __________ [sh: 100_ br: 98___] The remaining categories, Work Habits, Relationship with People, Initiative, Attitude, and Personal Fitness are considered “satisfactory” unless otherwise mentioned or discussed with a Supervisor. Keep up the excellent work you are doing in the areas of [“punctuality”] and [“reliability”]. We appreciate your good attendance and timeliness. Thank you. In the area of [“quantity”] good job meeting the productivity requirement by dialing at least 90% of the time you were logged in. Keep up the good work, you beat your total from last month. [Take care no to dip into the “needs improvement” range which you a percentage point above right now]. In the area of [“quality”], [in order to gather valid data the areas where you need to perform more consistently are]: [read every single question verbatim, speak and read at a moderate pace, and offer all on screen choices to the respondent]. [These are “biggies” and we covered them extensively in training.] [Consider this “corrective counsel” to increase your (quality) to at least (92.5%) for the whole month of (February)]. Overall evaluation rating = □ Outstanding, □ Above Satisfactory, □ Satisfactory,  Needs Improvement, □ Unsatisfactory. INTERVIEWER: ___________________________________________________________ EVALUATOR: ________________________________________________ DATE: ________________________

45. Feedback Examples COMMENTS: Evaluation Rating Period = 01/01/07 – 01/31/07. WDoe ______________, the scores for each job category were obtained using the following measures: Quality: Monitor Sheet Scores O>99.5% A 99.4%-98.8% S 98.7%-98.0% N 97.9%-97.1% U<97% Current Calendar Month __________ 99.1______ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* __________ __________ 98.0______ __________ __________ Quantity: Productivity Scores O>95% A 94.9%-92.5% S 92.4%-90% N 89.9%-87.5% U<87.4% Current Calendar Month 95.5______ __________ __________ __________ __________ FYI: Cumulative from Hire date to End of Curr Cal Mo* 95.0______ __________ __________ __________ __________ Reliability: % of scheduled shifts worked O>97.5% A 97.4%-95% S 94.9%-92.5% N 92.4%-90% U<89.9% Current Calendar Month 100%_____ __________ __________ __________ 50%______ FYI: Cumulative from Hire date to End of Curr Cal Mo 98%______ __________ __________ 90%______ __________ Punctuality: % on-time to shift/from break O>98% A 97.9-96% S 95.9%-94% N 93.9%-92% U<92 % Current Calendar Month 99%______ __________ __________ __________ __________ [sh: 98__ br: 100__] FYI: Cumulative from Hire date to End of Curr Cal Mo 99%______ __________ __________ __________ __________ [sh: 100_ br: 98___] The remaining categories, Work Habits, Relationship with People, Initiative, Attitude, and Personal Fitness are considered “satisfactory” unless otherwise mentioned or discussed with a Supervisor. Keep up the excellent work you are doing in the areas of [“quality”] [“quantity”][“punctuality”] and [“reliability”]. Your excellent attendance, punctuality and productivity have had a positive impact on the call center’s ability to handle the workload and complete projects on time. In the area of [“quality”], you have exceeded job standards by: [reading verbatim, speaking and reading at a moderate pace, using neutral probes, and marking all answers correctly]. Keep up the good work. In this area, in order to gather valid data, the one task we need you to perform more consistently is giving correct disposition codes. Specifically, every now and then you code “refusals” when the respondent is cooperative and needs an “appointment within the questionnaire” code. Otherwise, keep up all of the good work mentioned above. Overall evaluation rating =  Outstanding, □ Above Satisfactory, □ Satisfactory, □ Needs Improvement, □ Unsatisfactory. INTERVIEWER: ___________________________________________________________ EVALUATOR: ________________________________________________ DATE: ________________________

46. Closing • Interviewers are the most important link in the data collection chain. • They mark the point of entry for all data coming into analysis. • While interviewers can and should be expected to adhere to prescribed data collection practices, it is the supervisors’ responsibility to motivate interviewers to collect valid data, communicate project goals to interviewers in advance, and give them the feedback needed to succeed. “Without the heart there can be no understanding between the hand and the mind.” ~ Fritz Lang - Metropolis