Comprehensive Guide to Job Analysis and Validity Procedures

1. PSY 6430 Unit 6 Job Analysis Wednesday and Monday: Lecture Wednesday, 3/29: Exam

2. Introduction: Job Analysis • Job analysis is the bedrock of any selection procedure and is required from a legal perspective if the selection procedure results in adverse impact and is challenged in court • There are several different types of job analysis procedures • I am going to focus on two • Task analysis (and task questionnaire) • Position Analysis Questionnaire (PAQ) • Commercially available structured job analysis questionnaire; an off-the-shelf job analysis questionnaire (just need a little intro before I talk about the project)

3. Introduction: Job Analysis • The authors first talk about a job analysis interview • But that is rarely used by itself • Used by itself only when there are only a few job incumbents in the position • Usually combined with the task analysis and used only as the first step in the process • The task analysis and PAQ are the most popular types of job analysis • The authors also talk about the Critical Incidents Technique, but I am not going to focus on that • Critical incidents technique focuses on excellent and poor ways of handling important job-related situations/scenarios • It’s great to use for developing interview questions and situational tests but does not result in a comprehensive job analysis (always started with the exemplary performers; great to use after a comprehensive analysis to develop some of the instruments/methods for selection; main usefulness after a comprehensive analysis – supplement; face validity)

4. Relationship of job analysis to type of validity procedure* • As I indicated last week, the type of validity procedure you use is dependent upon several factors • Sample size • Whether the KSA is observable or not • Whether you use an off-the-shelf test or write the test • What type of cut-off score procedure you want to use *Learn for the exam! Next slide shows the relationship.

5. Content validity Criterion-related validity Validity generalization Task analysis only Task analysis, PAQ, any type Any type is OK, should be the same as the type used by the other company Relationship of job analysis to type of validity procedure • Once you decide what validity procedure you want to use, then you must make sure that your job analysis procedure is appropriate for that type of validity procedure Validity Procedure Type of Job Analysis (required by Uniform Guidelines; you cannot use the PAQ or any other type - I’ll explain why you cannot use the PAQ a bit later; predictive and concurrent; any type because the proof is in the correlation unlike content solely expert judgment; job component - PAQ)

6. Task Analysis Project • Most companies are now using content validity, so I am focusing on the task analysis • Due Monday, April 24 (final exam week) • Due Monday, April 17 if you want your grade before ME2 • Worth 70 points (Units 9 and 10) • Get started now: This is a time consuming project (but I obviously couldn’t assign it until we got to this point in the course)

7. Task Analysis Project • Task analysis is the first step in the development of a task inventory (job analysis questionnaire) • I am not going to have you develop the questionnaire or get ratings for the task statements or KSAs • Project consists of just the first step in the process - the identification of tasks and KSAs linked to each of the tasks • Two models in the course pack, Units 9 & 10 • Thor Flosason, & Julie Slowiak

8. Task Analysis Project • Task analysis is the most versatile job analysis procedure because • It can be used for both content and criterion- validity studies and • Is legally required for content validity (if tests are challenged in court)

9. Task Analysis Project • Resource on the web that will make this project a lot easier and a lot less time consuming for you • O*NET • Occupational Information Network – Dept. of Labor • Web site address is in the instructions • To start with, go to O*NET and use “Bartender” as the example position • Detailed instructions are in the study objectives (more detailed instructions in the description of project)

10. Task Analysis Project • You may conduct the analysis for one job incumbent • The job may be part-time or full-time • You cannot do the task analysis for a job you currently have or have had in the past • I want you to have the professional experience of collecting information about a job that you are not familiar with - this is much harder than you think

11. Task Analysis Project • The analysis will consist of • Task statements • KSAs linked to each task statement • Work activities: Physical activities and requirements (required for ADA) • Work Context: Environmental conditions (also good for ADA) • Work Styles: Typical working incidents (again good for ADA) • This is where many job analysts put things like “persuading individuals, working under stress, getting along with others” that are critical for ADA • Education and licensing or certification requirements (don’t forget job attendance -getting to work on time, and regular attendance, if that is required - ADA; Tasks MUST BE EXACT FORMAT)

12. Task Analysis Project • Task statement format • Gatewood, Feild, & Barrick, Figure 7.1, page 256 • My lecture tonight • Worksheets provided in course pack, Units 9 & 10 (Tasks MUST BE EXACT FORMAT)

13. Task Analysis Project • Caution! The task statement format differs in O*NET • O*NET is the end result of a task inventory, not the starting point • O*NET has 13 sections after the tasks and KSAs • I am requiring only the following four • work activities • work context • work styles • education and licensing/certification requirements

14. Task Analysis Project • You should have between 15-30 task statements • Part of the problem with writing task statements is determining how specific or how broad they should be • You will understand this difficulty when you begin to write the task statements

15. Task Analysis Project: Steps • See the project description which indicates where you find instructions for • Writing task statements • Writing KSAs • Documenting work activities, work context, work styles and education/license/certification requirements • Planning the analysis • Plan to meet with the job incumbent at least three times, for about 1.5 – 2 hours each time

16. Task Analysis Project: The Meetings • Preparing for the meetings • Before the meeting, obtain any written documentation that exists about the job • Job descriptions, organizational chart, etc. • Locate job title/position using O*Net (or a similar position) • First meeting: Task statements only (1.5-2 hrs) • Ask the incumbent to identify/state just the major aspects or areas of his/her job • After you have that list, go back to the first major area and ask for the tasks in that area • Move onto the next major area, etc. (different interview procedure from GFB; they provide interview guides on pages 254&259 are very good, but 3 meetings rather than 1; Also strongly recommend that you record the interview – never going to remember everything, miss things ; love to talk about their jobs)

17. Task Analysis Project: Steps • Second meeting (1.5-2 hours) • Confirm accuracy of task statements, obtain KSAs, and work activities, work context, work styles and education/licensure/certification (time permitting) • Before meeting with the incumbent, type up a list of the task statements • Identify any missing components to the tasks that you need to ask about • Have the incumbent review the task statements for accuracy and completeness • Take each task statement one at a time and ask the incumbent what KSAs are required to perform that particular task • Before doing this, reread “how to write KSAs” on page 60 of the study objectives

18. Task Analysis Project: Steps • Third meeting: Review of entire document • Type up the entire document and have the job incumbent review it • If revision is required, then you should ask the incumbent to review it one more time • Some students have done this last step remotely which is fine – but it is not good to do the first two steps remotely

19. Task Analysis Project: Format • See models in the course pack • Description of the company and position • Include the name of the position in the company and the corresponding O*Net position name, if there is one • Description of the methods/steps you used to develop the task analysis. • How many times did you meet with the incumbent? • What did you do during each meeting? • How long was each meeting?

20. Task Analysis Project: Format • Task Statements with KSAs linked to each task statement • Work Activities • Work Context • Work Styles • Education/License/Certification Requirements

21. Task Analysis Project: Some Grading Criteria • All sections included • Task statements and KSAs must be written/formatted per the study objectives • Grammar and spelling!! • You will lose credibility in a professional setting for grammar and spelling errors • Page numbers!! • Lateness • -7 points (10%) if later than 4/24, 5pm • No credit after Wednesday, 4/26 w/o my permission

22. Questionsabout the project?

23. SO2: Job Analysis (review) • Discrepancy between the professional and the legal requirements regarding job analysis • Job analysis is not required professionally for predictive and concurrent validity (it is required for content and validity generalization) • The proof of job-relatedness rests with a statistically significant correlation between test scores and job performance measures • It is, however, required legally for all types of validity procedures

24. SOs 6&7: Task Statement Format • What activity is performed? • Use a single present tense action verb - verb ends in an “s” • To whom or what? • The object of the verb • Why? • What is produced - what is the expected outcome? • How? • Using what materials, tools, procedures, guidelines, or equipment (moving to Sos 6&7; four components of a task statement; two tasks for a job you have held)

25. Task Statement ExamplesWelfare Eligibility Examiner • What activity is performed? • Asks questions, listens, and records answers • To whom or what? • Asks and listens to client, records on eligibility form • Why? • In order to determine eligibility for food stamps • How? • Using eligibility form, eligibility criteria in manual, interviewing techniques (from GFB; note more than one action verb that are all related to the same WHY; Why is the most frequently omitted component when students write task statements; clean up the task statement – next slide)

26. Task Statement ExamplesWelfare Eligibility Examiner • Clean it up Asks the client questions, listens, and records answers on standard eligibility form to gather information from which the client’s eligibility for food stamps can be determined using knowledge of interview techniques and eligibility criteria. (notice, several different verbs, actions that go together for the same “in order to”; often a question students have, next Slide another example)

27. Task Statement ExamplesWelfare Eligibility Examiner • What activity is performed? • Determines eligibility • To whom or what? • Of applicant • Why? • In order to complete client’s application for food stamps • How? • Using regulatory policies as a guide (another example - same job; once they obtain information from the client, then they must decide whether the person is eligible using policies; next slide different topic)

28. SO9: Problems with Job Interview as only source (this should not surprise you) • Lack of standardization across job analysts • Likely to get different information • Not practical for large numbers of incumbents because it’s one-on-one (2-3 hrs at least) • Once you have more than 5-10 job incumbents, it gets very time consuming and hence expensive • Legally, you must have a representative sample stratified by geographical region (if relevant), and protected classes, thus you may end up interviewing more incumbents than would otherwise be necessary (again, rarely, if ever used alone; I have just asked you to list any three of the ones given; more on next page)

29. SO9: Problems with job interview as only source, cont. • Easy to miss things when you are interviewing someone due to the quick verbal exchanges (as you will find out when you do your project) • Thus, unless you thoroughly document each interview, the legal requirements for the job analysis may not be met • Results are highly dependent upon the skills of both the interviewer and interviewee • Different incumbents may do different things in different ways and thus you may get an inaccurate view of the job – you need to find the “reasons why the job exists” and document those tasks and KSAs not ancillary tasks • Interviewee may exaggerate the activities and responsibilities to “look good” or because they believe they may get a pay raise

30. SOs 10 & 11: Task Statement and KSA Rating Scales • The Uniform Guidelines discusses the technical standards for a job analysis if it is used for content validation • All of the following four rating scales must be included for task statements • Frequency of task performance • Task importance or criticality (add essential function) • Task difficulty • Whether the task can be learned on the job within a relatively short period of time (within 6 months) (KSA rating scales next)

31. SOs 10 & 11: Task Statement and KSA Rating Scales • All of the following rating scales must be included for KSAs (if, again, used for content validation) • Is the KSA necessary for successful job performance? • Is the KSA required upon entry to the job or can it be learned within a relatively short period of time (within 6 months) • How difficult is it to obtain the KSA? • What type of education/experience/licensure/certification is necessary? (as you can see, the task inventory/questionnaire becomes quite extensive; 4 scales for tasks at least 3 for KSAs)

32. SO12: Obtaining names of the job incumbents • I strongly disagree with the authors about obtaining names on questionnaires • Employees tend to be very nervous and concerned about how the information will be used • When I have interviewed individuals, even one-on-one, I have almost always had to assure them that they would not be identified - rather the information would be combined with the information of others, with no identifiers. • Most employees just don’t trust the “corporate office,” HR or their bosses not to use the information against them (evaluation, downsizing, elimination of job) • You will probably be interviewing people you don’tknow (you have no established relationship with these individuals and if you are from “corporate” you are immediately distrusted)

33. SO12: Obtaining names, cont. (this slide NFE) • You should handle this just like you handle a participant list for research • You should create a master list with names and code numbers that correspond to your stratified sample • Geographical location, sex, race, etc. • Keep it under lock and key - you are the only one who should have access to it (or if you are not in charge of the analysis, the person that is) • Put the code numbers on the questionnaire, so you know whether you have a representative sample when the questionnaires are returned (Why codes, next slide)

34. SO12: Obtaining names, cont. (this slide NFE) • Why codes on questionnaires? • There is a legal requirement to have a representative sample for your job analysis • The legal requirement pertains to who participated in the job analysis, not simply who you sent questionnaires to • If you don’t get the sample you need, contact everyone in the underrepresented categories, with the standard, “We are contacting you because we recently sent you a questionnaire to complete. If you have not completed the questionnaire, we would greatly appreciate it if you would do so. If you have, please accept our thanks, and ignore this message.”

35. SO13: Advantages and Disadvantages of the Task Inventory • Advantages • It is an efficient way to collect data from a large number of individuals, particularly if they are geographically dispersed • Data can be easily quantified due to the rating scales

36. SO13: Advantages and Disadvantages of the Task Inventory • Disadvantages • Development is time consuming and expensive • Motivation to complete the questionnaire can be a problem because it is often very long due to the rating scales that must be included • Ambiguities and questions may arise when individuals complete the form that can’t be dealt with, which may result in inaccurate and less reliable information

37. SO14: Court case related to job analysis, the two implications: intro • Kirkland v NY State Department of Correctional Services, 1974 • The litigated test was a promotional exam for correction officers • The passing rates for whites was 31% while the passing rate for blacks was 8% • Adverse impact had been demonstrated thus the burden of proof shifted to the organization • Organization defended the exam on the basis of content validity • The court ruled in favor of the plaintiffs ruling that • The job analysis was not adequate, but more importantly because job analysis had been stressed in other cases • The process used to construct the test and the content of the test was not adequate (very important case related to job analysis when used for content validity; not in text)

38. SO14: Court case, the two implications • You must document the process used to construct the test (not just the process used to conduct the job analysis) • Did you develop a matrix that linked the KSAs to specific items on the test? • Who participated in that process and when did those meetings occur? • Remember, court cases often occur years after the development of the test and you are simply not going to remember the dates of the meetings, who attended, or the outcome of those meetings (first one this slide, second next slide)

39. SO14: Court case, the two implications • You must document and be prepared to defend each question on the test and the weight given to various questions/groups of questions This is why it is important/essential to construct a test if you use content validity rather than using an off-the-shelf test, and why I have been stressing that point

40. Introduction: Position Analysis Questionnaire (PAQ) (NFE) • The PAQ is a structured, standardized job analysis questionnaire that can be used to analyze virtually any job except managerial and professional positions • McCormick, Jeanneret, & Mecham • Professional and Managerial Position Questionnaire (PMPQ) • There is no other job analysis procedure that has been researched more extensively • Research started in 1974, over 40 years ago • Over 300,000 different jobs have been analyzed in the past 30 years • Can also be used for job evaluation (salary determination) and development of competencies (onto the PAQ)

41. Introduction: Position Analysis Questionnaire (PAQ) (NFE) • The authors moved a description of the PAQ to the validity chapter – job component validity • Determine job components using the PAQ analysis • Then select test instruments using validity data/information from the PAQ database (no criterion-related validity study) • I don’t recommend that for legal reasons although at least one court has upheld its use this way • I recommend a more conservative approach, using PAQ to identify selection tests and then conducting a local empirical validity study • Cannot be used for content validity – job/worker attributes, not tasks (relates job components to skills measured by selection tests; transportability for jobs for which job tasks are not similar – requirement for validity generalization; building block approach )

42. Introduction: Position Analysis Questionnaire (PAQ) (NFE) • The PAQ analyzes jobs in terms of 187 job elements or attributes that are broken down into six categories • Information input • When and how a worker gets information needed to perform the job • Mental processes • The reasoning, decision making, planning, and information processing activities that are involved in performing the job • Work output • The physical activities, tools, and devices used by the worker to perform the job

43. Introduction: Position Analysis Questionnaire (PAQ) (NFE) • Relationships with other persons • What relationships with other persons are required to perform the job? • Job context • The physical and social context in which the work is performed • Other job characteristics • The activities, conditions, and characteristics other than those already covered

44. Introduction: Position Analysis Questionnaire (PAQ) (NFE) • Information input (broken into visual, nonvisual, sensory & perceptual processes, and estimation activities) • Visual sources of job information - examples • Written material (books, reports, office notes, signs) • Quantitative materials (graphs, accounts, tables of numbers) • Pictorial materials (drawings, blueprints, diagrams, maps) • Nonvisual sources of job information - examples • Verbal sources (instructions, requests, interviews, orders) • Nonverbal sources (noises, engine sounds, sonar, music) • Touch (pressure, pain, feeling the texture of a surface) • Taste (food tasting, wine tasting) (break down Information input: each element/attribute is rated on a 6 point scale from 0, does not apply to 5, very substantial)

45. Introduction: Position Analysis Questionnaire (PAQ) • The PAQ focuses on worker attributes rather than job tasks • For example, Arvey & Begalla conducted PAQ analyses of a home maker’s position and a police officer’s position, and found the profiles to be very, very similar • Because of that, it cannot be used for content validity*, (which is its main disadvantage) • If you use the PAQ you must conduct a criterion-related validity study (or use validity generalization)* *SO23 asks you to learn these two points

46. SO20A: Underlying Rationale of the PAQ, the two assumptions) 1. There are a limited number of generalized human behaviors/attributes/job elements and all jobs can be characterized by the extent to which any particular job requires each of those job attributes/elements • That is, there are a certain number of job attributes/elements that are the “building blocks” for all jobs that exist (the PAQ authors identified 187), and each job is constructed from a set of these building blocks • Jobs can be analyzed by identifying which of the “building blocks” are required and the extent to which each is required in comparison to all other jobs that exist (*the authors refer to the “job elements” as generalized human behaviors, but they are not using the term the way we as Behavior analysts use it, so I prefer to talk about the job attributes or elements)

47. SO20A: Underlying Rationale of the PAQ (2 assumptions) 2. Each job attribute/element is normally distributed across all possible jobs that exist • Each of the 187 job attributes/elements has its own normal distribution across jobs (I’ll show a slide of this in a moment) • e.g., (a) the use of manually powered tools, (b) planning and scheduling, (c) the amount of oral communication, etc.

48. SO20B: How are the important job attributes for each job determined by the PAQ? • Each job analyzed is compared to the normal distribution for each of the 187 job attributes/elements to determine how much of this job attribute/element is required in comparison to all of the other jobs that exist • Data are reported in terms of percentiles • If a particular job attribute/element falls at the 90th percentile, it means that 90% of all other jobs require less of this attribute than the current job and thus it is a very important job attribute/element for this job • What would it mean if planning and scheduling fell at the 10th percentile?

49. Each job attribute is normally distributed across all jobs There is a normal distribution curve for each of the 187 attributes in the PAQ Percentiles The percentage of jobs that require less of this attribute than the target job position

50. Percentiles Use of manually powered tools: 80th percentile This job requires more use of manually powered tools than 80% of all jobs Percentiles Planning and scheduling: 10th percentile This job requires more planning and scheduling than only 10% of all jobs