September 21, 2011 Lance Anderson, Ph.D. Brian Cronin, Ph.D.

1. Social Security AdministrationCall Order 0001: Job Analysis MethodologyBriefing of Final Project ResultsOIDAP Quarterly Meeting September 21, 2011 Lance Anderson, Ph.D. Brian Cronin, Ph.D.

2. Agenda • Introductions/Overview of Team • SSA • ICF • Subcontractors • Introduction to Project and Purpose • Project Methodology • Recommendations • Job Analysis Procedures • Job Analysis Models • Summary • Overarching Recommendations • Potential Next Steps • Questions?

3. Introductions

4. Overview of Team SSA • Debra Tidwell-Peters • David Blitz • Michael Dunn • Elizabeth Kennedy • Mark Trapani ICF International • Brian Cronin, Ph.D. • Lance Anderson, Ph.D. • Beth Heinen, Ph.D. • Jessica Jenkins, MPhil • Allison Cook, M.S. • Daniel Fien-Helfman Subcontractors • Paul Davis, Ph.D. • Kelly Day, OTD • Len Matheson, Ph.D.

5. Introduction to Project and Purpose

6. Introduction to Call Order 0001 • SSA is developing new occupational information system (OIS) tailored specifically to SSA’s disability programs and adjudication process. • OIS will replace the Dictionary of Occupational Titles (DOT) and companion volumes, including the SCO and RHAJ. • To develop OIS, SSA needs detailed methodology and strategy for analysts to perform job analysis throughout U.S. labor market. • Purpose of Call 0001 was to perform the research needed to identify useful features of existing practices to support development of SSA’s job analysis methodology.

7. Project Methodology

8. Project Methodology

9. Call Order 0001 Terminology • Overarching Goal: Summarize wealth of information on job analytic procedures across disciplines to allow SSA decision makers to quickly understand and evaluate various job analysis practices. • To ensure consistency, we use the following nomenclature: • Project Method(ology) – Refers to steps taken to conduct research for this call order • Job Analysis Method(ology) – Refers to ultimate SSA job data collection process developed to address OIS needs • Practice – Refers to all job analysis approaches, models and procedures, identified through this call order • Model – Refers to an established, ‘off-the-shelf’ job analysis approach identified through this call order • Procedure – Refers to a data collection technique identified through this call order

10. Recommendations on Job Analysis Procedures

11. Procedure: Review of Written Materials • Recommendation: Effective and cost-efficient starting point for almost all job analysis. Should be included in SSA’s ultimate data collection methodology. • Potential Usage for SSA: Analysts are able to become more familiar with the target job. • Potential Challenges: Dependent upon the usefulness and availability of source documents. Should only be used in conjunction with other data collection procedures. • Outcome: Knowledge gained can be used to inform development of subsequent data collection techniques (e.g., interviewing).

12. Procedure: Job Observation • Recommendation: Use for jobs that include more manual, less cognitive tasks. Should be included in SSA’s final data collection methodology. • Potential Usage for SSA: Useful for collecting detailed information about job tasks, equipment/materials used, and work environment; does not rely solely on testimony of incumbents. • Potential Challenges: Can be costly and time-consuming; may not be appropriate for all jobs (e.g., highly cognitive jobs). • Outcome: Should be used in combination with other data collection procedures to determine prevalence of work activities observed.

13. Procedure: Survey • Recommendation: Use when gathering data from geographically dispersed incumbents. Should be further considered for inclusion in SSA’s ultimate methodology. • Potential Usage for SSA: Can be effective and efficient means of collecting data from numerous incumbents; useful for assessing prevalence of work activities/; can make comparisons across jobs. • Potential Challenges: May be costly to develop and administer; Some threats to validity, such as inflated ratings, lack of understanding rating elements, lack of effort in completing survey. • Outcome: Other data collection procedures should be used to supplement the data collected especially when more detail is needed.

14. Procedure: Structured Interviews • Recommendation: Use when job is more complex and needs additional clarification or when comparison is needed (as long as highly structured). Should be incorporated into SSA’s ultimate data collection methodology. • Potential Usage for SSA: Allow analysts to collect detailed job information through the direct questioning of incumbents. • Potential Challenges: Can be time-consuming and costly to prepare protocol, coordinate participants’ schedules, and arrange travel (for face-to-face interviews). • Outcome: Analyst should undergo training on interviewing skills; analysts should conduct multiple interviews to gain full value of this procedure.

15. Procedure: Focus Groups • Recommendation: May be more time-efficient and cost-efficient than interviews, but we recommend that interviews be used instead of focus groups whenever possible to allow for clarification questioning. • Potential Usage for SSA: Allows for discussion with multiple incumbents at one time; useful for idea generation or other data collection that depends on interaction to elicit information. • Potential Challenges: Group dynamics can influence participant responses; schedule coordination can be difficult; specialized training needed for job analysts to be skilled facilitators. • Outcome: Participants should be provided with an agenda, ground rules, and background information prior to the session.

16. Procedure: Physical Demands Measures • Recommendation: May be valuable for jobs requiring manual labor given the disability determinations SSA must make. Depending on SSA’s ultimate construct model and data collection instrument, should include instrument measurement of physical demands (i.e., tools that measure PDs) but with limited use. • Potential Usage for SSA: Advantages include precise nature of the collected data, high reliability, high validity, and data that are typically easy to aggregate. • Potential Challenges: Can be resource intensive to administer and require specialized training; can be intrusive to incumbents. • Outcome: Must have technical expertise in the use of specific measurement tools selected.

17. Recommendations on Job Analysis Models

18. Model: AET The AET involves conducting an observation and interview to complete an ergonomic questionnaire. • Examples of Effective Features • Use of descriptors that isolate specific types of physical effort • Use of scales that focus on frequency, duration, and significance • Use of examples to assist in coding level of demand • Example Limitations for SSA Context • Significant time and labor commitment to develop career ladders • Long term commitment to upkeep of career ladders

19. Model: Common Metric Questionnaire (CMQ) The CMQ collects data via a survey administered directly to incumbents and/or their immediate supervisors. • Examples of Effective Features • Matrix structure of the questionnaire • Computerized interface to allow for effective use of the questionnaire • Use of behavioral and observable descriptors that are easy for incumbents and supervisors to rate • Example Limitations for SSA Context • Does not include a comprehensive set of descriptors at a broad level • Some respondents might not have the access to a computer to use the computerized interface • CMQ-like items on the OIS would need to be continually updated

20. Model: Cognitive Task Analysis (CTA) The CTA approach involves using a variety of data collection procedures to ultimately identify the cognitive processes underlying a job with a particular focus on the processes that distinguish an expert from a novice. • Examples of Effective Features • Use of structured observation and interviews together with completion of a structured tool by the analyst • Identification of the various types of knowledge needed to do the job • Example Limitations for SSA Context • Tends to lack the detailed information needed by SSA on various physical abilities • Would require extensive training of analysts

21. Model: Fleishman Job Ability Requirement Scales (F-JAS) The primary data collection procedure for the F-JAS model involves the administration of the Ability Requirements Scales to collect data on 52 types of abilities. • Examples of Effective Features • Generally well supported by research • Use of level scales anchored with observable behaviors • Example Limitations for SSA Context • Lacks some generalizable physical abilities constructs that are important to SSA • Tends to provide details on the variation of jobs at the high end of many abilities, whereas SSA might be more interested in information about the variation in the jobs at the low end of abilities

22. Model: Functional Job Analysis (FJA) The FJA approach gathers a variety of different types of job analysis data typically collected via interview and observation but may also include other data collection procedures. • Examples of Effective Features • Inclusion of work context and worker environment variables • Use of procedures that can be easily trained • Structured framework and structured protocols build validity • Example Limitations for SSA Context • Lacks standardization on important issues such as how jobs are sampled, how interviews are conducted, and how many interviews are conducted • The DOT scales lack detail on cognitive abilities and interpersonal skills

23. Model: Job Element Model (JEM) JEM focuses on the human attributes required for superior performance on the job and collects data via focus groups, interviews, and surveys. • Examples of Effective Features • Approach to conducting interviews and observations to gather job specific information provides data for understanding the job • Example Limitations for SSA Context • Experts previously rated this model low in terms of reliability and standardization • Although a low cost approach, it involves a significant amount of time to administer

24. Model: O*NET O*NET was developed using a job analysis approach that focuses primarily on surveys, with supplementary use of interviews and reviews of written material. • Examples of Effective Features • The hierarchical arrangement and use of the content domain • A nationwide database supported and maintained by an external entity • Example Limitations for SSA Context • Some items will likely have low reliability relative to other instruments because constructs are not observable • Tends to focus on differentiating high-tech jobs as opposed to low skill jobs that are usually the focus of disability claims

25. Model: Position Analysis Questionnaire (PAQ) The PAQ is an existing job analysis model that uses a standardized 195-item instrument to collect data, which is typically completed by a job analyst based upon data collected in job observations and interviews. • Examples of Effective Features • Use of an instrument that focuses on generalizable work activities provides data for cross-job comparison • Focus on observable behaviors ensures greater verifiability of the findings • Example Limitations for SSA Context • Constructs at a level of abstraction that may not provide a clear picture of the job.

26. Model: Task Inventory The Task Inventory approach involves collecting data through procedures such as review of written materials, job observation, interviews, and surveys to ultimately develop a list of task descriptions. • Examples of Effective Features • Method of conducting interviews and observations to gather job specific information provides data for understanding the job • Example Limitations for SSA Context • No generalizable constructs or scales to allow for cross-job comparison

27. Model: Threshold Traits Analysis (TTA) The TTA collects worker trait, job demand, and job function data using data collection procedures such as review of written materials, job observations, interviews, and surveys. • Examples of Effective Features • The 33 traits include a parsimonious and simply worded set of constructs that might provide an effective perspective for sorting and locating jobs • In general, the TTA is well supported by research • Example Limitations for SSA Context • SSA would need more detail than is provided via the 33 trait focused scales • In our judgment, the TTA rating tool is not appropriate for use as incumbent/supervisor survey. It should only be used by trained analysts

28. Summary: Overarching Recommendations and Potential Next Steps

29. Recommendations and Potential Next Steps • Identify Work Taxonomy and Constructs to be Measured • Measuring different constructs may necessitate different data collection procedures • Data Should be Collected and Stored Using a Computerized System or Online Application/Tool • Provides a centralized location for data collection and minimize potential security issues/concerns • Need to Determine Factors that are Most Important and Consider Job Analysis Practices Accordingly • Fully conceptualize multiple prototypes of integrated systems that SSA might use and compare the systems side-by-side • Integrated system= Occupational analysis system

30. Comparing Prototype Integrated Systems (i.e., Occupational Analysis Systems) Side-by-Side

31. Recommendations and Potential Next Steps • Full Methodology Must Include a Comprehensive Set of Procedures that Include Guidelines for Maintaining Data Security and Confidentiality • Features of Existing Job Analysis Models Should be Adapted for SSA’s Specific Purposes and Data Needs • Additional job analysis questions or more precise questions may need to be added to an existing questionnaire • Need Pilot Testing to Ensure that the Final Methodology Meets SSA Objectives • Ensures the final set of combined data collection procedures and/or models appropriately measure the desired characteristics

32. Questions? • For more information, please contact: Dr. Lance Anderson Vice President ICF International (703) 934-3000 LAnderson@icfi.com Dr. Brian Cronin Senior Manager ICF International (512) 388-3389 BCronin@icfi.com