Common Tools in Vocational Evaluation for Marginalized Client Populations

1. Common Tools in Vocational Evaluation for Marginalized Client Populations Phillip W. Boswell MA, B.Ed, RRP R. Stephen Russell, MA. Sc, P. Eng

2. Learning Objectives • Introduction • Identify the difference between vocational assessment and vocational evaluation • Recognize the three levels of vocational assessment • Ascertain the factors involved in the appraisal of clients • Identify useful voc assessment/evaluation resources • Develop a working knowledge of common tools used in vocational evaluations • Work samples

3. Assessment - Defined Systematic procedures to obtain information from a variety of sources to draw inferences about people (Standards for Educational and Psychological Testing, 1999)

4. Five Phases of Vocational Rehabilitation • Same job with the same employer. • Same employer, modified job (light duties) • Different employer, similar or closely related job • Different employer, alternative employment in the same or related industry • Development of new occupational skills

5. Functional skills (basic skills & functional capacity) Interest Achievement Aptitude Intelligence Learning style Work Samples Personality Assistive technology Neuropsychological Sensory impairments Career Assessments Dexterity What can be assessed? Career maturity, competency, employment barriers, emotional and multiple intelligence, job readiness, personal factors, personal style, prior learning and values

6. Factors in Appraisal of the Individual

7. Vocational Evaluation Tools

8. Individual Appraisal/Tools

9. Vocational Assessment - Defined A general term for the process of identifying and appraising an individual’s level of functioning in relation to vocational preparation and employment decision making

10. Why Vocational Assessment? • Clients often do not have precise occupational goals and aspirations. They frequently have insufficient or inaccurate information on the labour market and limited self-knowledge of their own interests and abilities.

11. Purpose of Vocational Assessment • Plan a course of action • Enhance client self-knowledge and vocational decision-making abilities • Predict realistic employment outcomes that result in successful client vocational rehabilitation

12. Role of the Rehabilitation Professional • Collect enough information or ‘data’ about the client to diagnose and make predictive statements about his/her potential to obtain a successful rehabilitation outcome

13. Levels of Vocational Assessment Level 1 – Screening/Needs Assessment Level 2 – Clinical or Exploratory Level 3 – Comprehensive Vocational Evaluation

14. Level 1 – Screening/Needs Assessment Initial Process • Needs Assessment • Determine what is necessary to develop a plan of action (e.g. vocational plan)

15. Level 1 – Screening/Needs Assessment Includes • Interview with client • Collect routine background information (demographics) • Reliance on subjective consumer statements • Vocational choice/interest • Self-estimates of competence • Reported work history • Functional Assessment (e.g. personal capacity questionnaire) • Limited, if any, standardized testing (e.g. interest)

16. Level 2 – Clinical or Exploratory Intermediate process In depth exploration or case study approach to the client and vocationally related circumstances. Builds on level 1 information through the use of: • Additional interviews • Collect/analyze documents (schools records, medical records, etc.) • Career exploration • Vocational and/or adjustment counselling • Psychometric/standardized testing • Transferable skills analysis • Computerized job matching • Job Analysis and/or environmental assessment • Assistive technology considerations

17. Level 3 – Vocational Evaluation “Final” Process • Comprehensive individualized holistic process of assessment that utilizes specific instruments, procedures, and behavioural observation • Designed to measure, observe, and document interests, values, temperaments, work-related behaviours, aptitudes, skills, and physical capacities, in order to predict viable employment and/or training outcomes • Used when more in depth information about the client is necessary and not available from information in level 1 and 2

18. Level 3 – Vocational Evaluation Real or simulated work used as the focal point of the evaluation • Systematic observation and recording of work behaviour & performance • Occurs over time and uses multiple methods and techniques to validate finds. Some combination of the following methods are used: • Work samples, job samples • Situational assessments, community based assessments • Standardized testing • Interview • Transferable skills, job matching • Analysis of background information • Career exploration/job shadow • Assistive technology considerations • Prescriptive recommendations

19. Distinction Between VA & VE VE is holistic: considers disability/medical, psychological, social, vocational, educational, cultural and economic issues Ideally conducted by a certified vocational evaluator (CVE) VE occurs over time (days/week +) • evaluated over time with varying work-related demands and environments Systematic Behaviour Observation and Recording • Work performance • Work behaviour Adaptive and/or transferable skills are questionable or unknown Work is the specific focus of VE • Work samples • Situational assessments • Community-based assessments • Occupational resources • Career exploration

20. If we are facing in the right direction, all we have to do is keep on walking Buddhist proverb

21. Transferable skills Vocational Interests Worker Characteristics and Behaviours Academic Abilities and Potential Learning Styles Work Aptitudes and Abilities Suitable Employment Alternatives Vocational evaluation identifies….

22. Evaluating Assessments When assessing an individual, the first question is always……Should we use tests at all? Testing may not always be appropriate • Extreme anxiety • Sensory impairments • Minimal education • Limited language abilities

23. Questions to ask • Is the test reliable (repeatable) and valid (does it measure what its suppose to?) • Does my client meet the reading level required by the test? • Is my client disadvantaged in any way by completing this test? • Will the test provide the information that I am looking for?

24. Questions to ask • Can the test administration procedures be modified to accommodate the client? • Can clients answer the question on the test form rather than a separate score sheet? • Are all the questions relevant and correct for my client?

25. Questions to ask • Are there relevant norms/criteria to compare my client’s performance? • Issue in rehabilitation is the lack of representation of marginalized individuals in sample populations • Marginalized client interested in competitive employment is better served using “general working population norms” rather than “disabled norms”

26. Transferable Skills • Can be achieved through formal or informal training • Is work related • Should have a measurable component • Should have the capacity to be matched to other/alternative work options

27. Vocational Interests • Subjective measure of an individual’s preferences in work activities • Can add validity by incorporating variety of surveys with similar outcomes • Does not confirm that skills exist

28. Worker Characteristics and Behaviours • Vocational Evaluation relies heavily on the observational skills of the Counsellor • The Counsellor “takes on” different roles throughout the evaluation • Allows the inclusion of subjective information in the process

29. Academic Abilities and Potential • Understanding current skills • Can evaluate need for upgrading • To competitively seek employment • To pursue further training • Can predict potential • Timed vs. untimed testing • Needed to insure proper skill profile is created

30. Learning Styles • Generally subjective in nature • Can be helpful in identifying training environment • Can be helpful in identifying suitable work environment

31. Work Aptitudes and Abilities • Computerized assessment • Work Samples • Academic Achievement • Aptitude testing (paper/pencil) • Work history (transferable skills) All are used to create a composite profile

32. Worker Qualification Profile • General Education Development (GED) • Reasoning Math Language • Aptitudes • General Learning, Verbal Aptitude, Numerical Aptitude, Spatial Aptitude, Form Perception, Clerical Perception, Motor Co-ordination, Finger Dexterity, Manual Dexterity, Eye-Hand-Foot Co-ordination, Colour Discrimination

33. General Education Development-GED • Embraces those aspects of education (formal or informal) which contribute to a a worker’s reasoning development and ability to follow instructions; and, the acquisition of “tool knowledge” such as language and mathematical skills. (Revised Handbook for Analyzing Jobs - 1991)

34. GED FACTORS • Reasoning Development • Math Development • Language Development 1 to 6 scale 1 = Low 6 = High

35. Aptitudes • The capacity to acquire proficiency in and activity with a given amount of formal or informal training • Aptitudes may be general, such as learning aptitude or special, such as mechanical aptitude (Pruitt, 1986)

36. General Learning Verbal Aptitude Numerical Aptitude Spatial Aptitude Form Perception Clerical Perception Motor Co-ordination Finger Dexterity Manual Dexterity Eye-Hand-Foot Co-ordination Colour Discrimination 1-5 scale 1=High 5=Low Aptitudes

37. The Tools of Vocational Evaluation • Interest Surveys • Academic Achievement and Aptitude Testing • Computerized Assessment • Work Samples • Observation • Other Surveys

38. Interest Surveys • Strong Interest inventory (SII) • Canadian Work Preference Inventory (CWPI) • Jackson Vocational Interest Survey (JVIS) • Career Occupational Preference System Interest Inventory (COPS) • Reading Free Vocational Interest Inventory: 2 (R-FVII:2) • Career Assessment Inventory (CAI)

39. Ability=Achievement+Aptitude • Achievement Tests - What a person has already done! • Canadian Academic Achievement Test (CAAT) • Wide Range Achievement Test (WRAT-4) • Aptitude Tests – What a person has the potential to do! • General Aptitude Test Battery (GATB) • Differential Aptitude Test (DAT)

40. Work Samples • Simulated work activity • Criterion Referenced Test (CRT) • Method Time Measurement (MTM) • Standardized tasks • Directly related to aptitude factors • Valpar has over 30 work samples for functional and job specific assessments

41. Valpar Work Samples • VCWS03 Numerical Sorting • VCWS06 Independent Problem Solving • VCWS07 Multi-Level Sorting

42. Norm-Referenced test & scores • Compares the evaluee’s performance to the performance of members of a particular norm group • Selected norm group population should be well defined, relatively current and appropriate to the evaluee’s goal • Results are reported in terms of percentile rankings

43. Norms • Frequently referred to as ‘moving targets’ as an individual’s ‘relative’ rank can shift significantly depending on the norm group Pearl scored at the 87th percentile in arithmetic (grade 4 students) Pearl scored at the 11th percentile in arithmetic (grade 10 students)

44. When norms are not enough • The world of work expects people to be able to perform some job tasks based on criterion of success or precision rather than compared only to how well others perform • Expect airline pilots to take off and land an airplane with 100% level of success, not just a little better than their pilot classmates!

45. Criterion references tests & scores • Criterion-referenced tests have a predetermined external standard or criterion • Work related criterion generally come from industry standards • Number of widgets assembled per hour • Keyboarding speed 60 wpm 0 errors • MTM (Valpar) • Measures a specific knowledge, skill, or trait and compares person’s performance against the criterion • Focus and importance of on performance rather than relative rank of how others performed

46. Norm vs criterion • The “Standards for Educational and Psychological Testing” provide a basis on which to evaluate tests. established by APA, National Council on Measurement in Education, & American Education Research Association • The samples used for test validation and norming must be of adequate size and must be sufficiently representative to substantiate validity statements, to establish appropriate norms, and to support conclusions regarding the use of the instrument for the intended purpose

47. A Huge Issue---Norm vs. Criterion • Most norm referenced tests do not use representative norm samples that include individuals with disabilities • This often results in labels of disability and the identification of weakness • Important to note that in some instances it is helpful to understand how an individual is functioning compared to the “average” population when assessing for competitive employment

48. Competitive: compares people to other people Relative skills reported Content difficulty fixed High scores inferred from high performance on tasks with lower scores Non-competitive: compares people directly to jobs Actual skills reported Content difficulty adjusts to evaluee’s skills Scores measured directly at corresponding content difficulty Norms Criterion

49. Validity depends on appropriateness of norm group Strictly valid only for members of the norm group Translation from percentiles by traditional methods distorts DOT levels Validated by Method Time Measurement (MTM). No norm groups Accuracy verified by data from over thousands of assessments Measures DOT levels directly: no translation required Norms Criterion

50. Best Norm Group (when available) • Norm to the environment you are predicting to • e.g. employed workers, general working population, general population, education, training norms • 62nd percentile Form & Spatial GATB (GWP) • 12th percentile MN Paper Form Board (employed drafting technicians) • Obviously, criterion-referenced tests are better indicators of performance, so if you have a choice between a norm referenced and criterion referenced measure – choose criterion-referenced.