common tools in vocational evaluation for marginalized client populations

1. Phillip W. Boswell MA, B.Ed, RRP R. Stephen Russell, MA. Sc, P. Eng Common Tools in Vocational Evaluation for Marginalized Client Populations

3. Assessment - Defined Systematic procedures to obtain information from a variety of sources to draw inferences about people (Standards for Educational and Psychological Testing, 1999)

4. Five Phases of Vocational Rehabilitation Same job with the same employer. Same employer, modified job (light duties) Different employer, similar or closely related job Different employer, alternative employment in the same or related industry Development of new occupational skills

5. What can be assessed? Functional skills (basic skills & functional capacity) Interest Achievement Aptitude Intelligence Learning style Work Samples Personality Assistive technology Neuropsychological Sensory impairments Career Assessments Dexterity

6. Factors in Appraisal of the Individual

7. Vocational Evaluation Tools

8. Individual Appraisal/Tools

9. Vocational Assessment - Defined A general term for the process of identifying and appraising an individual�s level of functioning in relation to vocational preparation and employment decision making

10. Why Vocational Assessment? Clients often do not have precise occupational goals and aspirations. They frequently have insufficient or inaccurate information on the labour market and limited self-knowledge of their own interests and abilities.

11. Purpose of Vocational Assessment Plan a course of action Enhance client self-knowledge and vocational decision-making abilities Predict realistic employment outcomes that result in successful client vocational rehabilitation

12. Role of the Rehabilitation Professional Collect enough information or �data� about the client to diagnose and make predictive statements about his/her potential to obtain a successful rehabilitation outcome

13. Levels of Vocational Assessment

14. Level 1 � Screening/Needs Assessment Initial Process Needs Assessment Determine what is necessary to develop a plan of action (e.g. vocational plan)

15. Level 1 � Screening/Needs Assessment Includes Interview with client Collect routine background information (demographics) Reliance on subjective consumer statements Vocational choice/interest Self-estimates of competence Reported work history Functional Assessment (e.g. personal capacity questionnaire) Limited, if any, standardized testing (e.g. interest) PCQ � Personal capacities Questionnaire PCQ � Personal capacities Questionnaire

16. Level 2 � Clinical or Exploratory Intermediate process In depth exploration or case study approach to the client and vocationally related circumstances. Builds on level 1 information through the use of: Additional interviews Collect/analyze documents (schools records, medical records, etc.) Career exploration Vocational and/or adjustment counselling Psychometric/standardized testing Transferable skills analysis Computerized job matching Job Analysis and/or environmental assessment Assistive technology considerations

17. Level 3 � Vocational Evaluation �Final� Process Comprehensive individualized holistic process of assessment that utilizes specific instruments, procedures, and behavioural observation Designed to measure, observe, and document interests, values, temperaments, work-related behaviours, aptitudes, skills, and physical capacities, in order to predict viable employment and/or training outcomes Used when more in depth information about the client is necessary and not available from information in level 1 and 2

18. Level 3 � Vocational Evaluation Real or simulated work used as the focal point of the evaluation Systematic observation and recording of work behaviour & performance Occurs over time and uses multiple methods and techniques to validate finds. Some combination of the following methods are used: Work samples, job samples Situational assessments, community based assessments Standardized testing Interview Transferable skills, job matching Analysis of background information Career exploration/job shadow Assistive technology considerations Prescriptive recommendations

19. Distinction Between VA & VE VE is holistic: considers disability/medical, psychological, social, vocational, educational, cultural and economic issues Ideally conducted by a certified vocational evaluator (CVE) VE occurs over time (days/week +) evaluated over time with varying work-related demands and environments Systematic Behaviour Observation and Recording Work performance Work behaviour Adaptive and/or transferable skills are questionable or unknown Work is the specific focus of VE Work samples Situational assessments Community-based assessments Occupational resources Career exploration

20. If we are facing in the right direction, all we have to do is keep on�walking Buddhist proverb

21. Vocational evaluation identifies�. Transferable skills Vocational Interests Worker Characteristics and Behaviours Academic Abilities and Potential Learning Styles Work Aptitudes and Abilities Suitable Employment Alternatives

22. Evaluating Assessments When assessing an individual, the first question is always��Should we use tests at all? Testing may not always be appropriate Extreme anxiety Sensory impairments Minimal education Limited language abilities

23. Questions to ask Is the test reliable (repeatable) and valid (does it measure what its suppose to?) Does my client meet the reading level required by the test? Is my client disadvantaged in any way by completing this test? Will the test provide the information that I am looking for? Reading - if your client can�t read the questions in a mechanical reasoning test, you are measuring reading ability, not mechanical reasoning Disadvantaged - asking an individual with a high level spinal cord injury to complete a tasked that is speeded (your score depends on the number completed within a certain amount of time) will significantly disadvantage your client due to upper limb paralysis or dysfunction Mark - Keep in mind that accommodating your client by changing standardized administration procedures will add error to the true score Looking for � Be sure that you know exactly what the test purports to measure Reading - if your client can�t read the questions in a mechanical reasoning test, you are measuring reading ability, not mechanical reasoning Disadvantaged - asking an individual with a high level spinal cord injury to complete a tasked that is speeded (your score depends on the number completed within a certain amount of time) will significantly disadvantage your client due to upper limb paralysis or dysfunction Mark - Keep in mind that accommodating your client by changing standardized administration procedures will add error to the true score Looking for � Be sure that you know exactly what the test purports to measure

24. Questions to ask Can the test administration procedures be modified to accommodate the client? Can clients answer the question on the test form rather than a separate score sheet? Are all the questions relevant and correct for my client? Modified - limited vision/large font, read the questions out loud) Mark - Keep in mind that accommodating your client by changing standardized administration procedures will add error to the true score Relevant - (WRAT-3 spelling test enter, recognize Asking a person from Iqaluit �When does the sun rise? (In the spring/morning) Modified - limited vision/large font, read the questions out loud) Mark - Keep in mind that accommodating your client by changing standardized administration procedures will add error to the true score Relevant - (WRAT-3 spelling test enter, recognize Asking a person from Iqaluit �When does the sun rise? (In the spring/morning)

25. Questions to ask Are there relevant norms/criteria to compare my client�s performance? Issue in rehabilitation is the lack of representation of marginalized individuals in sample populations Marginalized client interested in competitive employment is better served using �general working population norms� rather than �disabled norms�

26. Transferable Skills Can be achieved through formal or informal training Is work related Should have a measurable component Should have the capacity to be matched to other/alternative work options

27. Vocational Interests Subjective measure of an individual�s preferences in work activities Can add validity by incorporating variety of surveys with similar outcomes Does not confirm that skills exist

28. Worker Characteristics and Behaviours Vocational Evaluation relies heavily on the observational skills of the Counsellor The Counsellor �takes on� different roles throughout the evaluation Allows the inclusion of subjective information in the process

29. Academic Abilities and Potential Understanding current skills Can evaluate need for upgrading To competitively seek employment To pursue further training Can predict potential Timed vs. untimed testing Needed to insure proper skill profile is created

30. Learning Styles Generally subjective in nature Can be helpful in identifying training environment Can be helpful in identifying suitable work environment

31. Work Aptitudes and Abilities Computerized assessment Work Samples Academic Achievement Aptitude testing (paper/pencil) Work history (transferable skills) All are used to create a composite profile

32. Worker Qualification Profile General Education Development (GED) Reasoning Math Language Aptitudes General Learning, Verbal Aptitude, Numerical Aptitude, Spatial Aptitude, Form Perception, Clerical Perception, Motor Co-ordination, Finger Dexterity, Manual Dexterity, Eye-Hand-Foot Co-ordination, Colour Discrimination

33. General Education Development-GED Embraces those aspects of education (formal or informal) which contribute to a a worker�s reasoning development and ability to follow instructions; and, the acquisition of �tool knowledge� such as language and mathematical skills. (Revised Handbook for Analyzing Jobs - 1991)

34. GED FACTORS Reasoning Development Math Development Language Development 1 to 6 scale 1 = Low 6 = High

35. Aptitudes The capacity to acquire proficiency in and activity with a given amount of formal or informal training Aptitudes may be general, such as learning aptitude or special, such as mechanical aptitude (Pruitt, 1986)

36. Aptitudes General Learning Verbal Aptitude Numerical Aptitude Spatial Aptitude Form Perception Clerical Perception Motor Co-ordination Finger Dexterity Manual Dexterity Eye-Hand-Foot Co-ordination Colour Discrimination 1-5 scale 1=High 5=Low

37. The Tools of Vocational Evaluation Interest Surveys Academic Achievement and Aptitude Testing Computerized Assessment Work Samples Observation Other Surveys

38. Interest Surveys Strong Interest inventory (SII) Canadian Work Preference Inventory (CWPI) Jackson Vocational Interest Survey (JVIS) Career Occupational Preference System Interest Inventory (COPS) Reading Free Vocational Interest Inventory: 2 (R-FVII:2) Career Assessment Inventory (CAI)

39. Ability=Achievement+Aptitude Achievement Tests - What a person has already done! Canadian Academic Achievement Test (CAAT) Wide Range Achievement Test (WRAT-4) Aptitude Tests � What a person has the potential to do! General Aptitude Test Battery (GATB) Differential Aptitude Test (DAT)

40. Work Samples Simulated work activity Criterion Referenced Test (CRT) Method Time Measurement (MTM) Standardized tasks Directly related to aptitude factors Valpar has over 30 work samples for functional and job specific assessments

41. Valpar Work Samples VCWS03 Numerical Sorting VCWS06 Independent Problem Solving VCWS07 Multi-Level Sorting

42. Norm-Referenced test & scores Compares the evaluee�s performance to the performance of members of a particular norm group Selected norm group population should be well defined, relatively current and appropriate to the evaluee�s goal Results are reported in terms of percentile rankings

43. Norms Frequently referred to as �moving targets� as an individual�s �relative� rank can shift significantly depending on the norm group Pearl scored at the 87th percentile in arithmetic (grade 4 students) Pearl scored at the 11th percentile in arithmetic (grade 10 students)

44. When norms are not enough The world of work expects people to be able to perform some job tasks based on criterion of success or precision rather than compared only to how well others perform Expect airline pilots to take off and land an airplane with 100% level of success, not just a little better than their pilot classmates! During World War II, the U.S. government discovered that its parachutes failed to open 5% of the time. Requiring parachute packers and inspectors to put on one of their products occasionally and jump out of a plane solved the problem. Parachutes quality quickly improved from 95 to 100% During World War II, the U.S. government discovered that its parachutes failed to open 5% of the time. Requiring parachute packers and inspectors to put on one of their products occasionally and jump out of a plane solved the problem. Parachutes quality quickly improved from 95 to 100%

45. Criterion references tests & scores Criterion-referenced tests have a predetermined external standard or criterion Work related criterion generally come from industry standards Number of widgets assembled per hour Keyboarding speed 60 wpm 0 errors MTM (Valpar) Measures a specific knowledge, skill, or trait and compares person�s performance against the criterion Focus and importance of on performance rather than relative rank of how others performed

46. Norm vs criterion The �Standards for Educational and Psychological Testing� provide a basis on which to evaluate tests. established by APA, National Council on Measurement in Education, & American Education Research Association The samples used for test validation and norming must be of adequate size and must be sufficiently representative to substantiate validity statements, to establish appropriate norms, and to support conclusions regarding the use of the instrument for the intended purpose

47. A Huge Issue---Norm vs. Criterion Most norm referenced tests do not use representative norm samples that include individuals with disabilities This often results in labels of disability and the identification of weakness Important to note that in some instances it is helpful to understand how an individual is functioning compared to the �average� population when assessing for competitive employment

48. Norms Criterion Competitive: compares people to other people Relative skills reported Content difficulty fixed High scores inferred from high performance on tasks with lower scores Non-competitive: compares people directly to jobs Actual skills reported Content difficulty adjusts to evaluee�s skills Scores measured directly at corresponding content difficulty

49. Norms Criterion Validity depends on appropriateness of norm group Strictly valid only for members of the norm group Translation from percentiles by traditional methods distorts DOT levels Validated by Method Time Measurement (MTM). No norm groups Accuracy verified by data from over thousands of assessments Measures DOT levels directly: no translation required

50. Best Norm Group (when available) Norm to the environment you are predicting to e.g. employed workers, general working population, general population, education, training norms 62nd percentile Form & Spatial GATB (GWP) 12th percentile MN Paper Form Board (employed drafting technicians) Obviously, criterion-referenced tests are better indicators of performance, so if you have a choice between a norm referenced and criterion referenced measure � choose criterion-referenced.

51. Computerized Assessment Bridges Ability Profile Multi-Aptitude Assessment Bridges Interest Profiler Career Cruising - Career Matchmaker Valpar Professional (PRO 3000) (Magellan and Aviator)

52. PRO3000 System Manager Computerized Assessment (COMPASS) with 3 specialized work samples DOT Descriptions and database Work History for transferable skills analysis Pictorial/Audio Interest Survey Spatial/Non-verbal PET Survey Norm Referenced Test Conversion (TECO)

53. Suitable Employment Alternatives Counsellor directed selection of vocational options Objective listing based on testing results Can be combined with career exploration

54. The Client Perspective Generally 3 to 6 hours Includes intake session Pre-screening Academic achievement testing Interest / Aptitude Testing Discussion of test results

55. VALPAR International Started in 1973 in Tucson, Arizona Product Developer with over 6000 customers world wide Component Work Samples Series (VCWS) JOULE Functional Capacity Evaluation PRO3000 Professional Vocational Assessment MAGELLAN Career Self Assessment/Explorer AVIATOR Aptitude & Interest Assessment SIGI PLUS Values-Based Career Planning

56. Career Systems Canada Limited Steve Russell P. O. Box 255 Buckhorn, ON. K0L1J0 tel: 705 657-8646 fax: 705 657-1572 cell: 416.721.5759 steverussell@necicom.net www.valparint.com

57. P.W. Boswell & Associates Phil Boswell 9534 Riverbend Road Black Creek, BC� V9J 1E8 tel 250.337.5087 fax 250.337.5867 toll� 877.790.7074 info@employment-training.org www.employment-training.org