Psychometric Testing within the SANDF: An Overview and Best Practices

1. PSYCHOMETRIC TESTING WITHIN THE SANDFPRESENTATATION TO THE PORTFOLIO COMMITTEE ON DEFENCE19 JUNE 2007

2. SCOPE • Statutory Control of the use of Psychological Assessment Measures within the SANDF. • The Psychometric Tests used within the SANDF. • The Role and Function of Assessment Centres within the SANDF. • The Application of Specialist Psychological Measures in the Recruitment and Selection of Pilots within the SANDF.

3. STATUTORY CONTROL OF THE USE OF PSYCHOLOGICAL MEASURES WITHIN THE SANDF

4. SCOPE (1) • Terminology • Characteristics of Assessment Measures • The Need for Control of Assessment Measures within the SANDF • Control of Psychological Assessment Measures within the SANDF • Psychological Assessment within the Democratic South Africa • Fair and Ethical Practices in the Use of Psychological Measures within the SANDF

5. SCOPE (2) • Factors Affecting Psychological Assessment Results • Professional Practices that Assessment Practitioners within the SANDF should follow • Basis Statistical Concepts: Reliability, Validity and Norms • Challenges faced by Psychological Assessment Practitioners within the SANDF

6. TERMINOLOGY (1) • Importantant Terms • Confusing and overlapping terms are used in the field of psychological assessment. • Understand the more important terms and how they are interlinked. • Tools are available to make it possible for us to assess (measure) human behaviour. • Various names are used to refer to these tools such as tests, measures, assessment measures, instruments, scales, procedures, and techniques.

7. TERMINOLOGY (2) • To ensure that psychological measurement is valid and reliable, a body of theory and research regarding the scientific measurement principles that are applied to the measurement of psychological characteristics has evolved over time. • Psychometrics • Refers to the systematic and scientific way in which psychological measures are developed and the technicalmeasurement standards (e.g. validity and reliability) required of measures.

8. TERMINOLOGY (3) • Psychological Assessment • A process-orientated activity aimed at gathering a wide array of information by using assessment measures(tests) and information from many other sources (e.g. interviews, a person’s history, collateral sources). • Evaluate and integrate all information to reach a conclusion or make a decision. • Testing • The use of tests, measures, etc. which involves the measurement of behaviour, is one of the key elements of the much broader evaluative process known as psychological assessment.

9. TERMINOLOGY (4) • Assessment Measure • In the SANDF preference is given to the term assessment measure as it is a broader connotation than the term test, which mainly refers to an objective, standardised measure that is used to gather data for a specific purpose (e.g. to determine what a person’s intellectual capacity is).

10. CHARACTERISTICS OF ASSESSMENT MEASURES (1) • Different Procedures • Assessment measures include many different procedures that can be used in psychological assessment and can be administered to individuals, groups and organisations. • Domains of Functioning • Specific domains of functioning (e.g. intellectual ability, personality, organisational climate) are sampled by assessment measures. • Standardised Conditions • Assessment measures are administered under carefully controlled (standardised) conditions.

11. CHARACTERISTICS OF ASSESSMENT MEASURES (2) • Systematic Methods • Systematic methods are applied to score or evaluate assessment protocols. • Guidelines • Guidelines are available to understand and interpret the results of an assessment measure. • Such guidelines may make provision for the comparison of an individual’s performance to that of an appropriate norm group or to a criterion (e.g. competency profile for a job).

12. CHARACTERISTICS OF ASSESSMENT MEASURES (3) • Evidence Based • Assessment measures should be supported by evidence that they are valid and reliable for the intended purpose. • The evidence is usually provided in the form of a technical test manual. • Context • Assessment measures are usually developed in a certain context (society or culture) for a specific purpose and the normative information used to interpret test performance is limited to the characteristics of the normative sample.

13. CHARACTERISTICS OF ASSESSMENT MEASURES (4) • Test Bias • The appropriateness of an assessment measure for an individual, group, or organisation from another context, culture, or society cannot be assumed without an investigation into possible test bias (i.e. whether a measure is differently valid for different subgroups) and without strong consideration being given to adapting and re-norming the measure. • Multidimensional • Assessment process is multidimensional in nature. • It entails the gathering and synthesising of information as a means of describing and understanding functioning. • This can inform appropriate decision-making and intervention.

14. CHARACTERISTICS OF ASSESSMENT MEASURES (5) • Limits of Human Wisdom • Recognise the limits of human wisdom when reaching opinions based on assessment information.

15. THE NEED FOR CONTROL OF ASSESSMENT MEASURES WITHIN THE SANDF (1) • Sensitive Item Content • In view of the potentially sensitive nature of some of the itemcontent and the feedback, and given that assessment measures can be misused, the use of assessment measures need to be controlled so that the public can be protected. • Trained Professionals • Controlling the use of psychological measures by restricting them to appropriately trained professionals.

16. THE NEED FOR CONTROL OF ASSESSMENT MEASURES WITHIN THE SANDF (2) • Practitioner Competency • Measures are administrated by a qualified, competentassessment practitioner and that assessment results are correctly interpreted and used. • Conveying the Results • The outcome of the assessment is conveyed in a sensitive, empowering manner rather than in a harmful way. • Psychometry Procurement • The purchasing of psychological assessment measures is restricted to those who may use them and that test materials are kept securely (as it is unethical for assessment practitioners to leave tests lying around) – this will prevent unqualified people from gaining access to and using them.

17. THE NEED FOR CONTROL OF ASSESSMENT MEASURES WITHIN THE SANDF (3) • Release of Assessment Materials • Test developers do notprematurely release assessmentmaterials (e.g. before validity and reliability have been adequately established), as it is unethical for assessment practitioners to use measures for which appropriate validity and reliability data have not been established. • Public Familiarity • The general public does not become familiar with the test content, as this would invalidate the measure.

18. CONTROL OF PSYCHOLOGICAL ASSESSMENT MEASURES WITHIN THE SANDF (1) • Statutory Control in RSA • In South Africa the use of psychological assessment measures is under statutory control. • A law (statute) has been promulgated that restricts the use of psychological assessment measures to appropriately registered psychology professionals. • Health Professions Act • Act 56 of1974 defines acts “specially pertaining to the profession of a psychologist”.

19. CONTROL OF PSYCHOLOGICAL ASSESSMENT MEASURES WITHIN THE SANDF (2) • Diagnosis • The evaluation of behaviour or mental processes or personality adjustments or adjustments of individuals or groups of persons, through the interpretation of tests for the determination of intellectual abilities, aptitude, interests, personality make-up or personality functioning, and the diagnosis of personality and emotional functions and mental functioning deficiencies according to a recognised scientific system for the classification of mental deficiencies.

20. CONTROL OF PSYCHOLOGICAL ASSESSMENT MEASURES WITHIN THE SANDF (3) • Method and Practice • The use of any method or practice aimed at aiding persons or groups of persons in the adjustment of personality, emotional or behavioural problems or at the promotion of positive personality change, growth and development, and the identification and the evaluation of personality dynamics and personality functioning according to psychological scientific methods.

21. CONTROL OF PSYCHOLOGICAL ASSESSMENT MEASURES WITHIN THE SANDF (4) • Evaluation • The evaluation of emotional, behavioural and cognitive processes or adjustment of personality of individuals or groups of persons by the usage and interpretation of questionnaires, tests, projections or other techniques or any apparatus, whether of South African origin or imported, for the determination of intellectual abilities, aptitude, personality make-up, personality functioning, psychophysiological functioning or psychopathology.

22. CONTROL OF PSYCHOLOGICAL ASSESSMENT MEASURES WITHIN THE SANDF (5) • Exercising of Control • The exercising of control over prescribed questionnaires or tests or prescribed techniques, apparatus or instruments for the determination of intellectual abilities, aptitude, personality make-up, personality functioning, psychophysiological functioning or psychopathology. • Development • The development of and control over the development of questionnaires, tests, techniques, apparatus or instruments for the determination of intellectual abilities, aptitude, personality make-up, personality functioning, psychophysiological functioning or psychopathology.

23. CONTROL OF PSYCHOLOGICAL ASSESSMENT MEASURES WITHIN THE SANDF (6) • Domain of Psychology • According to Act 56 of 1974, the use of measures to assess mental, cognitive, or behavioural processes and functioning, intellectual or cognitive ability or functioning, aptitude, interest, emotions, personality, psychophysiological functioning or psychopathology (abnormal behaviour), constitutes an act that fall in the domain of the psychology profession.

24. PSYCHOLOGICAL ASSESSMENT WITHIN THE DEMOCRATIC SOUTH AFRICA (1) • Post 1994 • Since 1994 and the election of South Africa’s first democratic government, the application, control, and development of assessment measures have become contested terrain. • Constitution and Labour Relations Act • With the adoption of the new Constitution and the Labour RelationsAct in 1996, worker unions and individuals now have the support of legislation that specifically forbids any discriminatorypractices in the workplace and includes protection for applicants as they have all the rights of current employees in this regard.

25. PSYCHOLOGICAL ASSESSMENT WITHIN THE DEMOCRATIC SOUTH AFRICA (2) • Employment Equity Act • To ensure that discrimination is addressed within the testing arena, the Employment Equity Act No. 55 of 1998 (section 8) refers to psychological tests and assessment specifically and states that: • Psychological testing and other similar forms or assessments of an employee are prohibited unless the test or assessment being used: • has been scientifically shown to be valid and reliable; • can be applied fairly to all employees; • is not biased against any employee or group.

26. PSYCHOLOGICAL ASSESSMENT WITHIN THE DEMOCRATIC SOUTH AFRICA (3) • Impact of Employment Equity Act • The impact of this Act on the conceptualisation and professionalpractice of assessment in South Africa in general is far-reaching as assessment practitioners and test publishers are increasingly being called upon to demonstrate, or prove in court, that a particular assessment measure does not discriminate against certain groups of people. • Despite the fact that the Employment Equity Act is not binding on Defence Act Personnel, Directorate Psychology is still obliged to ensure that its practices are fair and equitable.

27. FAIR AND ETHICAL PRACTICES IN THE USE OF PSYCHOLOGICAL MEASURES WITHIN THE SANDF • International Guidelines on Test Use (Version 2000) • Fair Assessment Practices • The appropriate, fair, professional, and ethical use of assessment measures and assessment results. • Taking into account the needs and rights of those involved in the assessment process. • Ensuring that the assessment conducted closely matches the purpose to which the assessment results will be put. • Taking into account the broader social,cultural, and political context in which assessment is used and the ways in which such factors might affect assessment results, their interpretation, and the use to which they are put.

28. FACTORS AFFECTING PSYCHOLOGICAL ASSESSMENT RESULTS (1) • Viewing Assessment Results in Context. • A test score in only one piece of information about how a person performs or behaves. Therefore, if we look at an individual in terms of a test score only, we will have a very limited understanding of that person. • A test score can never be interpreted without taking note of and understanding the context in which the score was obtained. • In addition to the test score, the information in which we are interested can be obtained by examining the context in which a person lives. • When you think about it, you will realise that people actually function in several different contexts concurrently.

29. FACTORS AFFECTING PSYCHOLOGICAL ASSESSMENT RESULTS (2) • At the lowest level there is the biological context, referring to physical bodily structures and functions, which are the substrata for human behaviour and experiences. • Then there is the intrapsychic context which comprises abilities, emotions, and personal dispositions. • Biological and intrapsychic processes are regarded as interdependent components of the individual as a psychobiological entity. • In addition, because people do not live in a vacuum, we need to consider a third and very important context which is the social context.

30. FACTORS AFFECTING PSYCHOLOGICAL ASSESSMENT RESULTS (3) • The social context refers to aspects of the environment in which we live such as our homes and communities, people with whom we interact, work experiences, as well as cultural and socio-political considerations. • Methodological Considerations • In addition to looking at the effects of the different contexts within which people function, we also need to examine methodological considerations such as test administration, which may also influence test performance and therefore have a bearing on the interpretation of a test score.

31. Professional Practices that Assessment Practitioners within the SANDF should follow (1) • Rights of Test-takers • Informing test-takers about their rights and the use to which the assessment information will be put. • Informed Consent • Obtaining the consent of test-takers to assess them, to use the results for selection, placement, or training decisions and, if needs be, to report the results to relevant third parties. • Treatment • Treating test-takers courteously,respectfully, and in an impartial manner, regardless of culture, language, gender, age, disability, and so on.

32. Professional Practices that Assessment Practitioners within the SANDF should follow (2) • Preparation • Being thoroughly prepared for the assessment session. • Confidentiality • Maintaining confidentiality to the extent that it is appropriate for fair assessment practices. • Language • Establishing what language would be appropriate and fair to use during the assessment and making use of bilingual assessment where appropriate. • Training • Only using measures that they have been trained to use.

33. Professional Practices that Assessment Practitioners within the SANDF should follow (3) • Administration • Administering measures properly. • Scoring • Scoring the measures correctly and using appropriate norms or cutpoints or comparative profiles. • Background Information • Taking background factors into account when interpreting test performance and when forming an overall picture of the test-taker’s performance (profile).

34. Professional Practices that Assessment Practitioners within the SANDF should follow (4) • Communication • Communicating the assessment results clearly to appropriate parties. • Subjectivity • Acknowledging the subjective nature of the assessment process by realising that the final decision that they reach, while based at times on quantitative test information, reflects their “best guess estimate”. • Utilisation of Assessment Information • Using assessment information in a fair, unbiased manner and ensuring that anyone else who has access to this information also does so.

35. Professional Practices that Assessment Practitioners within the SANDF should follow (5) • Research • Researching the appropriateness of the measures that they use and refining, adapting, or replacing them where necessary. • Storage • Securely storing and controlling access to assessment materials so that the integrity of the measures cannot be threatened in any way.

36. BASIC STATISTICAL CONCEPTS: RELIABILITY, VALIDITY AND NORMS (1) • Statistical Concepts • Psychological assessment measures often produce data in the form of numbers. • We need to be able to make sense of these numbers. • Basic statistical concepts can help us here, as well as when it comes to establishing and interpreting norm scores. • Statistical concepts and techniques can also help us to understand and establish basic psychometric properties of measures such as validity and reliability.

37. BASIC STATISTICAL CONCEPTS: RELIABILITY, VALIDITY AND NORMS (2) • Reliability • This refers to the degree to which a psychometric test consistently produces the same results by the same candidates. • Validity • This refers to the degree to which the psychometric test measures what it claims to measure.

38. BASIC STATISTICAL CONCEPTS: RELIABILITY, VALIDITY AND NORMS (2) • Norms • Norms refer to the recordsof performance by other candidates who have previously been assessed using the same test. • A candidate must be measured against norms taken from the context and population group to which that candidate belongs, i.e candidates are measured against other South African candidates who have previously undergone assessment on a specific test. • As a database of results is built, SANDF specific norms are developed and used. • The current SANDF database consists of primarily Black candidates.

39. NORMAL DISTRIBUTION: RAW SCORES TRANSLATED INTO NORMED SCORE

40. CLASSIFICATION OF STANINE SCALE

41. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (1) • Influence of Multiculturalism • In the latter part of the twentieth century and at the start of the twenty-first century, multiculturalism has become the norm in many countries. • As a result, attempts were made to develop tests that were “culture-free”. • It soon became clear that it was not possible to develop a test that is free of any cultural influences. • Consequently, test developers focused more on “culture-reduced”or “culture-common” tests in which the aim was to remove as much cultural bias as possible from the test by including only behaviour that was common across cultures.

42. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (2) • For example, a number of non-verbal intelligence tests were developed (e.g. Raven Progressive Matrices) where the focus was on novel problem-solving tasks and in which language use, which is often a stumbling block in cross-cultural tests, was minimised. • In an attempt to address issues of fairness and bias in test use, the need arose to develop standards for the professional practice of testing and assessment.

43. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (3) • Representivity of Assessors • Legitimate concern is sometimes expressed regarding the representivity of psychologists in the SANDF. • This is a challenge that the organisation is currently striving to meet and some degree of progress has already been made. • Directorate Psychology conducts targeted recruitment in order to recruit Black psychologists, and regularly engages the Professional Board for Psychology and academicinstitutions in this regard.

44. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (4) • However, the lack of availability of Black psychologists in South Africa remains a challenge. • The Professional Board for Psychology’s official registrationstatistics reflect that 11% (known disclosures) of South African psychologists are Black.

45. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (5) • Language • Language is generally regarded as the most important singlemoderator of performance on assessment measures. • This is because performance on assessment measures could be the product of language difficulties and not ability factors if a measure is administered in a language other than the test-taker’s home language. • When a test is written in a different language, it may present a rangeof concepts that are not accessible in our home language.

46. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (6) • Current Dilemma regarding Psychometric Tests • Historically the Human Science Research Council (HSRC) was mandated to provide cost effective psychometric tests that had been proven to be valid within the South African population. • After the advent of democracy in the Republic of South Africa, the HSRC underwent transformation. • The HSRC redefined its role regarding psychometric tests and surrendered the license to most of these tests to the private sector. • This led to the current dilemma where there is a shortage of cost effective psychometric instruments that are approved for use in the Republic of South Africa.

47. CHALLENGES FACED BY PSYCHOLOGY ASSESSMENT PRACTITIONERS WITHIN THE SANDF (7) • The situation has reached critical proportions within the broader industry sector. • Consequently, members of the Professional Board forPsychology have indicated that the HSRC will be requested to provide this essential service to the nation. • Due to the scarcity of validated psychometry for the South African context, the South African National Defence Force has been obliged to develop or validate some psychometric tests for use within the organisation. • This is done in consultation with the Psychometric Committee of the Professional Board for Psychology.

48. PSYCHOMETRIC TESTS USED IN THE SANDF

49. SCOPE • Academic Aptitude Test (AAT). • Blox Test. • Differential Aptitude Test (DAT). • Raven’s Progressive Matrices (RPM). • Potential Insight Battery (PIB). • Psychological Risk Inventory (PRI). • Vienna Test System (VTS).

50. ACADEMIC APTITUDE TEST (1) • Origin • South Africa. • Human Sciences Research Council. • Representative sample. • Different languages. • Northern Sotho Zulu • Southern Sotho Afrikaans • Tswana English • Tsonga Other • Venda • Xhosa