ASTB Update

1. ASTB Update CAPT John K. Schmidt, PhD MSC USN Operational Psychology Department Naval Aerospace Medical Institute DET Naval Operational Medicine Institute

2. Objectives Review History, Purpose, Content, and Scoring of the ASTB in Naval Aviation Where we have been Provide an Update on ASTB Administration Rates, Performance Trends, and Demographic Breakouts (Incl. Predictive Relationships w/Training Criteria & Student Attrition) Where we are now Present Short-Run and Long-Term ASTB Revision, Upgrade, Validation, and Administration Efforts Where we are going

3. Background & Overview Primary tool used to select SNAs and SNFOs flight officers for the USN, USMC, & USCG Different sections of the battery are also used to select OCS candidates and Intel officers ASTB was first implemented in 1942, with revisions in 1953, 1971, and 1992 1992 ASTB revision was validated separately for SNAs and SNFOs; also developed to be bias-free for gender/ethnicity (with ETS)

4. ASTB Content & Scoring The 1992 ASTB primarily consists of four subtests* Math - Verbal Test (MVT) Mechanical Comprehension Test (MCT) Spatial Apperception Test (SAT) Aviation and Nautical Test (ANI) *Discontinued Biographical Inventory & Aviation Interest Weighted subtest score combinations yield stanine (1–9): Academic Qualification Rating (AQR) - Predictor of performance in academic portions of API and ground school (also predicts attrition) Pilot Flight Aptitude Rating (PFAR) & Flight Officer Aptitude Rating (FOFAR) - Predictors of flight grades in primary flight training (also predicts attrition)

5. ASTB Administration 150 permanent custody locations: OSOs, NRDs, NROTCs, USNA, & select air stations All completed ASTBs are returned to NOMI for scoring, reporting, & database inclusion NOMI procures, distributes, scores, & controls of ALL ASTB test materials. On-line delivery system for the ASTB (APEX) was developed and is in use at over 20 sites Had successful overseas and shipboard administrations Transitioning to primarily on-line administration at all permanent custody locations in FY05

6. ASTB Administration Rates AVG 10,000 ASTBs (full or partial) per year Over 120,000 1992 versions taken (June 04) 75,000 Full ASTBs (Total Aviation Examinees) 19,000 ASTB Retests 26,000 OAR Exams 5% of all ASTBs currently taken on APEX

7. Historical ASTB Pass Rates by Service and Program

8. Predictive Relationships with Training Outcomes and Attrition Validity data presented for FY98-FY04 The validity coefficient (higher is better) Test Components: AQR: Predictor of NASC NSS SNA: USN .46 USMC .39 SNFO: USN .46 USMC .34 “Excellent” PFAR & FOFAR: Predictors of Primary NSS SNA: USN .32 USMC .21 SNFO: USN .27 USMC .18 “Good”

9. Predictive Relationships Among ASTB Components and SNA AttritionFY 98-FY 04 JUN

10. *: N<50*: N<50

11. Predictive Relationships Among ASTB Components and SNFO AttritionFY 98-FY 04 JUN

12. *: N<50*: N<50

13. Reasons for Attrition (from CNATRA Database)FY 95-FY 04 May

14. Semi-annual CNATRA Attrition Survey Received responses from 73% of attriters (n=120); 49% of those surveyed DORed Top three reason for DORs: Loss of (or never had) interest in flying (27%) Apprehension of flying or thinking of it (26%) Nervousness/Anxiety with flight program (22%)

15. Parallel Forms Development NAMI began item development for replacements of 4 current subtests in JUN 02; 2500 items developed by 30 item-writers over 13 mo Reduced these to 4 types (2 booklets each) 200 Math 172 Aviation Info 131 Reading Comp 152 Mech Comp NETC provided$130K for travel,printing, & labor

16. Parallel Form Development Phases I: 5 mo Data Collection OCT03-FEB04 Give ASTB and =1 of 8 prototype tests to 400 students each plus an AFOQT/ASTB sample At 1 test/student, required N at 400/prototype was 3200 Data collected from 18 sites including NASC, OCS, NROTCs, USNA, & TBS

17. PF Data Collection Effort

18. Parallel Form Development Phases II: 3 mo Data Analyses Mar04-May04 Compared current ASTB scores to prototype scores Created 3 new forms of Math Skills, Reading Comp, Mech Comp, Spatial Appreception, & Aeronautical Info Divided MVT into separate Math Skills & Reading Comp subtests Created Aviation Supplemental Test (AST) using items types from other 5 subtests; AST scores are figured into ASTB score computation using a proprietary algorithm Score components still: AQR, PFAR, FOFAR, & OAR Stat techniques used to scale the new subtests means old scores (Forms 1 & 2) can be compared to new ones (Forms 3, 4, & 5) (Note. BI will be reintroduced as part of full ASTB revision by FY09) Compromise damage remedied

19. Parallel Form Development Phases III: 30 Day Transition JUN 04-JUL 04 New versions produced: Test Books (Forms 3, 4, & 5)- new Forms measure same abilities as Forms 1 & 2, but content was reorganized Test Answer Sheets Examiner’s Manual APEX Examiner’s Guide Deployed JUN 04 to CNRC & MCRC sites Deploying JUL 04 to NROTC sites Deployed on APEX platform in JUL 04

20. ASTB Administration Policy Changes After release of new parallel forms there are two major ASTB policy changes: No more field scoring of paper tests and no scoring manuals issued, all scoring by NOMI Immediate official scores available through APEX Lifetime limit of 3 administrations (Forms 3, 4, & 5) Prior to 1997 2 time limit existed Previous administrations of Forms 1 & 2 do not count toward this limit Intervals between tests are 30 days after 1st exam and 90 days after 2nd; 30 days required between most recent Form 1 or 2 and any current form (3, 4, or 5)

21. Retest Analyses Summary Taking the ASTB multiple times do not generally improve scores, True for OAR examinees as well People who take the ASTB multiple times before passing are more likely to attrite from flight training Even people who improve on retest are attrite risks: AQR of 7 for a 5th time examinee does not imply the same aptitude that a 1st time AQR of 7 would Conclusion - A 3-test limit will: Maintain quality of accession pool Limit test and item exposure Still provides a chance to improve

22. Automated Pilot Examination (APEX) APEX is a computer-administered ASTB, with content identical to paper-and-pencil ASTB Original version developed by NAMRL Aviation Selection Division 1995-2001 Original version used at 4 sites in CONUS for official ASTB administrations from 1998-2002 2002 CNET and CNRC agreed to sponsor development of APEX 1.0 for fleet-wide deployment

23. APEX Deployment Deployable to all ~150 permanent ASTB test sites, including NRDs, OSOs, MCAFs, USCG, & Military Institutes Test administrators issued user ID, user password, & workstation ID remotely by NOMI APEX also available on a temporary basis to non-permanent sites like deployed ships and remote units APEX 1.0 successfully used OCONUS at Sigonella & shipboard in Jun03

24. APEX Computer Requirements (Unchanged for Version 1.1)

25. All FY05 ASTB O&M funds rec’d; out-year funding programmed into NOMI budget as a line item for the duration of the ASTB effort Programming improved APEX delivery system including capacity for computer adaptive testing and inclusion of PBMs and Personality Assessment On-going Test Item Bank Development utilizing NOMI staff, contactors, and NASC stashes Establishing NOMI Test Facility (OCS/API) space alteration complete test equipment purchased test facility start-up this fall ASTB Revision PlanProgress Report

26. ASTB Revision PlanProgress Report (cont.) Hosted new parallel ASTB forms on APEX for on-line delivery Near completion APEX CAT capability to Tailor ASTB Item Delivery Received two TBAS workstations from USAF for evaluation and testing Retrofitting NAMRL ASCAT and USAF TBAS source code for APEX platform integration Exploring potential personality assessment vehicles appropriate for aviation selection

27. Legal Issues Legal Requirements Title VII Civil Rights Act (1964, 1991) Uniform Guidelines (UG) on Employee Selection (USC 41.60-3, 1978) Professional Standards Standards for Educational and Psychological Tests (APA, 1999) Principles for the Validation and Use of Personnel Selection Procedures (SIOP, 1987) Ethical Principles of Psychologists and Code of Conduct (2002)

28. ASTB Job Analysis (Conducted by NAVAIR-TSD) Steps Gather info from SMEs and existing job analyses for all platforms and programs Develop draft JAQ consisting of task/KSA inventory to capture job information across pipelines Revise JAQ and administer to incumbents/SMEs Analyze JAQ responses & produce documentation Payoff Helps determine which PBMs should be used, and will maximize predictive validity Provides basis for developing new criterion measures for use internally or by CNATRA Identify similarities & differences across pipelines Will tell indicate where a personality measure may help Job Analysis may provide incite to future pipeline selection criterionJob Analysis may provide incite to future pipeline selection criterion

29. PBM Candidates to Be Evaluated USAF’s Test of Basic Aviation Skills (TBAS)   IHMC MMS (formerly NAMRL’s ASCAT) CogScreen (LCAC & UAV Selection tool) PMT and other select PBM tasks Since there is only one spatial abilities task, we would like to investigate some other Psychomotor testsSince there is only one spatial abilities task, we would like to investigate some other Psychomotor tests

30. Performance Based Measures PMT and CogScreen overlap on critical measures (multi-tasking and spatial skills). However, it should be noted that the PMT tasks tend to be more tasking. Multiple Measures System (MMS) - Institute for Human & Machine CognitionPMT and CogScreen overlap on critical measures (multi-tasking and spatial skills). However, it should be noted that the PMT tasks tend to be more tasking. Multiple Measures System (MMS) - Institute for Human & Machine Cognition

31. PBM Assessment Timeline Explore existing PBMs FY04 4thQTR save time and money Job Analysis (JA) concluded FY04 4thQTR Complete PBM studies FY05 2ndQTR Select PBMs for inclusion using JA and studies FY05 3rdQTR Program selected PBMs for on-line delivery FY05 4thQTR Conduct Longitudinal Study – follow students from selection through winging Existing tests good – Limiting factor was PCs and internetExisting tests good – Limiting factor was PCs and internet

32. Conclusion on PBMs Untapped in Naval Aviation selection Focuses on multi-tasking abilities May increase selection effectiveness Could provide pipeline selection Provide baseline neurological assessment

33. Tri-Service Cooperation Met with USAF Counterparts in March Agreed to coordinate efforts and share/leverage resources; Developing MOA Met with USA Counterparts in June Agreed to coordinate efforts & share/leverage resources; Developing MOA The Services Continue to Coordinate Efforts AND Leverage Efforts/Resources