General Information --- What is the purpose of the test?

1. Evaluating Tests --- Some Questions to Ask • General Information --- • What is the purpose of the test? • For what population is the designed? Is this population relevant to the people who will take your test? • What special qualifications are needed (if any) to acquire the test? • Test Content and Scoring --- • How many questions exist on the test? What is the format of the test questions? Is the format and number of questions sufficient for your needs? Is the difficulty level and nature of the questions appropriate for your sample? • What scores are yielded by the test? • How are test results reported? • What materials are available to aid in interpreting test results (e.g., narrative summaries, graphs, charts)?

2. Evaluating Tests --- Some Questions to Ask (cont.) • Test Administration --- • How is the test administered? • Are the test instructions clear? • Are the scoring procedures clear and appropriate for your use? • What time constraints (if any) exist in the administration of the test? • Are the test materials (e.g., test booklet, manual, appropriateness for examinees) sufficient for your purposes? • What special training is needed (if any) to administer the test?

3. Evaluating Tests --- Some Questions to Ask (cont.) • Norm/Validation Sample --- • What is the composition of the sample(s) used in validating and establishing norms for the test (e.g., race, age, gender, SES)? Is the composition of the sample relevant to your test takers? • What data is available for the normative sample (e.g., percentiles, standard scores)? • How many people were in the validation and normative sample(s)? Is the sample size large enough to yield accurate data and interpretations (e.g., standard error of measurement?

4. Evaluating Tests --- Some Questions to Ask (cont.) • Test Reliability --- • What is the reliability of the test and any subscales? What reliability techniqueswere used? Were these techniques appropriately applied? • Do reliability estimates exist (and are different) for any sub-groups? • Is the reliability of the test high enough for the purpose in which the test is going to be used being used?

5. Evaluating Tests --- Some Questions to Ask (cont.) • Test Validity: Content-Related Validity --- • What is the domain (e.g., skills, ability, traits) that is being assessed by the test? How was the test content collected and analyzed? What data was used to determine the appropriateness of the test content? • What was the composition of SMEs (if any) used in the selection of test content? How was the data from theses experts collected and analyzed? • Is the content of the test related to the qualifications needed by sample to which the test is being given?

6. Evaluating Tests --- Some Questions to Ask (cont.) • Test Validity: Criterion-Related Validity --- • What criterion measure (e.g., performance measure) was used to assess the concurrent or predictive validity of the test? Why was the criterion chosen? • What was the size and nature of the sample used in the validation of the test? Does restriction of range exist in the criterion measure (s)? • What is the criterion-related validity of the test (e.g., the correlation between test and criterion scores? Is the level of this correlation sufficient for your purposes? • Test Validity:Construct Validity --- • Is the conceptual framework for the test stated clearly? Is the construct, as defined and measured, relevant for your purposes?

7. Evaluating Tests --- Some Questions to Ask (cont.) Test/Item Bias --- 1. Were the items analyzed statistically for possible sub-group bias? What method(s) was used? How were items selected for inclusion in the final version of the test? 2. Was the test analyzed for differential validity across groups? How was this analysis conducted?