1 / 22

Creating Valid and Effective Measures

This presentation explores the importance of measures in translating theory into data for modeling, and discusses the criteria for creating good scales, including reliability and validity. The optifact Stata software component is introduced as a tool for analyzing candidate items and creating summated rating scales. The results of using optifact to create new behavior and parenting scales are presented, highlighting the potential implications and cost reduction.

cporter
Download Presentation

Creating Valid and Effective Measures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July 11-12, 2005

  2. Are measures important? • Measures are how theory is translated into data for modeling • Unlike art, where we want to engage individual interpretation as a major component of the work • Measures need to be unambiguously interpreted in precisely the same way by all

  3. Why Scales? • Combining the results of several questions is more reliable and precise, and reduces measurement error (Spector, 1992)

  4. What makes a good scale? • It is reliable • The same measurement, produces similar results across time and persons • The questions or items that make up the scale are consistent with one another • It is valid • It measures the concept that it is intended to measure • It measures one concept • It is regularly distributed

  5. Reliability • Internal Reliability • Internal consistency: the items should inter-correlate (Cronbach’s Alpha) • Test-Retest Reliability • Same measure taken again should produce consistent results

  6. Validity • Rationale • The items must be consistent with theory and the concept under consideration • Criterion Validity • The items should vary in the same way to other variables that are theoretically related to the concept • Canonical correlation

  7. Uni-dimensional, Efficient • Scales that measure a single concept are more useful in establishing causal relationships • Efficient scales use only enough items to reliably measure a concept • Regular distributions allow standard modeling techniques

  8. Research Question • If my scales are not uni-dimensional, are there sub-scales which are and meet the criteria associated with good scales? • Investigated using data gathered for the Canadian National Longitudinal Survey of Children and Youth by Statistics Canada.

  9. Data and Method • The NLSCY cycles 1, 2 and 3 • Measures on parenting and child behaviour taken every two years (allows the investigations of test-retest reliability) N > 13,000. • Common factor analysis • Dimensions, alpha • Canonical correlation • Criterion validity

  10. Problems • There are a lot of candidate scales • 2k – 1, if scales of all sizes are considered • The testing procedure is lengthy • Factor analysis (dimensionality) • Alpha (internal reliability) • Canonical Correlation (external validity) • Re-test (all of the above, twice) • More than 13 scales

  11. Solution: optifact • A Stata software component to analyze a list of candidate items for the creation of a summated rating scale

  12. -optifact- Specification optifact varlist [weight] [if exp] [in range] [, top(#) smallest(#) largest(#) maxfact(#) minslope(#)] Varlist: the list of candidate items (numeric) Options: Top: the number of candidate scales to list (10) Smallest: the smallest k allowed (3) Largest: the smallest k allowed (all items) Maxfact: the maximum number of factors allowed (1)

  13. -optifact- Output optifact abecq6b abecq6n abecq6w abecq6p abecq6i abecq6qq abecq6hh abecq6s [aweight=awtcw01c], top(20) criteria(childsex ammcq01) 8 items will be processed There are 219 potential scales 56 combinations of 3 items had one factor, 0 had more than one factor Top 5 scales using 3 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 3 0.774 0.201 Yes abecq6p abecq6i abecq6qq 2 3 0.769 0.203 Yes abecq6p abecq6i abecq6hh 3 3 0.744 0.231 Yes abecq6b abecq6n abecq6hh 4 3 0.742 0.169 Yes abecq6p abecq6qq abecq6hh 5 3 0.734 0.213 Yes abecq6n abecq6p abecq6i 55 combinations of 4 items had one factor, 15 had more than one factor Top 5 scales using 4 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 4 0.802 0.185 Yes abecq6p abecq6i abecq6qq abecq6hh 2 4 0.784 0.199 Yes abecq6n abecq6p abecq6i abecq6hh 3 4 0.776 0.191 Yes abecq6n abecq6p abecq6i abecq6qq 4 4 0.775 0.198 Yes abecq6b abecq6p abecq6i abecq6hh 5 4 0.768 0.172 Yes abecq6p abecq6i abecq6qq abecq6s

  14. Behaviour Scales

  15. Parenting Scales

  16. Results • Of these 13 scales only one was uni-dimensional • Most had larger number of items than desirable for the value of alpha

  17. New Behaviour Scales

  18. New Parenting Scales

  19. Implications • These scales are part of a large survey conducted every 2 years in Canada on more than 20,000 children • The survey has completed 6 cycles of data collection • Each question has been asked more than 120,000 times

  20. Implications (Cont’d) • The scales required 85 questions • The revised scales require 30 questions • More than 55,000,000 questions

  21. Caveats • Most scales in the NLSCY are not the best that can be made from the available data • Some of these scales should not be used • The NLSCY is an excellent survey, conducted by competent statisticians • Similar results might be expected in other major surveys

  22. Conclusions • -optifact- can help find better measures • Uni-dimensional • Valid • Reliable • -optifact- can reduce costs • Equivalent or better measure for same money

More Related