Data Collection, Publishing, and Dissemination of Results

Data Collection, Publishing, and Dissemination of Results Eric Grodsky, University of Minnesota Chandra Muller, University of Texas at Austin

Motivation & Overview • The impact of the STEP program and your STEP Type 1 project depend on dissemination of knowledge for application to other settings. • Based on our early experiences working with STEP Type 1 projects • Steps to designing a study, collecting data, publishing and disseminating results: an overview and then example

Study Design • A good study design makes it easier to publish and disseminate results • Begin the design phase early, alongside the program design • Consider collecting baseline (and control group) data before beginning program implementation

Control or Comparison Group • What is the impact of your program on production of STEM majors? To answer this you must be able to estimate: • What would have happened without the program? • And ideally, what the program did to have the effect, and for whom? • Consider collecting a control group sample while you are planning the program, before it is implemented • Target students for comparison who are similar to those you will target for the program

IRB: Institutional Review Board • Work with the IRB at your university to gain approval for your study • It is likely that your study will be considered “Exempt” • Example from University of Texas:

Data Collection & Analysis • Study design and response rates are crucial • A randomized control trial—where the only difference between your control and treatment groups is due to random assignment—produces the simplest analysis strategy to show program effects

Dissemination of Findings • Begin analyzing your data and publishing early—pay a writer to write up your results if you and your staff lack time • Early phase results might include: • Description of challenges faced by control group • Early evidence of climate change in college • As project matures (even at the end of second year phase) results might: • Compare control and treatment group progress in STEM courses, attitudes about STEM, other indicators relevant to your project? • Estimate effects on longer range goals of project and the possible mechanisms through which program works

Where do you publish and disseminate your findings? • Publications of field specific education journals (the education arm of professional organizations, have journals and newsletters, e.g., Journal of Engineering Education) and education journals like Journal of Higher Education, Education and Evaluation Policy Analysis • Early findings and news briefs might go into professional newsletters • Work with your office of public affairs to prepare press releases • Consider an end-of-project book • Get ideas from other STEP projects

Designing the study • What do you want to learn? How/why do you believe that the project succeed? • Who will you serve? What is your target population? • How do you recruit sample members and collect data?

Designing the study • What exactly do you want to ask? • What will you do with all of those data once you have them?

What do you want to learn? • Project will implement intervention you and NSF believe will work • What are the explicit objectives? • What are the interventions? • How/why will they work? (theory)

What do you want to learn? • Example: Professor Yu ? Study groups Chemistry degree

theory Knowledge Study groups Chemistry degree Enjoyment Self-confidence

The target population • Potential chemistry and physics students • Latino and African American students 1. Recruitment: High school students 2. Tutoring: In selected classes 3. Study groups: In selected classes 4. Faculty mentor: in selected classes

Who is in the control condition? • Recruitment: • Those not recruited? • In comparable non-participating schools? • Comparable non-participating classes? • Any way you cut it, must include kids in high school

The target population • Latino and African American students 1. Recruitment: At selected schools 2. Tutoring: In selected classes 3. Study groups: In selected classes 4. Faculty mentor: in selected classes

Peer tutoring • Target population: Students enrolled in gateway courses • Control group?

Study groups • Target population: those enrolling in gateway courses in chemistry and physics

Study groups • Control: • Entrants to same classes prior to project • Entrants to gateway courses in biology and computer science • Random assignment of sections to study groups or not

Designing data collection • Mode of collection • Sampling procedures • Protocol for recruitment • Timing • The IRB

Mode of collection • Survey • Interview (phone or in person) • Paper and pencil • Mail • In class • Online • Focus group

Sampling • Identify target population • Locate/create sampling frame • Make sure you have contact data you need • Structure sample • Stratification • Matching • Blocking

Sampling: The Yu study • Target population: Students enrolled in calculus 101, chemistry 101, biology 101 • Each course has 2-4 sections and each section 6-8 lab sections

Sampling: The Yu study • Randomly assign lab sections to have study groups or not starting in spring of 2013 • Sample size of 600 • 200 students in fall of 2012 from all students enrolled in target courses • 100 treatment and 200 control students in spring of 2013

Sampling: The Yu study • Threats to valid inference

Recruiting the sample • Sample nonresponse undercuts validity • Shoot for at least 70% • Getting and maintaining sample is often the hardest part of the study • Without a good sample the rest of your evaluation efforts are wasted

Recruiting the sample • Incentives for participation • There is actual research on this • Cash is king, other incentives (Amazon gift certificate, MP3 downloads, etc.) not very effective • Some evidence for effectiveness of lotteries, but weak

Recruiting the sample • Initial contact • There is actual research on this • Letter on letterhead in stamped envelope if possible • In Yu’s case, possibly hand-delivered in lab

Recruiting the sample • Non-response follow-up • Email • Snail mail • Phone

Data collection timing • Consider last date you can collect data and work backwards • One week prior to exams • Snail mail incentive letter • 2 email follow ups • Snail mail follow up • One more email • After another week, phone calls

Data collection timing • And of course the IRB…

Instrument design • Goes back to what you want to know • Mechanisms by which intervention has effect • Variation in effects of intervention and what might account for that variation

Instrument design • Do not make up your own items if you can avoid it • Consider extant instruments • Ask simple questions respondents can answer No ‘double barrel’ questions

Instrument design • Survey is a ‘conversation with a purpose’ • Script the conversation so it flows • Use skip patterns to avoid asking irrelevant questions • Keep is as short as possible

theory knowledge Study groups Chemistry degree Enjoyment Self-confidence

Self-confidence: Identity How much do you agree or disagree with the following statements? You see yourself as a science person Others see you as a science person 1=Strongly agree 2=Agree 3=Disagree 4=Strongly disagree

Self-confidence: Self-efficacy You are confident that you can do an excellent job on tests in this course You are certain you can understand the most difficult material presented in the textbook used in this course You are certain you can master the skills being taught in this course

Analyzing Data • Depending on design • RCT: difference in means • Other designs: • Covariate adjustment • Difference in means for certain subgroups

Analyzing Data • Our example: Does program impact self-confidence? • Mean change in self-confidence: • Control students in fall 2012 • Control students in spring 2013 • Treatment students in spring 2013 • Covariate adjustment

List of journals for possible publication of STEP Type 1 results (list generated by audience) • Advances in Engineering Education - http://advances.asee.org/ • CBE Life Sciences Education - http://www.lifescied.org/ • Chemical Educator - http://chemeducator.org/ • Community College Journal of Research and Practice - http://www.tandf.co.uk/journals/authors/ucjcauth.asp • CUR Quarterly - http://www.cur.org/publications/quarterlies.html • Journal of Applications and Practices in Engineering Education - http://japee.net/ • Journal of Chemical Education - http://pubs.acs.org/journal/jceda8 • Journal of College Science Teaching - http://www.nsta.org/college/ • Journal of College Student Retention -http://www.cscsr.org/retention_journal.htm • Journal of Research in Science Teaching - http://www.narst.org/publications/jrst.cfm • Journal of STEM Education: Innovations and Researchhttp://ojs.jstem.org/index.php?journal=JSTEM&page=index • Journal of Women and Minorities in Science and Engineering - http://www.begellhouse.com/journals/00551c876cc2f027 • Physics Teacher - http://tpt.aapt.org/ • Physics Today - http://www.physicstoday.org/ • Portland International Conference on Management of Engineering and Technology (PICMET) - http://www.picmet.org/main/ • STEPcentral.net A Community Forum for NSF STEP Projects - http://stepcentral.net/ • Student Affairs in Higher Education - http://www.naspa.org/

Data Collection, Publishing, and Dissemination of Results