Collaborative Data Management for Longitudinal Studies

1. Collaborative Data Management for Longitudinal Studies Stephen Brehm [coauthors: L. Philip Schumm & Ronald A. Thisted] University of Chicago (Supported by National Institute on Aging Grant P01 AG18911-01A1)

2. Agenda 1. Background on Study 2. Problem – Data Management Deficiencies 3. Solution – Collaborative Data Management 4. STATA Programs – maketest & makedata

3. Background on Study • NIH-funded Longitudinal Study • Loneliness & Health • Thousands of Measures • Loneliness • Depression • 230 subjects • Repeated Yearly

4. Problem – Data Management Deficiencies • Code Not Modular …Difficult to manage the data cleaning code …Limited code reuse from year to year …Difficult to collaborate among interns • No Established Set of Data Cleaning Steps …Difficult for research assistants (turn-over) …Inconsistent data cleaning techniques …Data cleaning code difficult to read

5. Problem – Data Management Deficiencies Research Assistant Research Assistant Research Assistant Core File Set Research Assistant Research Assistant

6. Solution – Collaborative Data Management • Process • Established Steps • File System Layout • Automated Tests • Collaboration • Concepts • Module • Batch • “Data Certification” • STATA Programs • maketest • makedata

7. Solution – Collaborative Data Management • Process • Established Steps • File System Layout • Automated Tests • Collaboration • Concepts • Module Ex:loneliness • Batch • “Data Certification” • STATA Programs • maketest • makedata

8. Solution – Collaborative Data Management • Process • Established Steps • File System Layout • Automated Tests • Collaboration • Concepts • Module Ex:loneliness • Batch Ex:yr1, yr2, yr3 • “Data Certification” • STATA Programs • maketest • makedata

9. Solution – Collaborative Data Management Set of Files for Each Module acquire-[module].do & fix-[module].do test-[module].do derive-[module].do label-[module].do Year-Specific 60% Code Reuse – Files Shared Between Years Acquire & Fix Test Derive Label

10. STATA Program – maketest • Purpose: • Auto-generation of Data Certifying Tests • Functionality: • Tests Variable Type • Checks Consistency of Value Labels • Verifies Existence of Variable

11. STATA Program – maketest • Syntax: • maketest [varlist] using, [REQuire(varlist) append replace] • Example: • maketest using filename.do, replace • Options: • using: specifies file to write • REQ: requires presence of variables in list • append: add to existing test .do file • replace: overwrite existing .do file

12. STATA Program – makedata “Bringing it all together”

13. STATA Program – makedata • Syntax: • makedata [namelist], Pattern(string) [replace clear Noisily Batch(namelist) TESTonly] • Example: • makedata ats, p("acquire-*.do") b(yr1) clear replace • Options: • p: pattern – file naming convention • replace: overwrite existing data file • clear: clear current data in memory • Noisily: full output (default = summary) • b: batch – year, wave, center • TESTonly: only run tests step

14. Other Applications • Beyond Longitudinal Data • Teaching Data Cleaning with STATA • Contact Information • Stephen Brehm: sbrehm@uchicago.edu • L. Philip Schumm: pschumm@uchicago.edu • Ronald A. Thisted: thisted@health.bsd.uchicago.edu • Supported by National Institute on Aging Grant P01 AG18911-01A1