A Case-Study of a Web-Based Method for Repeated-Measures and Multi-Source Research

1. A Case-Study of a Web-Based Method for Repeated-Measures and Multi-Source Research Michael J. Walk, M.S. University of Baltimore michael.walk@ubalt.edu SCiP—Chicago, IL, Nov. 2008

2. Web-Based Research • Valuable research tool for Psychology • Dominated by cross-sectional, between-subject designs

3. Speaking Abstractly • WebRTS: • Web-Based Research Task System • A web-based system for poly-task online research designs. • Adhered to methodological and ethical recommendations for online research. • Tested WebRTS in a multi-source research design (N = 28).

4. The Task Page – Purpose • To control the ordering of stimulus presentation • To prevent repeating or premature completion of tasks (Reips, 2000) • To separate the research design into short, distinct tasks (Reips, 2000) • To allow participants to return to complete at a later time

5. The Task Page - Function • Find out who the user is • Find out what tasks are done so far • Put a check mark by those tasks • Find out what the next task is • Make this task an active hyperlink • Find out what tasks are in queue • Make these tasks inactive text

8. WebRTS – Basic Page Structure • Header • Same on every page = coherence(Nosek, Banaji, & Greenwald, 2002) • Logout Button (if logged in) • University Logo = trustworthiness(Reips, 2000) • Content • Footer • Dynamic (page links to important pages) = navigability • Email the researcher link = experimenter presence • Quit the study link = debriefing(Nosek, Banaji, & Greenwald, 2002)

10. Internet Website Flow Current Participant Homepage Login Page Debriefing Quit? New Participant Yes Give consent Task Page No Informed Consent Last Task? Refuse consent Task 1 Task 2 Task k Pre-Consent Exit Interview Processing script Task update script

11. WebRTS – Auxiliary Pages • Forgotten password retrieval • Administrator’s page • Send reminder emails to participants • Add fields to database

12. WebRTS – Technology Specs • MySQL database (hosting and database provided FREE by www.agilityhoster.com) • PHP server-side scripts • JavaScript for form validation • Pilot version • www.ubpsychportal.org/ssa • Updated version (WebRTS 2.0) • www.ubcareerlab.org/cip

13. Case Study • Used in a multi-source, poly-task research design testing relationships between self-monitoring (Gangestad & Snyder, 2000) and accuracy of predicting personality ratings (e.g., Walk, Mitchell, & Yun, 2008) • Anecdotal usability evidence all positive • Brief follow-up survey administered online in Aug. 2008 (3 months after Case Study) • N = 6 (21%) • Green & Pearson’s (2006) Web-Site Usability Instrument • 16 items, (1) strongly disagree to (7) strongly agree • 4 Open-ended questions

14. Notable Usability Results • I completed the task on the Web site without much effort. (M = 6.33, SD = .82) • After learning to use part of the Web site, I easily learned to use another part. (M = 6.17, SD = 1.17) • The Web site interface was consistent throughout the site. (M = 6.17, SD = 1.17) • All items M > 5

15. Item means

16. Open-Ended Questions • What part or page of the website did you like the best? • I liked it all. It was user friendly. • I liked the fact that the survey was multiple choice. This meant that it took less time to complete the tasks. • I liked the fact that the website was easy to navigate when completing the tasks. • Easy to readfollow • The set-up of the whole thing was easy to use. • i dont know

17. Open-Ended Responses (cont.) • What part or page of the website did you like the least? • No complaints. • Unfortunately, I did not like the part of the survey that had someone else fill out a survey… • The part of the website that I did not like was the portion in which you had to have a supervisor or co-worker complete… • lacks color, kind of boring • NA • I dont know

18. WebRTS 2.0 • System pages are “easily” configurable • Titles, pictures, colors, researcher names, host institution, etc. • Can set open / close dates • Can set max participants • Can randomly assign to ordering conditions • Prevents repeat submissions by the same user.

19. WebRTS – Uses and Applications • Repeated measures designs • Longitudinal designs • Poly-task cross-sectional designs • Multi-source (e.g., participant & rater) designs • Diary designs

20. References Gangestad, S. W., & Snyder, M. (2000). Self-monitoring: Appraisal and reappraisal. Psychological Bulletin, 126(4), 530-555. Green, D., & Pearson, J. M. (2006). Development of a web site usability instrument based on ISO 9241-11. Journal of Computer Information Systems, 66-72. Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). E-research: Ethics, security, design, and control in psychological research on the internet. Journal of Social Issues, 58(1), 161-176. Reips, U. (2000). The web experiment method: Advantages, disadvantages, and solutions. In Birnbaum, M. H., ed. Psychological Experiments on the Internet. Academic Press: San Diego, CA. Walk, M. J., Mitchell, T., & Yun, G. (2008). Know thy social self? Self-monitoring predicts accuracy in rating one’s reputation. Poster session presented at the 20th Annual Meeting of the Association for Psychological Science, Chicago, IL, May 2008. Questions?