A Cloud-based S elf-assessment Tool for Measuring Information Literacy

1. A Cloud-based Self-assessment Tool for Measuring InformationLiteracy Wesley Leonard Senior Application Programmer Health Technology Group @ Central Michigan University

2. Introduction An online self-assessment tool, Ready:Set, was developed to measure information literacy and provide immediate feedback to the user. This tool made heavy use of visual analog scales or, as they are commonly known, “sliders.” Adaptive testing was added to the latest version of the assessment. This allowed the difficulty of questions to fluctuate based upon the user’s perceived (or actual) skill level. Ready:Set was deployed on the Amazon Elastic Computing Cloud. Since its original incarnation in 2003, the assessment has been completed by students from Central Michigan University and partner institutions around the world.

3. About me • Studied computer engineering and computer science at Western Michigan University • Participated in the Computer Club @ WMU and the Kalamazoo Linux Users Group • Worked for a failed dotcom company, 2000 – 2001 • Worked as a freelance programmer • Started as a student at Central Michigan University in 2003 • Worked as a grad assistant and was hired full-time in the new college of health professions • Was introduced to Dr. Lana Ivanitskaya and “RRSA” • Completed M.S. in computer science in 2008

4. History of Ready:Set • Development started in 2003 • Original idea from 1998 • Perl, MySQL, IBM AIX server • Developed by Lana V. Ivanitskaya, Ph.D. and Anne Marie Casey, A.M.L.S. with assistance from student programmers • Original version called “RRSA” • Research Readiness Self Assessment • Questions selected for individual subject areas and designed to accurately gauge users abilities • Original topic was students’ ability to utilize library resources and effectively find information

5. Subject Areas • Major topics: Library, Health • Information Literacy • Library resources • Search engines • Peer-reviewed journals versus less trustworthy content • Plagiarism • Evaluating quality of web sites • Online information (commercial sites disguised as research) • Online pharmacies • Dietary supplements

6. Major Versions • 30 different versions have been created for various groups and institutions from around the world (England, China, Brazil, Taiwan, Texas, Calgary… to name a few) • Questions, instructions, and feedback translated (as necessary) by volunteers • Questions customized for each institution

7. Feedback system • After completing the assessment, the user receives feedback based upon their results • User may also opt to receive an email with additional resources to help them improve their skills • User may email results to themselves at any time after completing assessment • User may not retake assessment using same PIN

8. Visual analog scales • Visual Analog Scales (VAS) or “Sliders” • Replaced discrete questions • 400 distinct points • Customizable

9. Slider Configurations • With/without feedback • With/without labels (midpoint, endpoints) • Custom widths • Reverse scale (less is better) • Custom background images and custom granularity

10. Slider Results Distribution of decimal values

11. Users like Whole Numbers

12. Minor (cool) features • PIN – Password used to access users’ version of the assessment • Self-enrollment – User is given a “group code” which can be used to generate a unique PIN • Fake web sites – Dietary supplement, online pharmacy, and several other fake web sites were created for users to evaluate while taking the assessment • Reporting – Completion, placement, performance reports were created so instructors from each institution can see what their students are doing • Export – All user information can be exported to CSV format for analysis

13. Adaptive testing • Original RRSA was long (30 – 40 minutes) • Adaptive testing was added to shorten test time • Currently, using simple “up or down” algorithm • Question “difficulties” determined by how often they were answered incorrectly in the past • A set of “shadow questions” determines initial starting difficulty • In our case, shadow questions are actually questions about the user’s perceived skills (the hypothesis being that perceived skills and actual skills are inversely-related)

14. Adaptive Example

15. outcomes • Subjects showed significant improvement on all measures from pre to post when time lag is great • Short-term effect: strong change in self-perceived skills and beliefs • Research with or based on RRSA has generated 20 publications • This work has been cited 21 times • (See http://rrsa.cmich.edu/twiki/bin/view/RRSA/Publications for a complete list) • Collaboration with groups of students and faculty from around the world – a truly global perspective on information literacy in the digital age • Additional research topics: “Rogue online pharmacies” • Revenue generation

16. Part 2: The Cloud How we brought Ready:Set to the Amazon Elastic Computing Cloud

17. How the Amazon EC2 Works • Sign up for account • Launch instance • Preconfigured or create your own • Connect to instance • Create security keys, connect as root/administrator with key • Install and configure software as needed • Execute applications • Terminate instance when completed

18. Our configuration • Small Linux instance (EC2) • Small storage space (Volume) • Simple email service • Custom script to install necessary software and bring up Ready:Set server

19. AWS Console

20. AWS Console

21. AWS Console

22. AWS Console

23. Redesign • In late 2009, the RRSA was to be redesigned and rebranded • Python framework • PostgreSQL database • Re-implement existing features but designed for the future • Decided to bring it to the cloud…

24. django • A high-level Python Web framework that encourages rapid development and clean, pragmatic design • Follows MVC paradigm • Designed to handle two challenges: the intensive deadlines of a newsroom and the stringent requirements of the experienced Web developers who wrote it. • Lets you build high-performing, elegant Web applications quickly • Focuses on automating as much as possible and adhering to the DRY principle • http://www.djangoproject.com

25. Startup Script echo '(hit ENTER to continue)...' read ln -s /app/postgresql /var/lib/postgresql echo '(hit ENTER to continue)...' read /etc/init.d/postgresql-8.3 start ### Start the server in the background: python rrsa/manage.py runserver 0.0.0.0:80 2>/dev/null 1>/dev/null & echo 'DONE' # Script to start up RRSA server on Amazon AWS ### Install necessary packages: apt-get install postgresql-8.3 pg_createcluster8.3 main --start apt-get install python-pygresql apt-get install python-django apt-get install python-psycopg2 ### Point Postgres to static volume: echo 'Now changing postgres directory... (hit ENTER to continue)' /etc/init.d/postgresql-8.3 stop read mv /var/lib/postgresql /var/lib/postgresql.BACKUP

26. Advantages of the cloud • Scale quickly • High security • Failover • Reliability • Can dedicate resources to certain institutions if necessary • No need to purchase or maintain infrastructure • Cost is comparable to utilizing CMU’s infrastructure • Only pay for resources used

27. Amazon pricing

28. Other cloud providers • Rackspace.com • Google • Vmware • Microsoft • + many, many more!

29. Acknowledgements • Lana Ivanitskaya, Ph.D. – CMU, DHA Program • Ann Marie Casey, A.M.L.S. – Formerly with CMU’s Park Library • Programmers: SherzodRuzmetov, Ryan Laus, Xinxin Wu, Jeff Pipas, Weber Schulz • Lawrence Fulton, Ph.D. – Texas State University, Computer Information Systems • Paul Albee, Ph.D. – CMU, Computer Science department

30. Additional info • For more information, see our wiki:http://rrsa.cmich.edu/twiki/bin/view/RRSA • We are always seeking research partners, so contact us to try out Ready:Set • Currently beta testing both cloud and adaptive versions