Corinne Boss é Derek Briton Cindy Ives 10 th EES Biennial Conference Helsinki, Finland

1. Using Open Source Survey Tools for Qualitative Inquiries on Educational Development at a Distance Online University Corinne Bossé Derek Briton Cindy Ives 10th EES Biennial Conference Helsinki, Finland October 5, 2012

2. Introduction “The web is more a social creation than a technical one. I designed it for a social effect—to help people work together—and not as a technical toy. The ultimate goal of the Web is to support and improve our weblike existence in the world”. Tim Berners-Lee (2000). Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web, (p.123).

3. Text

5. Quick Facts • Almost 40,000 students (7800 FTEs) • Almost 900 courses in 90 programs • Annual operating budget of $128 million • Student Stats: ~ average age: 29.4 undergrad.; 38.2 grad. ~ 81% work while they study ~ 67% are female ~ 70% are first in family to earn a degree ~ from every Canadian province & more than 90 countries • Faculty & Staff Stats: ~ over 1300 on 4 Alberta campuses ~ academics – 177 full-time; 177 part-time ~ tutors & markers – 382 part-time

6. History and Processes • Unpaced, individual study • Continuous enrollment • Author-editor development model • Industrialized production model • Open University processes adapted • ISD approach > template for courses in a box • Behaviourist, cognitivist pedagogies

7. Institutional Context • Distributed workforce (4 locations, home offices) • Self-paced mode of undergraduate course delivery, continuous enrollment • almost 900 courses; 90+ programs • Institutional goals • Quality and access • Flexible and open • Focused on student success and learning • Online as appropriate

8. Institutional Mandate (2009) • remove barriers to undergraduate and graduate education • provide high-quality, interactive learning environments • actively pursue technological innovations that can enhance teaching, research and administrative functions

9. Conceptualframework • Formative evaluation framework for the qualitative enquiries conducted on 2 evaluation projects • Educational development perspective: • Identify teaching and learning needs • Document & assess an ongoing process of improvements to online course learning activities • Participatory methods and utilization-focused approach (Patton, 2008) Patton, M.Q. (2008). Utilization-Focused Evaluation. (4th ed.). Thousand Oaks, CA: Sage.

10. Uses of Open Survey Tools • Open education is an integral part of Athabasca University’s organizational culture as one of the pioneering online and distance teaching universities. • Strong institutional support for open source tools such as Lime Survey and Moodle, which is the university’s learning management system (LMS). • Technical integration within the institution makes it easier to access and use these open source survey tools as part of the academic practice for both faculty and professionals.

11. Uses of Open Survey Tools • Integrating open source tools to conduct qualitative inquiries on recent educational development initiatives sponsored by AU‘s Centre for Learning Design and Development (CLDD) can be viewed as a strategic alignment towards supporting innovative teaching and learning activities.

12. Uses of Open Survey Tools • Rationale for using Moodle to conduct a needs assessment: • building on AU faculty’s familiarity with the LMS to raise their awareness about the Moodle questionnaire module. • One of the outcomes is to make use of this feature to gather additional qualitative feedback from students to enhance course design and development • Rationale for using Lime Survey to conduct expert review • Providing an opportunity for faculty and professionals to test the tool as well as responding to the object of the qualitative inquiry focused on improving future course learning objects design.

13. Evaluation Project I: Moodle Questionnaire (2010) • The main purpose of the needs assessment was to help determine which educational development activities could contribute to an enhanced teaching and learning experience at Athabasca University

14. Evaluation Project I: Moodle Questionnaire (2010)

15. Evaluation Project II: Lime Survey (2011) • Community Adjustment Fund Project ( part of multimillion dollars funded by provincial government to digitize AU resources) • Formative evaluation of 20 Athabasca University (AU) largest registration courses to assess the usability of the learning objects from both a technical and a pedagogical dimension

16. Evaluation Project II: Lime Survey (2011)

17. Evaluation Project II: Lime Survey (2011) • Purpose of Expert Review : • to obtain feedback from a community of expert practitioners in learning design and educational development that will inform and improve future designs and uses of AU course learning objects (LOs) • to assess the usability of selected learning objects on two design dimensions: technical and pedagogical

18. Evaluation Project II: Lime Survey (2011) • Reviewers’ selection • Learning designers (LDs) and instructional media analysts (IMAs) with at least five to more than ten years’ experience in their field participated in the expert review of showcase-course learning objects

19. Evaluation Project II: Lime Survey (2011) • Questionnaire Design • Drew from several sources based on literature review of LOs • Relevant sections of these validated evaluative instruments were adapted and contextualized to examine both the technical and pedagogical dimensions of the CAF showcase course enhancements • Iterative process: piloted with an AU expert team in learning design and educational development and it was revised a number of times before finalizing the version used in the formative evaluation • Scope: 30-45 minutes to complete an in-depth review of Los & complete the online Lime Survey

20. Evaluation Project II: Lime Survey (2011)

21. Evaluation Project II: Lime Survey (2011) • An overview of the expert review key findings for all the CAF showcase course enhancements: • Technical Dimension • The expert review indicates high levels of agreement about the presentation design with 97% indicating that the overall look and feel of the LOs is effective; 96% finding the LOs easy to use; and around 82% indicating an appropriate level of technical integration of the CAF showcase courses within the learning management system (i.e. Moodle). • The perceived level of technological interactivity tends to be low for the majority (74%) of LOs. • This information has subsequently being used to improve the latest version of all the learning objects that were assessed in the formative evaluation. Biology course enhancements were perceived to have the highest level of technological interactivity.

22. Evaluation Project II: Lime Survey (2011) • Pedagogical Dimension Top 5 Intended Uses • The majority of LOs (57%) are intended to help learner with foundational concepts • More than half (52%)of LOs are intended be used as revision or review of new knowledge, a concept or skill and to provide for multiple learning preferences beyond reading • Slightly less than half (48%) of LOs are intended to help learners develop new knowledge , a concept or skill • About a third (35%) of the LOs are intended to be used as an orienting or tuning-in activity • Slightly less than a third (28%) as an instructor-directed demonstration tool • In general, several components along the pedagogical dimension were rated with overall high levels of agreement in terms of intended use, design interaction, accessibility, reuse, learner interaction, motivation/engagement and effectiveness.

23. Recommendations/Future Directions In sum, it is hoped that the findings of the formative evaluation will be useful ‘to inform an ongoing cycle of reflection and innovation’ (Patton, 2008, p.116). A potential course of action that can be inferred from the expert review results of the course enhancements : • Specifying both the intent and expected outcomes while designing and integrating LOs into courses • Examining the showcase course enhancements from students’ perspective by integrating targeted evaluations on revised technical and pedagogical elements of usability into the various courses; triangulating the perceived technical and pedagogical dimensions with students’ actual use of the learning resources and actual outcomes. • Revisiting the reuse pedagogical components to integrate multiple formats and configurations into which these learning resources can be embedded in online learning environments • Further adapting and revising a taxonomy of technological interactivity to better define and assess how well LOs achieve interactivity

24. Recommendations/Future Directions • Incorporating to greater extent built-in automated feedback into the design of LOs to increase learners’ control (Nielsen, 2003) • Expanding the design feature of accessibility to be more inclusive by taking into account different range of teaching and learning contexts comprised of diverse learners who might experience disability (The Le@rningFederation ,2007) • Using expert reviews as one of the learning design strategies to enhance integration of multimedia learning resources into online course design and development • Tracking the actual use and reuse of these multimedia learning resources by integrating them into open education resources platforms that use analytics • Monitoring and evaluating the usability of the learning resources that are integrated into online courses over time to establish institutional design benchmarks.

26. Conclusions • Patton (2008) argues persuasively that participation and involvement of evaluation’ stakeholders substantially increase the likelihood of using the results of an evaluation in order to advance societal as well as institutional interests. • From an educational development standpoint, a valuable outcome of using open source web-based evaluative tools is: • to engage people both in the process of evaluation itself • to using tools to improve the online learning environment in which they operate

27. Conclusions • At this exploratory stage of the comparative review, it is anticipated that Moodle and Lime Survey will be embedded as part of AU systematic research-based responses to appropriately identify and address educational development needs and challenges.