Does Use of Learning Management Systems Aid in Student Learning?

1. Does Use of Learning Management Systems Aid in Student Learning? Peter Malo, PhD, JD, MBA, MPM Owner and Senior Consultant BluDesygn-Consulting®, LLC

2. Does Use of Learning Management Systems Aid in Student Learning? • At 2-year College granting Associate degrees, all prospective nursing students must complete several core requirements prior to application to the nursing program. • Two semesters of anatomy & physiology, both a lecture and laboratory course is one such example. • Typically, a more traditional classroom course, anticipation of future growth in this program prompted examination of a possible hybrid classroom, that ultimately, would become an online, Web-based course for this material.

3. Does Use of Learning Management Systems Aid in Student Learning? • As a first phase feasibility study for the hybrid format, two first-term sections were selected to compare the effectiveness of a course material platform, Blackboard®. • The goal for many of today’s teaching institutions in using Blackboard® “is to establish a Network Learning Environment™ (NLE), in which any student or teacher can: • view instructional content, collaborate with educators, • evaluate academic performance and • access any learning resource at any time in order to achieve their educational objectives.”†

4. Does Use of Learning Management Systems Aid in Student Learning? • A spring and fall section of Human Anatomy and Physiology I were selected to compare the effectiveness of Blackboard® software as a course material platform. • For students, each section had: • Similar number of students • Similar distribution of male vs. female students • Similar diverse backgrounds. • For material, each course had: • The same instructor, a veteran of over 12 years of teaching. • The same textbook, covering the same course objectives over a similar time period; similar type of handouts, supplemental material and lecture notes were utilized for both sections. • The same number of quizzes, exams and laboratory practicals, spaced in a similar temporal pattern across the entire course, to ultimately give the same total number of points a student could possible amass as his/her grade. • The only major difference between the two sections: • The Fall section utilized Blackboard®, • The Spring section did not.

5. Does Use of Learning Management Systems Aid in Student Learning? • The completed Spring term (without Blackboard®) had amassed fewer overall mean total points as a section (446.6) when compared to the completed Fall term (with Blackboard®)(543.0) that was statistically significant. • The most obvious trend was observed in exam performance. Both sections had to complete three Scantron® based exams, each exam having the same format, material, question number, and point score. • For each exam, the Fall section using Blackboard® performed consistently better than the Spring term without Blackboard®, by comparing the mean total point score obtained on corresponding exams for each section.

6. Does Use of Learning Management Systems Aid in Student Learning? For each exam, the Fall section using Blackboard® had statistically significant (p<0.01) higher mean total point scores per exam than the Spring section without Blackboard®.

7. Data Analysis Between Sections • Blackboard® access also had some effect on the mean points obtained per laboratory practical. • The Fall section utilizing Blackboard® did have a statistically significant higher mean point score on the first laboratory practical as compared to the Spring section without Blackboard®, as well as displaying a higher (albeit nonsignificant (p = 0.08)) trend in scoring for the second practical as compared to the Spring non-Blackboard® section.

8. Data Analysis Between Sections • However, unlike both the exam and lab practical profile, Blackboard® access had no effect on the mean points obtained per quiz. • In fact, the Spring class without Blackboard® achieved statistically significant higher mean points when compared to the Fall section utilizing Blackboard®.

9. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • With an apparent positive academic effect by Blackboard® use, the Fall section was analyzed (an intra-sectional analysis) for any usage patterns of material placed on the Blackboard® platform, as measured by the number of hits and academic performance. • Students were grouped into three segments based on course final grade: an Above Average Group (A/B group), an Average Group (C group), and a Below Average group (D/F group), and compared to their Blackboard® usage by either • the total number of hits on their Blackboard® account, or • by the “Course Content” hits as observed on a daily basis throughout the 16 week course. Temporal usage (daily) of Blackboard®’s “Course Content” was chosen to relate Blackboard access to key points of academic assessment throughout the course. Blackboard’s Course Content contains key course documents (i.e. PowerPoint presentations, handouts, and supplemental material). • Students were also given a detailed survey to complete regarding their opinions on a number of different aspects of Blackboard® usage in their section.

10. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • Blackboard® usage, as measured by total number of hits, showed a distinct trend in usage related to academic performance. • Clearly, the Above Average Group utilized Blackboard® more than either the Average or Below Average Groups.

11. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • The Above Average (A/B) Group had consistently high hits in Course Content, especially clustered around testing dates.

12. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • The Average (C) Group had lower hits in Course Content throughout the course when compared to the Above Average Group, however increased hits were observed clustered around testing dates, a pattern also observed with the Above Average Group.

13. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • The Below Average (D/F) Group had lower hits in Course Content throughout the course when compared to the Above Average Group, exhibiting a temporal pattern similar to the Average (C) Group, in both number of hits and daily usage.

14. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • Summary of Survey Analysis: questions that appeared on the student survey are summarized below. Student answers marked “Agree” or “Strongly Agree” were expressed as percents of the total choices (Strongly Agree, Agree, Disagree, Strongly Disagree) for students grouped into two categories: either Above Average (Green) or Average/Below Average (Blue), and follow the appropriate question. • This was the first time I used Blackboard®. 59% v 35% • I found the Quick Help Guide distributed with the syllabus to be helpful in using Blackboard®. 92% v 92% • I feel a short course in the use of Blackboard® would be helpful. 56% v 61% • I used Blackboard® at least once a week. 88% v 100% • The more I used Blackboard®, the easier I found it to use. 100% v 100% • I feel using Blackboard® has helped my computer skills. 72% v 84%

15. An Intra-Sectional Analysis of Sections Utilizing Blackboard® • I thought that having Blackboard available with this course was a good idea. 88% v 100% • I feel better prepared to use Blackboard® in other courses. 100% v 100% • I feel that using Blackboard® has helped me prepare for taking a course that is on-line or distance learning based (using the Internet).83% v 75%

16. Conclusions • Students consistently scored higher in most assessment parameters when an anatomy & physiology section was based on a Network Learning Environment™ such as Blackboard®, as compared to students who were enrolled in the traditional classroom-based pedigogy. (Slides 6,7 and 8.) • For students using Blackboard® to study anatomy and physiology, an analytical breakdown of student groups ranked by achievement and their corresponding utilization of Blackboard®, revealed the Above Average Group (A/B grades) had higher number of total hits (indicating usage of some kind) when compared to the Average (C grade) or Below Average (D/F grade). (Slide 10.) • In addition, examination of Course Content usage showed the Above Average Group trended higher on daily usage of this feature, especially clustered around academic performance assessments, than either the Average or Below Average Groups. (Slides 11, 12, 13.) • Finally, students own attitudes, as measured by survey questions, showed an overall positive experience across all achievement groups regarding the use of Blackboard® as a platform for teaching anatomy & physiology and as a preparatory tool for an on-line, Web based course. (Slides 14 and 15.)