Evaluating and Assessing Adult Student Learning Outcomes: A Quantitative Methodology

1. Evaluating and Assessing Adult Student Learning Outcomes: A Quantitative Methodology Presented at the International Assembly for Collegiate Business Education (IACBE) ConferencePortland, OROctober 18-19, 2012 Gary F. Keller, Ph.D. College of Business Eastern Oregon University, LaGrande, OR gkeller@eou.edu

2. Introduction • Quantifying the effect that a teacher has on the learning outcomes of their students is a continuous and contentious issue in higher education. • Over the past decade, upgraded professional accreditation requirements, reinvigorated regional accreditation associations, government mandates and increasing competition from for-profit educational institutions has forced changes in the evaluation process. • Unlike scholarship, artistic productions, scientific experiments, and etc. which results in a tangible product the aftermath of teaching is contained in the minds of the learner and may or may not manifest itself for years after the final lecture or test is administered (Hooper & Page, 1986).

3. Purpose of the Study • The issue addressed in this quantitative study was the inability of standard post course evaluations of university level faculty teaching effectiveness to accurately and reliably measure changes in student learning outcomes. • A secondary goal of the research was to study the sluggish feedback faculty members receive (Wetzstein and Broder 1985) from institutional sources that inhibits faculty from utilizing data to make adjustments rapidly.

4. Location of Study: Macro View Milwaukee, Wisconsin

5. Location of the Study: Micro View City of Milwaukee

6. Some Facts About Milwaukee, WI • Milwaukee, Wisconsin, is one of the 25 largest cities in the United States with a population of nearly 600,000 people. • According to the 2010 Census, 44.8% of the population was White (37.0% non-Hispanic white), 40.0% was Black or African American, 0.8% American Indian and Alaska Native, 3.5% Asian, 3.4% from two or more races. 17.3% of Milwaukee's population was of Hispanic, Latino, or Spanish origin (they may be of any race). The metropolitan area was cited as being the most segregated in the U.S. in a Jet Magazine article in 2002.The source of this information was a segregation index developed in the mid 1950s and used since 1964. In 2003, a non-peer reviewed study was conducted by hired researchers at the University of Wisconsin–Milwaukee which claimed that Milwaukee is not "hypersegregated" and instead ranks as the 43rd most integrated city in America • Milwaukee is home to ten Fortune 1000 company headquarters, including such household names as Johnson Controls, Northwestern Mutual, and Harley-Davidson. • Milwaukee experiences all four seasons and rapidly changing weather. The "lake effect" of Lake Michigan makes for cooler summers, warmer winters (than surrounding areas further from the lake), and plenty of snow. http://www.citytowninfo.com/places/wisconsin/milwaukee

7. Literature Search

8. Methodology • In this research project, a pre/post evaluation of learning assessment was developed and tested with a sample of 72 adult students enrolled in an accelerated degree program at a large urban university. • Eight randomly selected courses (7 MBA and one Bachelors). • The type of data collected was quantitative and acquired from a 10 question pre /post survey administered during the first and last night of the subject's course. Respondents were asked to answer questions by selecting a response from a 10 point Likert type scale. • All institutional research protocols followed.

9. Methodology An example of a question and response is provided below. Please rank your reply (circle one number) on a 1 (indicates a low level of satisfaction or skills) to 10 (indicates a very high level of satisfaction or skills). 1. I have some knowledge or experience in the content area of the course. (if so please briefly state your professional or academic experience related to the course) 1 2 3 4 5 6 7 8 9 10 Low High

10. Hypothesis • H1o. There is no statistically significant difference in the learning outcomes of adult students enrolled in the course due to the teaching practices of the faculty member. • H1a. There is a statistically significant difference in the learning outcomes of adult students enrolled in the course due to the teaching practices of the faculty member.

11. Methodology • Data was analyzed using a t-Test: Paired Two Sample for Means to ascertain the significance of each respondent’s pre/posttests. • On an individual class basis, each respondent’s pre/posttests were tested to determine the significance level of changes in learning outcomes. • The significance levels for the pretest and posttest were then complied and compared to one another to determine the level of significance for the entire class. • Each class’s total significance level score was recorded in a master file. The final significance level for the study was determined by averaging all of the course significant level scores. • A .05 level of significance was used to determine significance.

12. Findings This research study consisted of adult learners in accelerated undergraduate and graduate program. In this research project, a pre/post evaluation of learning assessment was developed and tested with a sample of 72 adult students enrolled in an accelerated degree program at a large urban university. This metropolitan area is composed of a diverse population. The response rate to survey was nearly 100%.

13. Key Findings The t-Test: Paired Two Sample for Means was employed to compare the averages of the subjects’ pre/posttests – example is the result of one course average significant level.

14. Key Findings The decision whether to accept or reject the null hypothesis was determined by averaging the P one-tail scores of every class. A .05 level of significance was used to define significance. The results for the null hypothesis are shown in Slide 13.

15. Key Findings

16. Key Findings • The null research hypothesis was not accepted in regard to changes in student learning outcomes as a consequence of the faculty member’s teaching practices. • The .004 level of significance indicated that there is a 99.996% chance of a type 1 error if the null hypothesis was accepted. • The alternative hypothesis of this study was accepted that there is a statistically significant difference in the learning outcomes of adult students enrolled in the course due to the teaching practices of the faculty member.

17. Summary, Conclusions And Recommendations • This survey and methodology should motivate others to test it and begin a discussion of the larger issues. • For example, a comparison between adult learners and traditional aged students would be of interest to ascertain if these two groups learn differently and therefore need faculty members to modify or specialize in teaching these groups. • Other interesting questions include do males and females learn differently; do different ethnic groups learn differently than the majority population; do specific academic disciplines learn differently than others; do on-line students learn differently than face to face students? • A final question is how should student evaluation surveys and their results be used within an academic institution’s promotion, institutional effectiveness and accreditation processes? • To a degree, the standardized test instruments administered regularly are useful methods to accumulate (like an autopsy) what transpired across the curriculum during a given semester. For example students in CMB 510 (Table 1) rated this faculty member a 5 (on a 1 (low) to 5 (high) scale for the key question on the institution’s standard end of course survey: “I was satisfied with the performance of the faculty member for this course” (See Appendix B for complete post course survey results). A perfect or high score is always welcomed by a faculty member and impressive for promotion. However, this score does not indicate if any measureable learning took place. • There is room for both types of analysis.

18. Questions and Discussion