W. Hui,* P.J.-H. Hu,† T.H.K. Clark,‡ K.Y. Tam‡ & J. Milton§

Technology-assisted learning: a longitudinalfield study of knowledge category, learningeffectiveness and satisfaction inlanguage learning W. Hui,* P.J.-H. Hu,† T.H.K. Clark,‡ K.Y. Tam‡ & J. Milton§ *College of Information Technology, Zayed University, Abu Dhabi, United Arab Emirates †Accounting and Information Systems, David Eccles School of Business, University of Utah, Salt Lake City, Utah, USA ‡Information and Systems Management, School of Business and Management, Hong Kong University of Science and Technology, ClearWater Bay, Hong Kong, China §Language Center, School of Humanities, Hong Kong University of Science and Technology, ClearWater Bay, Hong Kong, China Journal of Computer Assisted Learning, Vol. 24, 245-259, 2007

Introduction • Some researchers, including Zhang et al. (2004), suggest technology-assisted learning can substitute for some conventional, face-to-face, classroom-based learning. • In this context, an instructor can deliver course materials through a designated Web site, from which students access materials and interact with the instructor (and perhaps peers) remotely. • A competing view holds that technology-assisted learning should be used only to complement face-to-face learning, which demands a hybrid or blended approach to leverage the respective strengths of each type of learning, such as using technology-assisted learning in some areas rather than replacing face-to-face, classroom-based learning altogether.

Introduction • According to Masie (2002) and Frederickson et al. (2004), this hybrid approach represents a preferable and arguably more advantageous means of using technology-assisted learning. • However, with either approach, a fundamental question : Does the use of technology-assisted learning improve students’ learning effectiveness and satisfaction? • With this background, we propose a factor model that explains students’ learning satisfaction and empirically test the model using evaluative responses collected from a longitudinal field experiment. • We focus on students’ learning of English as a foreign language, which typically spans different aspects of language learning, including vocabulary, grammar, listening, speaking and reading.

Introduction • Therefore, we examine the following research questions: (1) Is learning effectiveness associated with technology- assisted learning contingent on target knowledge? (2) What are the essential antecedents of learning satisfaction in technology-assisted learning? • Our longitudinal field experiment investigates the effects of technology-assisted learning by comparing students’ learning effectivenessacross different important aspects of English learning with technology-assisted versus face-to-face learning. • Our first study group contains students who use face-to-face learning exclusively, whereas the second uses both technology-assisted and face-to-face learning (i.e. hybrid approach).

Introduction • Our study design thus supports an analysis of between-group differences with respect to the target knowledge category and the combined effect of target knowledge and learning medium. • We also use the collected data to test the proposed learning satisfaction model, which consists of essential satisfaction antecedents (i.e. perceived learning community support, course learnability, learning effectiveness) in technology-assisted learning.

Literature review Previous research into learning effectiveness in technology-assisted learning • Several studies report positive effects of technology-assisted learning, including Johnson et al. (2000), who compare learning methods in human resource developments and show that students in the technology-assisted group perceive the instructor more positively and rate the overall course quality higher than their counterparts in the face-to-face group. • Abraham (2002) designs a virtual classroom for student learning about information systems and finds that technology-assisted learning improves learning feedback to students but the higher resulting learning effectiveness is not significantly better than that observed in face-to-face, classroom-based learning.

Literature review Previous research into learning effectiveness in technology-assisted learning • A meta-analysis by Bernard et al. (2004) suggests that theimpact of technology-assisted learning is not significant, consistent with Clark’s (1983) contention that the delivery medium has only marginal effects on the outcomes of planned instruction, measured according to learning effectiveness or satisfaction.

Literature review Previous research into learning satisfaction in technology-assisted learning • A review of extant literature on the critical topic of learning satisfaction (Allen et al. 2002; Wang 2003) suggests limited investigations of the essential factors that affect learning satisfaction, even though such investigations are particularly important when considering the relatively high dropout rate associated with technology-assisted learning (Hiltz & Wellman 1997; Kumar et al. 2001). • Consistent with Keller (1983), we define learning satisfaction as the perception of being able to achieve success and positive feelings about achieved outcomes.

Literature review Previous research into learning satisfaction in technology-assisted learning • Furthermore, on the basis of an extensive literature review, we identify three essential satisfaction determinants: (1) Learning effectiveness (Keller 1983; Wang 2003), (2) Perceived course learnability (Roca et al. 2006) and(3)Perceived learning community support (Wang 2003; Liaw 2004; Chou & Liu 2005). • According to Martin-Michiellot and Mendelsohn (2000), materials delivered in an easy-to-learn fashion can enhance students’ learning effectiveness and satisfaction.

Literature review Previous research into learning satisfaction in technology-assisted learning • In this study, perceived course learnability refers to the degree to which a student considers the course materials delivered through technology-assisted or face-to-face learningeasy to learn. • Consistent with Wang (2003), we define perceived learning community support as the extent to which a learning environment creates an active, strongly bonded community that encourages and facilitates knowledge exchanges among peers and their instructors.

Experiential learning model and implications for language learning • The experiential learning model assumes an iterative nature of learning through experience, from reflection and conceptualization to action and then enhanced experience (Osland et al. 2001). • According to this model, technology-assisted learning may be less effective for some aspects of language skills. For example, by engaging in live speaking drills or role plays, students can recognize their speaking problems directly and concretely (i.e. concrete experience). • Such iterative processing reinforces student learning, but technology-assisted learning provides only limited support in this sense.

Experiential learning model and implications for language learning • However, technology-assisted learning may better support other aspects of language learning because of the convenient access it offers to learning materials pertinent to vocabulary, reading, or grammar, which students may study repetitively at their preferred time and pace.

Hypotheses and research model • We objectively measure students’ learning effectiveness using test scores on listening, vocabulary and grammar exercises. • An online learning environment can provide listening exercises, but the effectiveness may not be comparable to classroom-based learning because all students in the classroom acquire listening comprehension when one student engages in a speaking exercise with the instructor. • As a result, technology-assisted learning should offer less learning support through concrete experience, which diminishes the effectiveness of the learning cycle conceptualized by Kolb (1976).

However, in the acquisition of vocabulary and grammar skills, concrete experience plays a lesser role, so the electronic channel can provide effective lessons. • Because the learning materials are available to the students anytime and anywhere, they can absorb materials better at their own place and take the time to reflect on the proper use of words and grammar. • H1: Students in the face-to-face group show greater improvement in listening comprehension than their counterparts in the technology-assisted learning group. • H2: Students in the technology-assisted learning group show greater improvement in vocabulary than their counterparts in the face-to-face group. • H3: Students in the technology-assisted learning group show greater improvement in grammar than their counterparts in the face-to-face group.

Hypotheses and research model • In addition, we examine students’ satisfaction with technology-assisted learning using a factor model that contains key satisfaction determinants, such as perceived learning community support, learnability and effectiveness. • Existing pedagogical theories emphasize the socially constructed nature of learning, which indicates it essentially involves sharing and negotiation (Gulz 2005). • Neo (2003) empirically supports collaborative learning for enhancing students’ problem-solving and critical thinking skills. • Accordingly, we posit a positive correlation between perceived learning community support and learning effectiveness.

Hypotheses and research model • H4: In technology-assisted learning, perceived learning community support is positively correlated with perceived learning effectiveness. • H5: In technology-assisted learning, perceived learnability is positively correlated with perceived learning effectiveness. • As suggested by Keller (1983), learning satisfaction relates directly to perceptions and feelings about learning effectiveness or outcomes. • Therefore, we expect a positive correlation between perceived learning effectiveness and learning satisfaction. • Both Liaw (2004) and Chou and Liu (2005) reveal that information and experience sharing among peers and group members increases students’ learning satisfaction.

Hypotheses and research model • A relatively learnable course gives students a sense of satisfaction because they overcome challenges they encounter during the learning process. • H6: In technology-assisted learning, perceived effectiveness is positively correlated with learning satisfaction. • H7: In technology-assisted learning, perceived learning community support is positively correlated with learning satisfaction. • H8: In technology-assisted learning, perceived course learnability is positively correlated with learning satisfaction.

Fig 1. Hypotheses and research model for explaining learning satisfaction

Study Design Experimental design • A computer program assigned subjects to either the technology-assisted learning or face-to-face learning scenario, which creates to a between-groups design. • Our control group uses face-to-face learning exclusively, whereas the treatment group receives a combination of face-to-face and technology-assisted learning. (Hybrid approach) Subjects • The participants are first-year students at a major university in Hong Kong who enrolled in the freshman English class mandated by the university.

Study Design Dependent variables and measurements • We measure learning effectiveness objectively by comparing the difference between the pre- and post-study test scores, conducted at the beginning and end of the semester. • We examine subjects’ learning satisfaction and assessments of perceived course learnability and learning community support (Piccoli et al. 2001; Aragon et al. 2002;Wang 2003) by adapting previously validated question items to operationalize each investigated construct, with some minor wording changes appropriate to the targeted learning context.

Study Design Dependent variables and measurements • All question items are based on a seven-point Likert scale, with 1 as ‘strongly disagree’ and 7 as ‘strongly agree’.

(LS; 6 items) (PLE; 6 items) (CL; 3 items) (CLS; 3 items)

Study Design Data collections • Our data are longitudinal, collected in the fall semester (September – December) of 2004. • At the beginning of the semester, each subject took an English test online(pre-test), and this score serves as a baseline against which we evaluate the subject’s learning effectiveness at the end of the semester.

Data analysis and results • A total of 507 subjects, 29.4% of the first-year student population, voluntarily took part in the study. • As a result, our effective sample includes 438 subjects who averaged 19.1 years of age and were fairly balanced in their gender distribution. • Noticeably, more male than female subjects appear in the technology-assisted learning group, but more female than male subjects were in the face-to-face group in effective samples.

(four kinds of learning style) It appears that more abstract thinkers joined the face-to-face group than the technology-assisted learning group (i.e. 63% v.s 53%), whereas more reflective observers were in the face-to-face group than in the technology-assisted learning group (i.e. 38% v.s 28%).

Table 2, the Cronbach’s alpha value of each investigated construct exceeds or is close to 0.7, the commonly suggested threshold for reliability (Nunnally & Bernstein 1994). • We also assess the instrument’s convergent and discriminant validity by performing a principal components analysis using the Varimax method with Kaiser normalization rotation. Table 2. Summary of descriptive statistics and construct reliability analysis.

The eigenvalue ofeach extracted factor exceeds 1.0, the common threshold value. • Overall, our analysis shows that the instrument exhibits adequate convergent and discriminant validity. Table 3. Reliability and discriminant validity of the study instrument.

Data analysis and results Technology-assisted versus face-to-face learning • For each of the learning aspects we study, we perform the following regression: where ‘technology-assisted learning’ is the dummy variable and has a value of 1 if the subjects are in the technology-assisted learning group and 0 otherwise. • If the coefficient of technology-assisted learning is significant, we conclude there is a significant difference between technology-assisted and face-to-face learning. [o, 1]

As we show in Table 4, technology-assisted learning has a significant impact on students’ performance with regard to listening comprehension and vocabulary. • As hypothesized, the face-to-face group achieves better performance in listening than the technology-assisted group, but the latter reveals enhanced vocabulary skills. • Likewise, for grammar skills, the technology-assisted learning groupperforms better than the face-to-face group, though the difference is not significant.

Thus, our results suggest that technology-assisted learning can enhance certain aspects of language learning, particularly those that emphasize reflective observation and do not require extensive human interaction (i.e. explicit knowledge). • However, for aspects of language learning that rely more on concrete experience through human interactions (i.e. tacit knowledge), technology-assisted learning may be less effective.

Data analysis and results H8 0.31* • We use the CALIS procedure in SAS to test the hypothesized structural equation model. • The estimated structural model appears in Fig 2, together with the measurement model. • Overall, our model shows satisfactory explanatory power, accounting for 59% of the variance in perceived learning effectiveness and79% of the variance in learning satisfaction. • All factor loadings and path coefficients are statistically significant. H5 0.47 H6 0.76* H4 0.50* H7 0.38*

Bentler’s comparative fit index (CFI) is greater than 0.90, a common cut-off for a good fit. • The root mean square error of approximation is less than 0.08, indicating an adequate model fit. • Furthermore, the goodness-of-fit index (GFI), GFI adjusted for degrees of freedom, and Bentler-Bonett normed fit index are all close to the commonly suggested 0.90 benchmark. • Collectively, these fit index values suggest a satisfactory fit of our model and the data.

Technology-assisted learning can better support the acquisition of vocabulary (H2), whereas face-to-face learning seems to facilitate students’ listening skills more effectively (H1). • Students in the technology-assisted learning group perform slightly better than those in the face-to-face group, as hypothesized, but the difference is not significant (H3). X

Our data further show that perceived course learnability and learning community support represent important predictors of perceived effectiveness, in support of H4 and H5. • In addition, perceived course learnability, perceived effectiveness and perceived learning community support are significant predictors of learning satisfaction, in support of H6–H8. Perceived Course Learnability H8 H5 Perceived Effectiveness Learning Satisfaction H6 H4 Perceived Learning Community Support H7

Summary and future research directions • This study contributes to technology-assisted learning literature by empirically examining target knowledge category (type) as a key moderating factor of online learning effectiveness. • We also contribute to technology-assisted learning research by proposing and empirically testing a factor model that explains and predicts students’ learning satisfaction in such settings, which directly affects their dropout decisions. • We also contribute to research in technology-assisted language training by showing that the use of technology-assisted learning can improve students’ vocabulary skills but may undermine their listening comprehension, with probable explanations.

Summary and future research directions • In addition, this study contains several limitations that suggest future research directions. 1. our sampling process might introduce bias. - For example, our study might have appealed more to students who value monetary rewards or are more eager to share their learning experiences. - Therefore, readers should take caution when generalizing our findings. 2. We do not directly measure each subject’s effort or workload, such as the average number of hours each student spends on English learning per week. - Therefore, we cannot comfortably rule out the possibility that the differences we observe between the two groups result from the differential workloads of their students.

Summary and future research directions 3. The course Web site represents another potential limitation, in that though it resembles typical technology-assisted learning platforms, it is designed primarily to support asynchronous learning. - Continued research should examine advanced technology-assisted learning platforms that support synchronouscommunications and live human interactions via rich media, such as video conferencing. 4. We study technology-assisted learning effectiveness and satisfactiononly in the context of language training. - Further research should expand to other areas (e.g. information systems, management) to generate empirical evidence with greater validity and generalizability.

W. Hui,* P.J.-H. Hu,† T.H.K. Clark,‡ K.Y. Tam‡ & J. Milton§