Evaluating Task-Based Language Programs Colloquium – TBLT 2009

1. NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics, to reduce the storage and downloading requirements (sorry, no pretty photos!). Evaluating Task-Based Language Programs Colloquium – TBLT 2009

2. The colloquium Why bother with TBLT program evaluation? Three presentations + clarification questions (2:00-3:30): Re-framing the evaluation of task-based language education Evaluating a TBLT Spanish immersion program Evaluation of TBLT in Flanders Open audience-panel discussion (3:30-3:50)

3. John M. Norris University of Hawai‘i at Mānoa TBLT 2009 Please cite as: Norris, J. M. (2009, September). Reframing the evaluation of task-based language education. Paper presented at the refereed colloquium “Evaluating task-based language programs”, at the 3rd International Conference on Task-Based Language Teaching, Lancaster, UK (September 14, 2009). Re-framing the evaluation of task-based language education

4. TBLL v. TBLT: Disconnects between inquiry and practice

5. What is task-based language learning (TBLL)? 1. Societal need for change in language education… Value Outcomes Methods 2. Emerging notions of L2 acquisition… Processes Impediments Indicators

6. What is task-based language learning (TBLL)? Practices 1. Societal need for change in language education… Value Outcomes Methods Observations Proposals Opportunity for a researched language pedagogy Discussions 2. Emerging notions of L2 acquisition… Processes Impediments Indicators Hypotheses Findings

7. What is task-based language learning (TBLL)? Pedagogic principles, such as… Promote learning by doing, experiential learning Use task as the unit of analysis for instruction & assessment Provide rich L2 input Elaborate (rather than simplify) L2 input Respect learner-internal syllabuses Enable inductive/chunk learning Promote collaborative-cooperative interaction Provide focus on form, negative feedback (e.g., Doughty & Long, 2003; Ellis, 2003)

8. What is task-based language teaching (TBLT)? Rationales and Principles L2 Education Programs Applied to Philosophy of Education Learners Task-Based Language Teaching Cognitive Psychology Needs Curriculum Sociocultural theory Instruction SLA Materials Curriculum theory Assessment Planning and policy Inform??? Teacher development

9. What is the role of task-based inquiry? Test hypotheses DISCRETE GENERALIZABLE Discover robust, if small, truths Task-Based Language Learning Generate theory Task-Based Language Teaching Understand what works, where, when, & why Inform curriculum, course design HOLISTIC SITUATED Improve teaching practice

10. What is the role of task-based inquiry? Test hypotheses DISCRETE GENERALIZABLE Discover robust, if small, truths Task-Based Language Learning Generate theory Task-Based Language Teaching Understand what works, where, when, & why Inform curriculum, course design HOLISTIC SITUATED Improve teaching practice

11. What is the role of task-based inquiry? Test hypotheses DISCRETE GENERALIZABLE Discover robust, if small, truths Task-Based Language Learning Generate theory Task-Based Language Teaching Understand what works, where, when, & why Inform curriculum, course design HOLISTIC SITUATED Improve teaching practice

12. Challenges for task-based inquiry 1. The scope of task-based research does not match the scope of our claims about (for or against) TBLT. 2. The focus of theoretical task-based research does not relate to the situated realities of task-based teaching. Needs Curriculum Materials Instruction Teachers Learners Assessment Goals, outcomes Scope, sequence Resources Practices History, training Individual differences Intended uses, users TBLT Education Programs

13. Framing TBLT inquiry through program evaluation

14. Inquiry through evaluation Research emphasizes theoretical, conclusion-oriented inquiry Evaluation operationalizes decision-oriented inquiry Cronbach & Suppes (1969)

15. Inquiry through evaluation Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology. Norris (2006) MLJ Perspectives Inquiry frame and focus Inquiry impetus Inquiry question prioritization

16. Inquiry through evaluation Paradigms Cronbach et al. (1980) “The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7) Epistemology 1 Epistemology 2 Methodology 1 Methodology 2

17. Inquiry through evaluation Paradigms Cronbach et al. (1980) “The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7) Epistemology 1 Epistemology 2 Methodology 1 Methodology 2

18. Inquiry through evaluation Pragmatism Cronbach et al. (1980) “The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7) Why? Who? What? When? Method 1 Method 2 Method 3 Method 4 Method 5

19. Inquiry through evaluation 1. Participation – stakeholders, representatives, primary intended users 2. Prioritization – challenges, questions in immediate need of answers 3. Instrumentation – what data will answer the questions? 4. Collection – how can we get data in available time/resources? 5. Interpretation – what do findings mean in context? 6. Utilization – what decisions & actions are taken? Participation by language educators is essential throughout evaluation if contextual relevance is sought. A focus on specific intended uses for evaluation findings is essential from the outset, if evaluation is to make any difference. Language educators are ultimately responsible for what happens in language education.

20. Context: Program features Learner needs, institutional resources, program goals and outcomes, curriculum, materials, instruction, assessment, teachers, teacher development, learners, etc. Context: Intended uses Context: Intended users Evaluating TBLT Programs in situ Understand Improve Educate Demonstrate worth Hold accountable Empower (Test theory) Teachers Administrators Curriculum writers Learners Parents/public Funders (Researchers) Questions + Methods Values clarification Implementation Process-product Outcomes Patton (1997) Utilization-focused evaluation

21. Learning from evaluation examples: the brief history of TBLT

22. Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Prabhu’s “Bangalore Project” (See Prabhu, 1987) Context Theory Program English L2 Education in Bangalore, India; Seeking Improvement via Innovation L2 learning by processing meaning; Unconscious grammar construction by learners Project/task- based work; 4 experimental schools; Implemented 1979-1984

23. Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985) Initial inquiry, final year of the project: “To assess, through appropriate tests, whether there is any demonstrable difference in terms of attainment in English between classes of children who have been taught on the CT project and their peers who have received normal instruction in the respective schools.” Beretta & Davies (1985)

24. Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985) Purpose Test theory Demonstrate method effectiveness Findings Structures test: Control > CTP Contextual grammar: Control = CTP Dictation: Control = CTP List/Read comp: CTP > Control Task-based test: CTP > Control Claims??? Task-based learners achieved as much or more than traditional on all but the least functional outcomes Task-based instruction is successful Warranted claims??? What do we really know??? Methods Quasi-experimentation Class/method comparison Outcome achievement assessments

25. Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 2 (See Beretta, 1986, 1990, 1992) Purpose Understand program implementation Illuminate relation with apparent outcomes Findings Lack of comparability (intact classes, no baseline data) More qualified teachers in CTP classes Implementation of CTP highly variable (over time, between classes, with structures) More confident teachers = better results Methods Retrospective interview protocols Teacher level of concern questionnaires Document analysis

26. Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Lessons Learned (See Beretta, 1992) Theory testing, methods comparisons, “what works” claims are rarely feasible in real educational programs Apparent differences in learning achievement, behaviors, etc. can only be explained by observation of multiple factors as they are experienced in real program contexts Even poorly executed evaluations (e.g., post-hoc) can shed light on how programs function and help explain why learning does or does not occur Teachers (beliefs, training, commitment, time) play a key role in implementing programs: what they actually do must be understood

27. Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Context Theory Program French FL Education, UK university, Salford; Restructuring advanced FL teaching Input, text, task; Learning through form- function mapping in tasks Task-based syllabus; Multiple levels at university; Implemented 1988-96

28. Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Multiple iterations of development, implementation, evaluation, revision: “Curriculum design, evaluation, application and enhancement is a slow process, and subject to a number of extraneous influences which make it impossible to measure with totally scientific precision…use of diaries and questionnaires on the first occasion enabled a number of lessons to be learned and these helped considerably in creating a second application where the testimony of the student population through a detailed questionnaire shows the success of the operation.” Towell & Tomlinson (1999)

29. Learning from TBLT evaluation Program: Initial TBLT long group projects Program: Revised TBLT staged, short projects • Methods: • Learner diaries • Learner surveys • Assessments/exams • Methods: • Learner surveys • Focus groups • Assessments/exams STAGE 1 STAGE 2 • Findings: • Projects too long (6 wks) • Training in group work • Staged task objectives • Gains in text/task learning • Developing accuracy? • Findings: • Increased satisfaction • Higher learning of skills • Improved oral translation • Written translation? • Developing accuracy?

30. Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Student/learner perspective on teaching with tasks sheds important light on the realities of implementation (how + how well) Building evaluation activities into curricular delivery from the outset (e.g., student diaries), enables longitudinal insights about change, development, response to instruction Triangulated learner feedback (diaries, self-assessments, questionnaires, exams) can lead to effective improvements in curriculum and task design, and in turn to higher evaluations Learners can change how they learn—acculturating to TBLT—especially when tasks, instructions, assessments are intentionally designed and staged to do so

31. Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007) Context Theory Program English FL education, Thai university; Improving EAP instruction Integrated-skills, communication; Life-long learning; Learner needs + interest orientation Task-based syllabus; English department; Implemented 12 months

32. Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007) Inquiry for developing and improving TBLT experiences: “…relatively few empirical studies have documented how teachers and learners react to entirely task-based courses, as opposed to the use of individual tasks…The purposes of this case study were (a) to identify teacher and learner reactions to the course and (b) to describe how their concerns, if any, were addressed.” McDonough & Chaikitmongkol (2007)

33. Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007) • Methods: • Learner task evaluations (repeated) • Learning notebooks • Class observations • Student course evaluations • Teacher/student interviews • Observer field notes • Findings: • Increased learner independence, language skills, learning strategies • Decreased grammar obsession • Non-specific real-world relevance • Need time to adjust (T&L) • More support, guidance from teachers • Too much to cover, disparate materials • Uses: • Intro unit on language learning • Teacher’s guide to instruction + workshop • Enhanced task guidelines, built-in feedback opportunities • Reduced number of tasks • Consolidated materials

34. Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007) Cycles of evaluation planned into TBLT innovation, and carried throughout, can lead to increased likelihood of effectiveness Systematic evaluation (a) from multiple stakeholder perspectives and (b) focused on multiple program elements (materials, preparedness, outcomes) enables balanced change Teachers and learners both require support in implementing TBLT, especially during early phases of introducing task-based instruction TBLT based on learner needs can work well in EAP contexts, especially when evaluation is used to support on-going effectiveness of delivery from the outset

35. Learning from TBLT evaluation Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006) Context Theory Program Dutch SL Education in Flanders; Nationwide; K-16; Ensuring Educational Access, Equity Large-scale Task-Based LT Innovation; Improving Functional DSL Abilities School-based Teacher- Training Programs; Enabling Change, 1994-2003

36. Learning from TBLT evaluation Van den Branden (2006): “…the teacher tries to act as a true interactional partner, negotiating meaning and content with the students, eliciting and encouraging their output, focusing on form when appropriate and offering them a rich, relevant and communicative input” (p. 217). Evaluation PROBLEMS Evaluation USES Teacher cognition Understand teachers Illuminate context Improve T-dev program Encourage teacher agency Ensure teaching success Enable TBLT learning Demonstrate outcomes • What do they theorize about TBLT? • How do they learn about TBLT? Teacher action • Are they willing to change with syllabus? • How do they adopt/adapt TBLT in practice? Teaching context • What are the social constraints on T-Dev? • How can T-Dev be optimized?

37. Learning from TBLT evaluation Program: Theoretical inservice training Program: TB training + syllabus support Program: Training + coaching + agency • Methods: • Teacher survey • Training observation • Methods: • Teacher logs, interviews, classroom observations • Methods: • Coaching obs, classroom obs, coach/ teacher interviews STAGE 1 STAGE 2 STAGE 3 • Findings: • Transmission model • Short term (3 hrs.) • “Try that with my students”…Post-coursal depression! Findings: +awareness of TBLT +student enthusiasm ?teacher adoption -teacher control -task complexity -groupwork Findings: +conscious decisions +TBLT adaptation +self-evaluation ?teacher control -transfer -groupwork

38. Learning from TBLT evaluation Program: Sustained implementation of TBLT T-Dev with coaching, support • Methods: • Pre-post student learning outcomes, teacher surveys, classroom observations STAGE 4 Findings: +incorporation of TBLT correlated with higher Dutch L2 proficiency outcomes +3-year gains in DSL higher in TBLT intensive adopting schools ?mixed incorporation of TBLT across schools, teachers

39. Learning from TBLT evaluation Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006) Long-term evaluation of TBLT sheds light on how ideas are implemented, how participants change, and what support is needed Multi-directional evaluation (political, social, school, individual) increases our capacity to explain why task-based ideas work or do not Persistent follow-through on evaluation findings (use) underlies effective innovation Teachers can learn to engage with TBLT, but change takes time, requires individualized support, and must be valued

40. Learning from TBLT evaluation What have we learned? Reframing evaluation in TBLT From summative to intentional From assessment-driven to multi-methodological From external to participatory From method-testing to program-illuminating From one-shot to longitudinal, cyclical From theoretical conclusions to educational decisions

41. Research, evaluation, and the future of task-based education

42. TBLL research  Sociocultural, cognitive, and other theories provide useful starting points for thinking about language teaching and learning, and offer principles for building educational programs  Task-based language learning research helps in that it raises our awareness about particular factors that we should pay attention to in the instructed L2 learning process  Task-based language learning research cannot tell us much about how or why language education programs work; findings from TBLL research should not be interpreted as direct implications for TBLT education

43. TBLT evaluation Answers questions & informs decisions of local interest Sheds light on how TBLT ideas work in practice Provides truths situated in rich contexts of programs Intentional Evaluative Inquiry Relates outcomes to TBLT delivery and other factors Focuses on scope that is meaningful to teachers, learners Tests and informs innovation on the ground, in situ Empowers participants to learn, and learn to change Forces an honest accounting of TBLT

44. TBLT evaluation Resources: It takes time and money to do evaluation well and to sustain it within L2 educational programs. Challenges For TBLT Evaluation Training: Effective evaluation calls upon skills that may not be easily available among personnel at hand. Dissemination: There are few venues for publishing evaluation reports, thereby limiting learning. Actual uses: There are many possible uses/needs for evaluation that we are not sufficiently attuned to, yet.

45. Cheers! (Mahalo!) References Beretta, A. (1986). Program-fair language teaching evaluation. TESOL Quarterly, 20, 431-445. Beretta, A. (1990). Implementation of the Bangalore Project. Applied Linguistics, 11(4), 321-340. Beretta, A. (1992). What can be learned from the Bangalore evaluation? In J. C. Alderson and A. Beretta (eds.), Evaluating second language education (pp. 250-273). Cambridge: Cambridge University Press. Beretta, A., & Davies, A. (1985). Evaluation of the Bangalore Project. ELT Journal, 39(2), 121-127. Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass. Cronbach, L., & Suppes, (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan, 1969. Doughty, C., & Long, M. H. (2003). Optimal psycholinguistic environments for distance foreign language learning. Language Learning & Technology, 7, 50-80. Ellis, R. (2003). Task-based language learning and teaching. New York: Oxford University Press. McDonough, K., & Chaikitmongkol, W. (2007). Teachers’ and learners’ reactions to a task-based EFL course in Thailand. TESOL Quarterly, 41(1), 107-132. Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597. Norris, J. M. (2009). Task-based teaching and testing. In M. Long and C. Doughty (Eds.), Handbook of language teaching (pp. 578-594). Cambridge: Blackwell. Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage. Prabhu, N. S. (1987). Second language pedagogy. Oxford University Press, Oxford. Towell, R., & Tomlinson, P. (1999). Language curriculum development research at university level. Language Teaching Research, 3(1), 1-32. Van den Branden, K. (2006). Training teachers: Task-based as well? In K. Van den Branden (ed.), Task-based language teaching in practice (pp. 217-273). Cambridge: Cambridge University Press. http://www2.hawaii.edu/~jnorris