Evaluating uses of Learning Technology

1. Evaluating uses of Learning Technology Martin Oliver,MST/London Knowledge Lab

2. Overview • Some general definitions and history • What is it we’re discussing, exactly? • Issues for evaluating learning technology • What problems do people face when doing this? • Data and distance • What can we find, and what does it tell us? • Tools to support people (very brief) • What resources exist to support people? • The ELT toolkit project (A detailed look at a tool) • How can tools like these make a difference? • What can we learn from them? • Conclusions

3. Prelude • This first section’s really about orientation • Think about your experiences of evaluation • If you can think of a positive experience: • What was it that made it good? • …even if you can’t think of one: • What was it about your experiences that were bad? • Capture these and revisit them later

4. What do I mean by evaluation? • A contested term • Judgements about the value (benefits) and worth (costs) of something • A way of describing something (ethnographic) • What evaluators do… • What I don’t mean • Entirely personal judgements like reviews (or checklists), without data collection • Assessing student learning • …entirely valid things, but not what I’m talking about

5. A brief history of educational evaluation • The beginning… • A tradition that grew from measurement theory • Firmly rooted in the experimental method • Educational interventions as things applied to populations • …and the backlash (c. 1970s) • Alternative traditions rejected this approach • Illuminative, ethnograpic, naturalistic approaches arose • Sought to re-define what counted as ‘valid’ evaluation, but politically weaker • This pattern still evident • US legislation endorses controlled experiments

6. A paradigm war? • Certainly two opposed traditions • Battle lines seem drawn around methods • …but Hammersley suggests it’s more about philosophy • Logical positivists looking for stable, controllable interventions - even if they’re doing qualitative grounded theory • Relativists looking to interpret what’s happening and make recommendations based on personal judgements • Points out that many people are actually eclectic, rather than hidebound (“principled”?)

7. A third way • Patton: utilization-focused evaluation • Most commissioned reports are never read • A good evaluation isn’t one that’s methodologically rigorous - it’s one that helps people act (make decisions) • Principle of designing for “intended use by intended users” • No good working for people who won’t act - choose a different audience • No good working for those who are powerless to act - instead, influence those with power(There are ethical issues here…)

8. Repositioning evaluation • Evaluation positioned as a social, political activity, not as value-free science • Emphasis on rhetoric - persuading an audience (understood not as a ‘type’ or ‘role’ but as a list of names) • ‘Stakeholders’ - whose voice will be included in this process? What authority, if any, will it have? (Will they just provide data, or help frame the study, interpret, present, etc.?) • Recognition that the evaluator has a stake in the process too - reflects on their credibility and integrity

9. Repositioning evaluation • Also: treating evaluation as an educational intervention - “process use” • What can those involved learn? From negotiating its scope, gathering data, contributing to analysis, debating interpretations…? • Looking at opportunities for feedback - ongoing interventions for improvement (action), not just summative judgement (cf. Feedback in learning) • Creating opportunities for dialogue between ‘stakeholders’ - socially constructed understandings, involving multiple perspectives

10. Revisiting your experiences • Review the list from the start • To what extent are the “good” features consistent with utilization-focused evaluation? • Are the “bad” experiences linked to a particular approach? • Are any of you unwitting utilization-focused sympathisers?!

11. End of the first part • Overview of relevant issues from educational evaluation • Intended to provide a common ground and insights into the specific problems of learning technology • Also directly relevant to any other evaluation you do • Next, on to our specific concern…

12. So what about Learning Technology? • This section will look at the things that are distinctive about evaluation in relation to learning technology • Evaluation within this context echoes wider shifts in educational evaluation • …although usually a few years later… • Same contestation between paradigms

13. Evaluating Learning Technology • Some novel features • Large number of practitioner-researchers • No formal training as evaluators • ‘Common sense’ evaluators - no theoretical foundation to their work • Large number of funded projects are told to evaluate their work(esp. after TLTP phase I - lots of development, no information about its value!) • Many have to evaluate the project that pays their salary, sometimes with an external ‘check’

14. Evaluating Learning Technology • Same power-laden confrontation between paradigms • Qualitative, interpretative perspective common in action research/practitioner studies • Quantitative, positivistic perspective championed by policymakers and Evidence-Based Practice • Say “what works”, not explain why (“easy answers” for funders) • Hierarchy of evidence – from RCTs to qualitative methods and GOBSAT

15. Evaluating Learning Technology • Particular problems with comparative studies • A difficulty for all educational evaluation • If one condition is believed to be better, can you justify withholding it, ethically? • If outcomes are affected by teaching, changing outcomes changes what was learnt – so can’t use same assessment as no longer appropriate (Constructive Alignment) • Particular difficulty for Learning Technology • What is “E-Learning”, “Blended Learning”, etc anyhow?

16. Evaluating Learning Technology • “Is e-learning better than traditional forms of learning?” • What we mean by e-learning today isn’t what we meant a year ago • It’s not the type, it’s the specific instance – is this well designed? • It’s not what it is, it’s what you do with it – are you using it well? • What exactly is “traditional” learning? Do we really want to assume this is a stable point of comparison? • “No significant difference” phenomenon – tends to be different, not “better”, unless study is designed to measure what e-learning does well • “Are books better than other resources for learning?”

17. Evaluating Learning Technology • The difficulty of attribution • One innovation amongst many: post-compulsory education is riddled with new initiatives; what’s the root of any particular change? • False negatives: students learn differently but cover this up in order to perform on ‘normal’ tests (learn new and old forms of knowledge) • False positives: technology a symptom of a wider change in attitudes or practice (is technology the symptom or cause of widening access in post-compulsory education, or are both symptoms of something else?) • Coincidence: groups may just be different (even if randomly created, but particularly if cohorts)

18. Evaluating Learning Technology • What kind of comparisons can we draw? • People can and do undertake comparative studies • Tend to compare preferences for one intervention or another • Sometimes measure group performance against some invariant test (e.g. standardised exam) – point of reference • Typically try to ‘control out’ things interpretative researchers find interesting (influence of teacher, etc.) • Can be informative and useful if you’re treating this as a case study – an insight into specific performance, rather than building a general law

19. The problem of impact • A common request: • “Establish the impact of this new form of teaching” • Remarkably hard to answer • A brief exercise (2-3 minutes): • List ways in which introducing a new form of technology can have an impact • Consider a variety of roles/people • What kinds of evidence might you look for in each case?

20. The problem of impact • The TLTP III EFFECTS project • National initiative to see whether accreditation and staff support was an incentive to lecturers to adopt new technology • Required evidence of selecting, planning, implementing, evaluating, disseminating technology use • Now established as a SEDA award (PDF-ELT) • External evaluation: establish the impact of the project • How could we make sense of this?

21. The problem of impact • Multi-layered model • What was the impact on learners? • What was the impact on lecturers? • What was the impact on the organisation? • What was the impact nationally?

22. The problem of impact • Initial plan was different evaluators for different levels • Lecturers evaluate impact on learners • Local project team evaluate impact on lecturers and organisation • Project evaluators evaluate national impact • Worked ok, but… • Lecturers hadn’t the time and found this hard • ‘Evaluation’ the weakest outcome in assessment across all sites • A picture with lots of holes

23. The problem of impact • As an example: impact on staff • Evidence of change in teaching practice (observation, documentation) • Evidence of changed role(Promotion, involvement in committees) • Evidence of change in career direction(Publishing educational research) • Evidence of change in attitudes(Self report, changed use of discourse) • Detailed picture, but not amenable to widespread study; relied on snapshots

24. The problem of impact • To summarise: • Easy to find evidence of change, hard to establish what it means for our study • Hard to draw out general conclusions when the ‘focus’ is so ill-defined • Much can be said that’s relevant to local practice – evidence of some kind of impact is almost unavoidable (but is it the right kind…?) • Need a position to interpret this against and good rhetoric to present this to others • http://ifets.ieee.org/periodical/vol_3_2002/v_3_2002.html

25. End of the second part • This section’s focused on the specifics of evaluating learning technology • Highlighted issues of design and interpretation facing studies in this area • Raised a number of general concerns, including disposition and approach of those asked to evaluate • Next section will look at the issues of gathering data

26. Data and distance • The previous section has explained why it’s hard to establish impact • This section will look at some of the problems of getting hold of data at all • Issues arise in relation to the point of new technologies • Many introduced to increase flexibility • Many claim to support new forms of learning • What can we gather as evidence and what can we infer from it?

27. Data and distance • “Informating” (Zuboff) • Computerised activities make things explicit and generate information • Such information is, potentially, data • Discussion archives, use logs, site hits, etc. • A helpful source of information? • Ready for processing - in electronic form • Only a partial account of what took place

28. Data and distance • Can we see what we need to? • Chris Jones: ethnographic study of a course with online collaboration • Incident of ‘cheating’ observed face-to-face • Calls into doubt the veracity of ‘easy’ data • ‘Distance Pedagogics’ (Peters) • Lecturing at a distance (e.g. video link) is still lecturing and needs no new pedagogy • Private study may be on-campus but provides flexibility and choice to students and raises issues familiar to traditional distance educators • Similar issues here: public or private? • Whose context?

29. Data and distance • Importance of triangulation • Each part reveals an element of the wider picture • Interpreting multiple sources of data reassures and (potentially) explicates • However… • Ongoing problem: most of what’s important with learning is private, so how can we learn about this?

30. Data and distance • Old methods • Travel for observations - time intensive; fine for cases, but less good for general conclusions • Travel for interviews, or interview by phone • Surveys etc • All are opportunities, but access becomes a serious issue (and no guarantees data will be provided)

31. Data and Distance • Old methods in new formats • Online survey - higher response rates, but caution about missing out those least happy with technology (usually an important group) • Online interviews/focus groups - readily-captured data, but different skills and pace required; more thoughtful, less spontaneous; if open (rather than selected), vocal minority an issue • However, mostly self-report; what else can be accessed?

32. Data and distance • New(er) methods • Traces - hit logs: ‘dumb’ data (about access, not use or intention) that needs interpretation • Discussion archives - access to exchanges that are fleeting in traditional settings • Can raise ethical issues • ‘spyware’ for data • status of comments (as permanent, as data) in online discussions - data protection act • Can be easy but inappropriate to gather lots of data

33. Summary of part three • General issues • Getting the data you need is harder, as it’s private and distributed • Online data collection methods harder to control, which may raise questions about interpretation • Questions of interpretation raised: ethical justifications, relationship to context(s) • Next section will look at tools designed to help with this

34. Supporting people who evaluate learning technology • This section builds upon a problem mentioned in passing in EFFECTS evaluation • A pressure to evaluate technology use • Lots of conceptual and methodological issues • No support or training for teachers

35. Supporting people who evaluate learning technology • The response • A plethora of tools to ‘fix’ this • Assumption that a sensible teacher with a bit of information and guidance will get through ok • Relevance to this: • Raise awareness of tools like this • Highlight ‘good’ (sanctioned) approaches to practice

36. Tools to support evaluation • Existing tools support data analysis • SPSS, NVivo, etc. • …but only if you know what they should be used for • Another collection of tools focuses on evaluation design • LTDI Evaluation cookbook • TLT Flashlight project (US) • MEDA Evaluation tool for training software • ELT Toolkit

37. Different kinds of tool • MEDA - a handbook of questions • Flashlight - a database of question • Works on the assumption that “very different educators need to ask similar questions”, using surveys • LTDI - a cookbook of methods • Assumes educators can choose a topic but might need help with methods • ELT toolkit - tries to do both http://www.elt.ac.uk/materials.htm#evaldiss

38. The evolution of the ELT toolkit • The brief: produce something that will help practitioners evaluate in spite of the difficulties • The initial idea: a structured walkthrough supporting study design • Selection of methodology • Selection of methods (guided by methodology) • Selection of data analysis methods

39. The evolution of the ELT toolkit • It didn’t work • First study: couldn’t even get to using the toolkit • Participant didn’t know what they wanted to know • Goals kept shifting • Discussion led to re-framing… endlessly • Patton’s “process use” • Being involved in evaluation design was educating for the participant but this didn’t actually help them carry out a study!

40. The evolution of the ELT toolkit • Revision of the tool • Introduced new steps to address context • Identification of stakeholders (individuals) • List their concerns and turn these into questions • (Existing three steps: methodology, data collection methods, data analysis methods) • Approaches to communicate findings to audience(s) • Explicitly framing this as a social process • Similar to Draper/TILT’s “inner” and “outer” steps for evaluation design

41. The evolution of the ELT toolkit • Developed a paper-based version of the tool • Tested it • It worked fairly well, but took a long time • Received funding from the JISC • Implemented as the online Evaluation of Learning and Media Toolkit(Incorporated another tool for curriculum design - badly) • http://www.ltss.bris.ac.uk/jcalt/ • Expanded functionality

42. The evolution of the ELT toolkit • Three main sections • Evaluation planner • Evaluation adviser • Evaluation presenter • Plus things we didn’t want lost • The methodology section JISC didn’t like • References and links • Within each section three types of activity • Tell us something (free text entry) - usually context • Make an open choice (list of suggestions with ‘other: please specify’ option) • Enter data and a model recommends things

43. The evolution of the ELT toolkit • Data entered pulled together in a final report • Printed off as 2-4 page evaluation plan • Summarises decisions made • Captures contextual information • Presents this in an ordered way • Amenable to sharing with others (managers, funders, research assistants) or using as an outline workplan • Simple idea; huge success with users! • Further development: option to share plans

44. An example • Forming questions had been a problem • Developed an activity to support question framing • Start with concerns • Rephrase these as a series of questions • Combine or contrast different types of questions • Pick the one that seems most useful

45. An example • Concern: student learning • Exploratory questions • What do students learn? Who learns best with this resource? How do students use it? What do they think about it? • Comparative questions • Do some students use it differently to others? Does this group perform better on tests than a group that doesn’t use it? • Measurement questions • How long do they use it for? Do they use it all? If students do better on tests, how much by? How many complaints where there about it?

46. An example • Negative questions • What was wrong with it? Why did students dislike it? Did it hinder their learning? Who found it hard to use? What problems did it cause for people? • And then, combined questions • How much better did these students do than those? Why did some students like it more than others? What led to problems arising for students? • And finally - select the one question (ok, at most, three) that your study will seek to answer

47. Evaluating the Evaluation Toolkit • Had to do it to ourselves… • Used the toolkit to do our evaluation plan • Implemented it • Published in JCAL 18 (2), 2002 • What we learnt • The usability of the tool was fairly poor • Took about 3-4.5 hours to do a full plan • Experts thought this was wonderful • Novices thought this was far too long • Editing shared plans one way to reduce this

48. Evaluating the Evaluation Toolkit • A success? • It did its job - even complete novices produced credible plans • In addition - experts prompted to think about methods they hadn’t previously used • Suggests people were learning from this as well as ‘just’ designing studies • However… • “Comfort zone” - if novices wanted to do a survey and it wasn’t recommended they’d over-ride the list - needs to be “challenging enough” • No evidence of impact on practice - did anyone implement the plan? (Longitudinal studies)

49. Summary of part three • Given the complexities of evaluating learning technology, it’s no surprise people need support • Tools are regularly developed as a way of providing this • They can be interesting in their own right • Impact of ELT Toolkit on design • However, hard to judge their own impact • But also interesting as a representation of “good” practice

50. Summary of part four • Toolkit designed as a stand-alone resource • …although works best when introduced in a workshop, preferably with peer discussion • Whether used or not, useful as a way of highlighting issues and suggesting structure • Provides outline of a decision making process (stakeholders, questions, data collection, analysis, presentation) – useful for planning, but also for staff support, training, etc. • Highlights complexities – each activity relates to an area worth thinking about; sensitises to issues • Can be educational to look at tools like these, whether or not you follow them