‘Towards a competency model for adaptive assessment to support lifelong learning’

1. TENCompetence workshop Service Oriented Approaches and Lifelong Competence Development Infrastructures Manchester, 11th – 12th January 2007 ‘Towards a competency model for adaptive assessment to support lifelong learning’ Onjira Sitthisak , Lester Gilbert and Hugh C. Davis Learning Technologies Group ECS,University of Southampton

2. Assessment • The main goal of assessment has shifted away from content-based education to intended learning outcomes. • Assessment is related to the accomplishment of learning outcomes, i.e. competence. • Learners expect to be able to maintain and expose their competency profiles to multiple services throughout their life.

3. Adaptive Assessment System • Assessment is part of the process of diagnosing the learner’s knowledge level. • One of the main goals includes offering personalized support according to the personal needs and ability of each learner (Brusilovsky, 1996) • The “learner’s estimated knowledge level” can be used to guide the adaptation of the system.

4. Three problems with adaptiveassessment • Inconsistency in estimating the learner’s knowledge level • Each system classifies ability or knowledge level and difficulty level of assessment using different approaches and techniques. • This causes interoperability and reusability problems if the learner’s knowledge level in one system needs to be used in other systems.

5. Three problems with adaptive assessment • Limitation of a single numerical value for a learner’s knowledge level • Many well-known theories for selecting questions assume one numerical measure for the relationship between ability and item response. • A numerical value may be appropriate to decide who the best learner is, but educational evaluation intends to assess the learners’ readiness for further learning (Falmagne et al., 2003). • Selecting a question in adaptive assessment should therefore be multidimensional.

6. Three problems with adaptive assessment • Dependency on a specific knowledge domain • In most cases, adaptive assessment systems are developed for a specific knowledge domain using particular rules and assessments without potential for knowledge reuse. • There is no standard to combine different knowledge domains with their assessments and learned capabilities (Cheniti-Belcadhi and Braham, 2004). • This highlights the problem of supporting the assessment of lifelong learning across multiple domains.

7. The need for competency • The proposed solution is the use of “learned capability” instead of estimated “knowledge level”. • Knowledge space theory suggests that a capability for solving problems can be established from achieved competencies (Doignon and Falmagne, 1985). • A competency model should support • storing, organising and sharing • achieved, current, and intended performance data • all aspects of education and training • persistent and standard way.

8. Criteria for a competency model • Competency should be defined with a rich data structure to support a learner’s competency profile throughout life. • Meeting lifelong personal needs requires a highly flexible competency model. • A competency model should support the selection of suitable questions in an adaptive assessment system. • Competency should be concerned with specific, identifiable and measurable behaviours (Draganidis and Mentzas, 2006).

9. Proposed competency model

10. Competencymodel and Learning Design • In IMS Learning Design (LD), prerequisites and learning objectives can be defined using an unstructured textual resource or an IMS RDCEO specification (Koper and Tattersall, 2005). • This is inadequate for an instructional designer seeking to design learning activities, environments and assessments (Paquette and Rosca, 2004). • IMS LD should incorporate a structured competency definition in order to implement a Unit Of Learning (UOL) with a solid instructional design foundation.

11. Competency model and service oriented architecture (SOA) • Competency modelling should be the shared responsibility of governments, educational institutions, and businesses. • SOA is currently considered the approach of choice in supporting cross-institutional cooperation and the design, build, and management of a distributed computing infrastructure. • The proposed competency model is compatible with the SOA that may support a collaborative virtual teaching and learning environment (CVTLE) SOA (Gilbert et al., 2006).

12. Future work • The proposed competency model is being developed and implemented in the JISC-funded Placement Learning and Assessment Toolkit (mPLAT) project (http://www.mplat.ecs.soton.ac.uk/) at the University of Southampton. • mPLAT project aims to provide a mobile learning toolkit to support practice based learning, mentoring and assessment of trainee nurses.

13. Conclusion • A competency model is critical to successfully managing adaptive assessment and achieving the goals of resource sharing, collaboration and automation to support lifelong learning.

14. Thank you • Questions, comments, discussion…

15. References • Brusilovsky, P., 1996. 'Methods and Techniques of Adaptive Hypermedia', User Modeling and User-Adapted Interaction, Vol. 6, No. 2-3, pp.87-129. • Falmagne, J.-C., Cosyn, E., Doignon, J.-P. and Thiery, N. (2003) 'The Assessment of Knowledge, in Theory and in Practice', Integration of Knowledge Intensive Multi-Agent Systems. • Cheniti-Belcadhi, L. and Braham, R. (2004) 'A generic framework for assessment in adaptive educational hypermedia', Proceedings of IADIS International Conference WWW/Internet 2004 (ICWI 2004). Madrid, Spain. • Doignon, J.-P. and Falmagne, J.-C. (1985) 'Spaces for the Assessment of Knowledge', International Journal of Man-Machine Studies, No. 23, pp.175-196. • Draganidis, F. and Mentzas, G. (2006) 'Competency based management: a review of systems and approaches', Information Management & Computer Security, Vol. 14, No. 1, pp.51-64. • Koper, R. and Tattersall, C. (Eds.) (2005) Learning Design · A Handbook on Modelling and Delivering Networked Education and Training, Springer. • Paquette, G. and Rosca, I. (2004) 'An ontology-based Referencing of Actors, Operations and Resources in eLearning Systems', the 2nd International Workshop on Applications of Semantic Web Technologies for E-Learning. Eindhoven, The Netherlands. • Gilbert, L., Sitthisak, O., Sim, Y. W., Wang, C. and Wills, G. (2006) 'From collaborative virtual research environment to teaching and learning', In Proceedings of TENCompetence Workshop: Learning Networks for Lifelong Competence Development. Sofia, Bulgaria.