NCME April 11, 2007

1. NCME April 11, 2007 Assessing Knowledge in a Learning Space: Validity and/or Reliability Jean-Claude Falmagne University of California, Irvine

2. Eric Cosyn Chris Doble Nicolas Thiery Hasan Uzun

3. Content: • Assessing Knowledge in a Learning Space • What is a Learning Space? • The Fringe Theorem • The Projection Theorem • Uncovering a Knowledge State • Validity/Reliability of the Assessment • Summary and Discussion

4. Content: • Assessing Knowledge in a Learning Space • What is a Learning Space? • The Fringe Theorem • The Projection Theorem • Uncovering a Knowledge State • Validity/Reliability of the Assessment • Summary and Discussion

5. Content: • Assessing Knowledge in a Learning Space • What is a Learning Space? • The Fringe Theorem • The Projection Theorem • Uncovering a Knowledge State • Validity/Reliability of the Assessment • Summary and Discussion

6. Assessing Knowledge in a Learning Space In its principle, a learning space enables a very accurate assessment engine capable of uncovering the knowledge state of a student among many possible ones.

7. In beginning algebra, for example, there are around 250 different types of problems (concepts, skills) that a student must master.

8. What do we mean by a “type of problem”?

9. Here is an example of a type of word problem. One story line--out of 5--goes as follows: "Abdul works mowing lawns and raking. He earns $5.40 an hour for mowing and $4.40 an hour for raking. How much will he earn for 5 hours of mowing and 1 hour of raking?"

10. Here is an example of a type of word problem. One story line--out of 5--goes as follows: "Abdul works mowing lawns and raking. He earns $5.40 an hour for mowing and $4.40 an hour for raking. How much will he earn for 5 hours of mowing and 1 hour of raking?” Another story line for the same type of word problem is: "In the past month Felipe rented 1 video cassette and 7 DVDs. The rental price for the video cassette was $2.80 . The rental price for each DVD was $3.20 . What is the total amount that Felipe spent on video cassette and DVD rentals in the past month?"

11. Let us say that there are five such story lines for that problem type. When selecting an instance of that word problem, one of those five story lines has to be chosen randomly, and the particular numbers involved (dollar amounts and whole numbers), must also be chosen. (Integer dollar amounts are excluded.) Overall, there are 28,125 instances to choose from, randomly, for that particular word problem.

12. In other words, in this particular case, one type of word problem corresponds to 28,125 items in the usual sense of a standardized test.

13. A knowledge state is a feasible subset of that set of 250 problem types.

14. The number of feasible knowledge states is typically very large. In beginning algebra, this number is on the order of 107.

15. Despite this very large number of states, it is nevertheless possible to assess a student’s knowledge state in the topic, accurately, in the span of 25-35 questions.

16. Content: • Assessing knowledge in a learning space • What is a Learning Space?

17. A learning space is a collection K of knowledge states. Each knowledge state is a subset of a set Q of problem types. Any knowledge state contains all the types of problems that some student in the population of reference might be capable of solving. The set Q (Not a partition) forming the collection K.

18. There are serious constraints on this concept. We suppose that a learning space always contains the empty set  and the full set Q of problem types. Thus, it is in principle possible for a student not to know anything, and for some other student to know everything.

19. Moreover, any learning spaceKsatisfies two pedagogically reasonable principles:

20. First Principle: If a knowledge state K is included in some other knowledge state K’ K’ K

21. First Principle: If a knowledge state K is included in some other knowledge state K’, then K’ K

22. First Principle: If a knowledge state K is included in some other knowledge state K’, then K’ K

23. First Principle: If a knowledge state K is included in some other knowledge state K’, then K’ K

24. First Principle: If a knowledge state K is included in some other knowledge state K’, then K’ K

25. First Principle: If a knowledge state K is included in some other knowledge state K’, then K’ K

26. First Principle: If a knowledge state K is included in some other knowledge state K’, then K’ K there exists at least one chain of states going from K to K’, each differing from the previous one by a single problem, so that learning can proceed from K to K’ by mastering the concepts one at a time.

27. In other words, the material must be learnable one step at a time, no matter where you are, and where you want to go beyond that.

28. Second Principle: If a problem type c is learnable from some knowledge state K K

29. Second Principle: If a problem type c is learnable from some knowledge state K c K

30. Second Principle: If a problem type c is learnable from some knowledge state K and if that knowledge state K is included in some knowledge state K’, c K K’

31. Second Principle: If a problem type c is learnable from some knowledge state K and if that knowledge state K is included in some knowledge state K’, c K’ then c should also be learnable from the state K’.

32. In other words: Knowing more can only help to learn something new.

33. These axioms have a Guttman’s scale flavor, but that intuition would be misleading: if a Guttman’s scale interpretation of a learning space was attempted for any realistic learning space, literally billions of Guttman’s scales would ensue. Such a construction would not be helpful. Also, a formal concept of skill maps has been developed within this framework (Doignon and Falmagne, 1999). So far, we do not use it.

34. A learning space on the set of problems {a,b,c,d,e,f }

35. Content: • Assessing Knowledge in a learning space • What is a Learning Space? • The Fringe Theorem

36. Except for the empty state (the empty set) and the full set Q, each knowledge state has two fringes: the outer fringe and the inner fringe.

37. Except for the empty state (the empty set) and the full set Q, each knowledge state has two fringes: the outer fringe and the inner fringe. The outer fringe of a state K in a learning space is the set of all problem types q such that K + {q} is also a state. In other words, intuitively: the outer fringe of a state K is the set of all problem types that are learnable from K. Outer fringe

38. Except for the empty state (the empty set) and the full set Q, each knowledge state has two fringes: the outer fringe and the inner fringe. The inner fringe of a state K in a learning space is the set of all problem types q such that K - {q} is also a state. In other words, intuitively: the inner fringe of a state K is the set of all problem types that could have been learned last. Inner fringe

39. Outer fringe of state K   set of problem types learnable from K Inner fringe of state K  set of most recently learned problem types in K

40. EXAMPLE OF AN INNER AND OUTER FRINGE OF A STATE IN ARITHMETIC (High Points)

41. A FUNDAMENTAL RESULT The Fringe Theorem: In a learning space, any state is characterized (or can be accurately summarized) by its inner and outer fringe.

42. A FUNDAMENTAL RESULT The Fringe Theorem: In a learning space, any state is characterized (or can be accurately summarized) by its inner and outer fringe. The two fringes provide a very good summary of an assessment, immediately giving an entry into learning: the student may be offered the choice of studying any problem type in the outer fringe of his or her state.

43. A FUNDAMENTAL RESULT The Fringe Theorem: In a learning space, any state is characterized (or can be accurately summarized) by its inner and outer fringe. The two fringes provide a very good summary of an assessment, immediately giving an entry into learning: the student may be offered the choice of studying any problem type in the outer fringe of his or her state. One can obviously also give a total score, or a score in any of the subtopics.

47. Content: • What is ALEKS? • What is a Learning Space? • What are the fringes? • The Fringe Theorem • The Projection Theorem

48. What is a Projection of a Learning Space? Intuitively, it is a smaller learning space giving a macroscopic view of the bigger one, a concept useful for a placement test or similar tests.

49. A learning space on the set of problems {a,b,c,d,e,f }

50. We begin by selecting a subset of {a,b,c,d,e,f}. We first choose Problem type b. This choice determines a bipartition of the set of states.