Analyzing CS Competencies using The SOLO Taxonomy

ITiCSE'09 – Keynote Analyzing CS Competenciesusing The SOLO Taxonomy Claus Brabrand ((( brabrand@itu.dk ))) ((( http://www.itu.dk/people/brabrand/ ))) Associate Professor, IT University of Copenhagen Denmark

Outline • 1) Introduction • Constructive Alignment • The SOLO Taxonomy • 2) From Content to Competence • Advocate a shift in perspective • Elaborate The SOLO Taxonomy • 3) Analyzing CS Competencies • …using The SOLO Taxonomy • Compare: CS vs NAT vs MAT

Introduction to…: • Constructive Alignment & SOLO Taxonomy: John Biggs’ popular and heavily cited book: “Teaching for Quality Learning at University - What the student does” Note: 3rd Edition now available [J.Biggs & C.Tang, 2009] “Teaching Teaching & Understanding Understanding” 19 min award-winning short-film on Constructive Alignment (available on DVD in 7 languages, epilogue by John Biggs)

T Activation Exercise • Discuss with your neighbour: What are the ‘main messages’of the film (which did YOU findparticularly relevant, …if any)?

From Content to Competence • My old course descriptions (Concurrency 2004): • Given in terms of a 'content description': • Essentially: • Goal is…: • To understand: • deadlock • interference • synchronization • ... This is a bad idea for two reasons...!

Problem 1 ! • Problem with 'content' as goals ! analyze ... theorize ... analyze systems explain causes define deadlock describe solutions agreement Stud. C • Goal is…: • To understand: • deadlock • interference • synchronization • ... tacit knowledge from a research-based tradition not known by student Teacher name solutions recite conditons analyze systems explain causes Stud. B  P.S.: even if it were possible to agree, we know that the exam will dictate the learning anyway. Stud. A Censor

Problem 2 ! • Problem with 'understanding' as goals ! • Goal is…: • To understand: • deadlock • interference • synchronization • ... 'concept of deadlock' ?!  The answer is simple: It cannot be measured !

Competence ! • 'Competence' as goals ! Competence:= knowledge+ capacity to actupon it Have the student dosomething; and then "measure" the product and/or process • Objective ! • To learn how to: • analyze systems for... • explain cause/effects... • prove properties of... • compare methods of... • ... Note: 'understanding' is of coursepre-requisitional !  Note':inherently operational (~ verbs) 'SOLO' = Structure of the ObservedLearning Outcome

SOLO Advantages • Advantages of The SOLO Taxonomy: • Linear hierarchical structure • Aimed at evaluating student learning • Converges on research(at SOLO 5) Research:Production ofnewknowledge

Note: the list is non-exhaustive Graphic Legend problem / question / cue known related issue - given! hypothetical related issue - not given! student response Q R R Q R Q R Q Q R R' SOLO (elaborated) QUALITATIVE QUANTITATIVE SOLO 2 ”uni-structural” SOLO 3 “multi-structural” SOLO 4 “relational” SOLO 5 “extended abstract” • define • identify • count • name • recite • paraphrase • follow (simple)instructions • … • combine • structure • describe • classify • enumerate • list • do algorithm • apply method • … • analyze • compare • contrast • integrate • relate • explain causes • apply theory (to its domain) • … • theorize • generalize • hypothesize • predict • judge • reflect • transfer theory (to new domain) • …

Using SOLO in Practice • Recommendations on course descriptions: 1) Use 'standard formulation': a) puts learning focus on the student b) competence formulation: "to be able to" Intended Learning Outcomes [Algorithms 101] After the course, the students are expected to be able to: identifyand formulatealgorithmic problems ; classifyand comparealgorithms ; construct and analyzealgorithms using standard paradigms; implementalgorithms for simple problems. 4)Avoid 'understanding-goals': "To understand X", "Be familiar with Y", "Have a notion of Z", ...! V N V V N V V V N N V 3) Use 'Verb + Noun' formulation: What the student is expected to dowith a given matter 2) List sub-goals as 'bullets': Clearer than text N V

T Activation Exercise Which do you predict are keyCScompetences ? Concurrency: analyze systems compare models

Outline • 1) Introduction • Constructive Alignment • The SOLO Taxonomy • 2) From Content to Competence • Advocate a shift in perspective • Elaborate The SOLO Taxonomy • 3) Analyzing CS Competencies • …using The SOLO Taxonomy • Compare: CS vs NAT vs MAT Joint work withBettina Dahl at Aarhus University

Grade Scales Conversion (between EU countries): 7 steps: 4 steps 4 steps 8 steps 8 steps ECTS 10 steps 10 steps SCALE ... ... ... ... ... 21 steps 21 steps A, B, C, D, E, Fx, F Grade := Degree of realizationof course objectives! All Universities: Explicit ILO's The SOLO Taxonomy!

Massive DATA set • Unique Opportunity…: • Systematically formulated ILO'sfor all courses • Quantifiable (analyzable) via The SOLO Taxonomy competencies 5,608 courses 734 21 institutes universities TWO

SOLO Mapping Mapped by: • B. Dahl & C. Brabrand With help from: • 3 Educational research colleagues (medicine) • J. Biggs & C. Tang

Top 10 Competencies • Top 10 Competencies: Natural Sciences " ":={Physics, Chemistry, Biology, Molecular Biology }

Histogram of Top Competencies • If we look closer (comparative visualization)...: More than 2x …also: • program • construct • structure More than 3x More than 3x MAT % NAT NAT MAT CS MAT MAT MAT CS CS MAT CS CS: 15 % NAT: 1.0 % MAT: 0.3 % CS: 14 % NAT: 14 % MAT: 60 % CS: 4.5 % NAT: 4.4 % MAT: 40 % Legend: Computer Science Natural Science Mathematics withapply

SOLO Distribution • SOLO distribution: • The 15% "programming competences"(all at SOLO 4): • {implement, program, design, construct, structure }  15%  E[X] = 3.7 E[X] = 3.4 E[X] = 3.1 Legend: SOLO 2 SOLO 3 SOLO 4 SOLO 5

Assumptions Assumptions: • SOLO is an appropriatecompetence measure (we refer to [J.Biggs & K.F.Collis, 1982] ) • Context independenceof SOLO mapping (for each competence we inspected several goals) • Subject independenceof SOLO mapping (we limit ourselves to a 'science context') • Equal weightassumptions (Competences in a goal & goals in a course have equal weight) • Outcomes: intendedformulatedachieved (we “analyze” formulated, but “reason about” achieved) [Biggs’ studies] [approximation] [approximation] [approximation] [implicational]

Conclusions • Most frequent CS Competences are: • describe (13%), explain (10%), apply method (9%), implement (7%), analyze (6%), … • "Programming-related" skills: • 15% of CS-curriculum • The "Essence of Math" is: • reproducing, formulating, proving, solving, argueing,(and applying) • SOLO-levels of subjects: • CS >SOLONAT >SOLOMAT 15%

Keynote Points • Constructive Alignment • …addresses many teaching / learning problems; e.g.: • Esp. student motivational issues (learning incentives) • ...and student performance issues (learning support) • The SOLO Taxonomy • …is good for reasoning about competencies: • Esp. for designingcourses and curricula • DATA • Study, analyze, and reflect on teaching / learning • …using (objective)DATA!

R Q R' Questions... Cognitive processes My research and teaching Course descriptions "understanding" content  competence Association new ~ old The SOLO Taxonomy 'TLA' Teaching / Learning Activities Teacher models levels 1 - 2 - 3 The Short-Film Susan & Robert The Book ? Student activation Tips'n'Tricks CS v. NAT v. MAT recite generalize 15% programming Students at University "What is good teaching?" Constructive Alignment John Biggs Top 10 Competences

Thank You! Film's homepage: ((( http://www.daimi.au.dk/~brabrand/short-film/ )))

Related References • ”Teaching for Quality Learning at University (what the student does)”John Biggs & Catherine TangSociety for Research into Higher Education, 2007. McGraw-Hill. • ”Evaluating the Quality of Learning: The SOLO Taxonomy”John Biggs & Kevin F. CollisLondon: Academic Press, 1982 • ”Teaching Teaching & Understanding Understanding”Claus Brabrand & Jacob Andersen19 minute award-winning short-film (DVD)Aarhus University Press, Aarhus University, 2006 • ”Using the SOLO Taxonomy to Analyze Competence Progression of University Science Curricula”Claus Brabrand & Bettina DahlHigher Education, 2009 • "Constructive Alignment & The SOLO Taxonomy: a Comparative Study of University Competencies in Computer Science vs. Mathematics"Claus Brabrand & Bettina DahlCRPIT, Vol. 88, ACS 3-17, R. Lister & Simon, Eds., 2007

Implementing Alignment • Alignment Implementation Process: 1) Think carefully about: overall goal of course (what are the stud. to learn?) 2)Operationalize these goals and formulate them as SOLO intended learning outcomes alignment learning incentive learning support 3)Choosecarefully the form(s) of assessment (~ intended learning outcomes) 4)Choosecarefully the form(s) of teaching (~ intended learning outcomes) Think of teaching activities as ”training for exam”

SOLO Progression • SOLO Progression: • Computer Science vs. Mathematics vs. …

Conclusion (Progression) • What have we really shown?!? A) SOLO has "proved" that progression exists in curricula (since we "believe" in SOLO as a measure) xor B) SOLO has "been proven" to be a good tool for analyzing competence progression (since we "believe" in the existence of progression)

Progression Assumptions Extra assumptions wrt. Progression: • Numeric quantification of SOLO [assumption] (i.e., numeric step from 2-3 is comparable to 3-4 and 4-5) • Progressionmanifests itself as competences [assumption] (i.e., in 'verb'-, not 'noun'-dimension)

SOLO Calculation Method • Calculation Example (for a course): • "SOLO average": • [ (2+3)/2 + (3+4)/2 + (4+4)/2 + 4 ] / 4 = 3.50 • "SOLO distribution": identify (2) and formulate (3) algorithmic problems; classify (3) andcompare (4) algorithms; construct(4)and analyze (4) algorithms using standard paradigms; implement (4) algorithms for simple problems. "double weight averaging"

T Neighbour Discussion Discuss with neighbour: "does this make sense ?!?" (content  competence) E.g.: ("Learning about programming"vs."Learning to program" )

T Activation Exercise III • Discuss with your neighbour: Discuss what you predict wewould find in the DATA set ? • Questions: • a) most frequent CS competences? • b) percentage of "programming-related" competences? • c) CS v. NAT v. MAT (wrt. SOLO levels)?

T Post-It exercise Write down 1-2 key competences (i.e., verbs) (for your course) Concurrency: analyze systems for deadlock compare models wrt. behavior

Tips'n'Tricks (activation) • Neighbour discussions: • Post-It exercise: • Form variation: • focus: zoom in • anonymous (!) • swap'able • everyone will engage • empathetic control • shared knowledge pool • more questions(students dare ask them) • better questions(students had a chance to discuss) [Phil Race] 1-2 min timeout • Frequent breaks: pulse reader measurements: lecturing blended with in-class activation exercises

NEW OLD Tips'n'Tricks (cont'd) • Use many examples:(build on student pre-knowledge) • Explicit structure: • Student 'recap' at end: 1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4. wwwwwww 1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4. wwwwwww 1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4. wwwwwww 1. xxxxxxxxxx 2. yyyyyyyyyy 3. zzzzzzzzzz 4. wwwwwww  • self evident to you [ teacher ] • not to a learner [ student ] (esp. during learning process) • "Less-is-more": • analyze • compare • relate common deadlock, uncommon deadlock, A-synchronization, B-synchronization, hand-shake, multi-party synchronization, multi-party hand-shake, binary semaphores, generalized semaphores, blocking semaphores, recursive locks, ... vs. now after 1 day after 1 week after 2 weeks after 3 weeks Emphasize depth over breadth (coverage)

Now, please: "3-minute recap" • Please spend 3' on thinking about and writing down the most important points from the talk – now!: Immediately After 1 day After 1 week After 2 weeks After 3 weeks

Problematic Courses • E.g. course: ”Databases”(at RUC/Roskilde): • Note: almost entirely non-operational(!) • i.e. measure how?! • obtain knowledge aboutthe structure of database systems; • be familiar with design of databases by useof special notations like E/R and analysisthrough normalization; • get an overview of the most important database models and a detailed knowledge about the most important model - the relational model as well as the language SQL; • get an overview of database indexing and query processing; • obtain knowledge about application programming for DB systems. Familiar with ?!

BONUS SLIDES

Based on John Biggs' Theories • 2nd edition • (3rd edition expected this fall) "Teaching for Quality Learning at University", John Biggs

UNALIGNED COURSE  Teacher’s intention Student’s activity • e.g. • explain • relate • prove • apply "Dealing with the test" Exam’s assessment • e.g. • memorize • describe • e.g. • memorize • describe

ALIGNED COURSE  Teacher’s intention Student’s activity • e.g. • explain • relate • prove • apply • e.g. • explain • relate • prove • apply • e.g. • explain • relate • prove • apply Exam’s assessment • e.g. • explain • relate • prove • apply • e.g. • explain • relate • prove • apply

Top 10 Competencies • Top 10 Competencies: Natural Sciences " ":={Physics, Chemistry, Biology, Molecular Biology }

Note: the list is non-exhaustive Graphic Legend problem / question / cue known related issue - given! hypothetical related issue - not given! student response Q R R Q R Q R Q Q R R' R1 Q R2 R3 SOLO (elaborated) QUALITATIVE QUANTITATIVE SOLO 2 ”uni-structural” SOLO 3 “multi-structural” SOLO 4 “relational” SOLO 5 “extended abstract” • define • identify • count • name • recite • paraphrase • follow (simple)instructions • … • combine • structure • describe • classify • enumerate • list • do algorithm • apply method • … • analyze • compare • contrast • integrate • relate • explain causes • apply theory (to its domain) • … • theorize • generalize • hypothesize • predict • judge • reflect • transfer theory (to new domain) • …

T Exercise • Buzz Session: 1) Discuss w/ neighbour: 2) Write it on a Post-It 3) SwapPost-Its… "which film messages did you find particularly relevant?" Just Keep Swapping…

Student Motivation • Susan: (”intrinsic motivation”) - wants to…: learn ! • Robert: (”extrinsic motivation”) - to…: pass exams!

Constructivism • ”Transmission is Dead…” : (lectures = ) • Knowledge is…Actively Constructed! ! active teacher & passive students risk

SOLO Taxonomy • Hierarchy for Competences: • Deep learning (not surface) ! 5: generalize, theorize, predict, … 4: explain, analyze, compare, … 3: describe, combine, classify, … 2: recite, identify, calculate, …

Stud Learning Focus • Focus on Student Learning ! (instead of ”what teacher does” & labelling students: ’good/bad’) • Studentactivitation  learning

Analyzing CS Competencies using The SOLO Taxonomy