1 / 37

MHCI PSLC Data Shop Project

MHCI PSLC Data Shop Project. Spring Progress Report. The Team. Sandi Lowe Project Manager. Jason Hum Design Lead. Sam Zaiss Documents Manager. Jeff Wong Tech Lead. Meghan Myers Client Liaison. Outline. What is the PSLC? What is the Data Shop? Analyzing Data Exporting Data

tobias
Download Presentation

MHCI PSLC Data Shop Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MHCI PSLC Data Shop Project Spring Progress Report

  2. The Team Sandi Lowe Project Manager Jason Hum Design Lead Sam Zaiss Documents Manager Jeff Wong Tech Lead Meghan Myers Client Liaison MHCI PSLC Team | Spring 2005

  3. Outline • What is the PSLC? • What is the Data Shop? • Analyzing Data • Exporting Data • Project Themes • Preliminary Design • Project Plan • Final Takeaways MHCI PSLC Team | Spring 2005

  4. What is the PSLC? PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free- Form Export MHCI PSLC Team | Spring 2005

  5. Background on PSLC • Further current education research • Enable new education research • Support collaboration • Support LearnLab courses MHCI PSLC Team | Spring 2005

  6. Background on LearnLab Courses • 7 course areas under PSLC • 3 Language (Chinese, French, ESL) • 2 Math (Algebra, Geometry) • 2 Science (Physics, Chemistry) • Various stages of development MHCI PSLC Team | Spring 2005

  7. What is the Data Shop? PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free-Form Export MHCI PSLC Team | Spring 2005

  8. Collecting the Data • Allows researchers to set up an in-vivo LearnLab experiment • Facilitates logging of data in these experiments MHCI PSLC Team | Spring 2005

  9. Processing the Data • Strip away student information • Perform computations on data to streamline analysis • Create supporting data tables MHCI PSLC Team | Spring 2005

  10. Accessing the Data • Built as a web application • Selected data analysis techniques • Data filtering and exportation MHCI PSLC Team | Spring 2005

  11. Analyzing Data: Pre-Defined Reports PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free-Form Export MHCI PSLC Team | Spring 2005

  12. Learning Curves • Displays students’ performance over time on a particular knowledge component • Log Files must include: • Knowledge Components • Student performance on individual opportunities • Match Knowledge Components to Problems / Sub-Goals • Supporting Data: • Requirements Solicitation (Koedinger, Aleven, Ritter) • General Research Contextual Inquiries (U2) • LearnLab Research Contextual Inquiries (U13) Courtesy of Data Shop MHCI PSLC Team | Spring 2005

  13. Help Effectiveness Report • The probability of the Section getting task or sub-goal x incorrect after a particular help message • Log Files must include: • Help / Hints • Whether a student received help, and if the answer that followed was correct or incorrect • Measure of student’s knowledge before help message • Supporting Data: • Requirements Solicitation (VanLehn, Ritter) • General Research Contextual Inquiries (U1, U10) • LearnLab Research Contextual Inquiries (U11, U12) MHCI PSLC Team | Spring 2005

  14. Error Report • Determine common mistakes that students make on tasks • Log Files must include: • Erroneous Input & Tutor Response to Error • Error Classifications • Supporting Data: • Requirements Solicitation (VanLehn, Koedinger, Ritter) • General Research Contextual Inquiries (U3, U8, U10) • LearnLab Research Contextual Inquiries (U11, U12) MHCI PSLC Team | Spring 2005

  15. Problem Profile • Context and student performance on a particular problem • Log Files must include: • Problem Text • Record of students’ answers on the problem and whether the answers were correct or incorrect • Supporting Data: • General Research Contextual Inquiries (U1, U3, U7, U8, U9, U10) • LearnLab Research Contextual Inquiries (U11, U12) • Course Committee Survey (Chem) • Think Aloud Pilot (U2) MHCI PSLC Team | Spring 2005

  16. Analyzing Data: Free-Form Analysis PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free-Form Export MHCI PSLC Team | Spring 2005

  17. Timeline Visualization • Visualizes the pattern of each student’s actions • Log Files must include: • Time measurements (ms) • Supporting Data: • Users want to drill down and tell stories • General Research Contextual Inquiries (U1, U5, U8, U9, U10) Courtesy of Andrea Knight MHCI PSLC Team | Spring 2005

  18. Session Browser Courtesy of Jack Mostow • Visualizes the pattern and context of each student’s actions • Log Files must include: • Time measurements (ms) • Event Hierarchy • Supporting Data: • Users want to drill down and tell stories • Requirements Solicitation (VanLehn) • General Research Contextual Inquiries (U1, U3, U4, U5, U7, U8, U9, U10) MHCI PSLC Team | Spring 2005

  19. Event Filtering (aka Rollup Table) • Allows researcher to specify their variables and calculations over those variables • Supporting Data: • Requirements Solicitation (Koedinger, VanLehn) • General Research Contextual Inquiries (U5, U8) • U5: “By the time I get the data I need out of the system, I’ve forgotten the question I wanted to ask” MHCI PSLC Team | Spring 2005

  20. Exporting Data PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free-Form Export MHCI PSLC Team | Spring 2005

  21. Project Themes PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free-Form Export MHCI PSLC Team | Spring 2005

  22. Project Themes 1. Support All LearnLab Tutors MHCI PSLC Team | Spring 2005

  23. Project Themes 2. Facilitate Inter-Report Navigation MHCI PSLC Team | Spring 2005

  24. Project Themes 3. Let Researchers See Only the Data They Want to See, When They Want to See It. • Reports show only what’s relevant • Find out how researchers want to narrow down their data in reports • Minimize interface elements for each report MHCI PSLC Team | Spring 2005

  25. Preliminary Design MHCI PSLC Team | Spring 2005

  26. Design for Theme 2 • Large buttons to move between reports • Key Considerations: • Data transfer from report to report • Saving reports with interesting data along the way • Issues of scale MHCI PSLC Team | Spring 2005

  27. Design for Theme 3 • Specialized filtering for each report • Key Considerations: • Condensing report specification & modification into one set of controls • Distinguishing between singular & multiple selection • Enabling fast selection • Design Support: • List boxes for long lists of variables (StatTransfer) • Clear representation of curriculum hierarchy (Froogle) • Design Challenges: • Minimizing space while maximizing speed • Providing sufficient context to the user MHCI PSLC Team | Spring 2005

  28. User Testing Findings • 2 users doing an analysis with learning curves • 1st year HCI researcher PhD, familiarity with tutor studies • 4th year education researcher PhD, no tutor studies • Needs discovered: • Compare curves against pre- and post-tests • Show p-value and regression equation • Describe sample • Click on a Learning Curve spike to understand context • Show text of the problem MHCI PSLC Team | Spring 2005

  29. Project Plan 1 wk Pre-Workshop: 6 wks Post-Workshop: 5 wks Pre-Summer Workshop • Add breadth to current structure • Error Reports • Help Effectiveness • Problem Profiles • Develop Filtering Designs (Mid-Fi) • Prototype Session Browser & Timeline Visualization (Low-Fi) • Heuristic Evaluation • User Testing MHCI PSLC Team | Spring 2005

  30. Project Plan 1 wk Pre-Workshop: 6 wks Post-Workshop: 5 wks Summer Workshop (Week of June 27th) • High-Intensity User Testing, Iteration Post-Summer Workshop • Database Connections for Pre-Defined Reports • Session Browser & Timeline Visualization (Mid-Fi) • User Testing • Revisions MHCI PSLC Team | Spring 2005

  31. Final Takeaways 1. Rules for Data Sharing • Sharing outside research group • Competitive Analysis revealed need for sharing regulations • U5: “I’m happy to share my data if the researchers follow PSLC’s guidelines” • For now, no open sharing • Sharing inside research group • Seen as a need in General Research CIs (U1 - U8) • Designed & tested by end of summer MHCI PSLC Team | Spring 2005

  32. Final Takeaways 2. Context Matters • Understanding data is hard without context • Issues discovered to date: • Complex naming => complex design (CA) • Tutor writers use intimate knowledge of the tutor frequently (U5, U7, U9, U11, U12) • Researchers unfamiliar to the tutor need that intimate knowledge (TA Pilot Test) • TA Pilot Test U2: “So you’d have… a master book that would tell you what all the problems are? Because I’d never remember.” MHCI PSLC Team | Spring 2005

  33. Final Takeaways 3. Courses in Development Should Log Necessary Data • Survey course committees as needed • Proposed reports will function with current PSLC tutors • Can’t predict what future tutors may and may not log MHCI PSLC Team | Spring 2005

  34. Fin Feedback?MHCI PSLC Project Teamhcii-mhciproject-pslc@lists.andrew.cmu.edu MHCI PSLC Team | Spring 2005

  35. Behavior Graph • Illustrates the strategies students are using to solve problems • Log Files must include: • Sub-goals • Supporting Data: • Requirements Solicitation (Koedinger) • General Research Contextual Inquiries (U8, U10) • Course Committee Survey (Chem) • Think Aloud Pilot (U2) Courtesy of Data Shop MHCI PSLC Team | Spring 2005

  36. Physics Survey MHCI PSLC Team | Spring 2005

  37. Chinese/French Survey MHCI PSLC Team | Spring 2005

More Related