1 / 58

MHCI PSLC Data Shop Project

MHCI PSLC Data Shop Project. Final Design Presentation. The Team. Sandi Lowe. Jason Hum. Sam Zaiss. Jeff Wong. Meghan Myers. Outline. Background: PSLC & The Data Shop The MHCI Project Use Case Scenario Prototype Demo Implementation Timeline Wrap Up. Outline.

dillian
Download Presentation

MHCI PSLC Data Shop Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MHCI PSLC Data Shop Project Final Design Presentation

  2. The Team Sandi Lowe Jason Hum Sam Zaiss Jeff Wong Meghan Myers MHCI PSLC Team | Summer 2005

  3. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  4. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  5. What is the PSLC? PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free- Form Export MHCI PSLC Team | Summer 2005

  6. PSLC Goals • Further current education research • Enable new education research • Support collaboration • Support 7 LearnLab courses MHCI PSLC Team | Summer 2005

  7. What is the Data Shop? PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free- Form Export MHCI PSLC Team | Summer 2005

  8. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  9. The MHCI Data Shop Project Access Pre-Defined Free-Form Learning Curves Error Report Problem Profile Help FX Behavior Graph Timeline Viz Session Browser Data Export Export MHCI PSLC Team | Summer 2005

  10. Project Requirements • High-Fidelity Proof-of-Concept Prototype • In-Depth Research Including Weekly User Testing • 13 Contextual Inquiries • 8 Requirements Interviews • 12 Competitive Analyses • 37 User Tests • Deliverables • Current Prototype • Requirements Document • Design Specification • Supporting Data • Design Iterations MHCI PSLC Team | Summer 2005

  11. MHCI Project Timeline Start of Summer Summer Workshop End • Began With Low Fidelity Paper Prototypes • Gradually Added Features & Increased Fidelity • Weekly User Testing Throughout UserTesting Iteration Hi-Fi Prototype Paper Prototypes Mid-Fi Prototypes MHCI PSLC Team | Summer 2005

  12. Project Themes • Context Matters • Facilitate Inter-Report Navigation • Create Specialized Reports • Emphasize Visual Communication MHCI PSLC Team | Summer 2005

  13. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  14. ITS Background • Intelligent Tutoring Systems (ITSs) used: • To help students learn • To gain insight into how students learn • Consist of a series of problems in a particular subject. • Order of problems is random • Each problem composed of a number of steps; each of which test one or more knowledge components MHCI PSLC Team | Summer 2005

  15. Meet Dave Jargenson Name: David Jargenson, Ph.D. Age: 35 Affiliation: Research Scientist, Wisconsin Center for Education Research • Interested in how students learn Algebra. • Has been doing education research for 10 years. • The PSLC’s newest member, he has already run a study with the Center. • Trying out the Data Shop with a couple basic studies. MHCI PSLC Team | Summer 2005

  16. Dave’s Data Exploration Determine How Students Are Performing Overall Compare Performance of 2 Samples Select Certain Transactions from the Data Shop For Further Analysis Determine Most Frequent Unexpected Error MHCI PSLC Team | Summer 2005

  17. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  18. Error Report • Supporting Data: • Requirements Solicitation (VanLehn, Koedinger, Ritter) • General Research Contextual Inquiries (U3, U8, U10) • LearnLab Research Contextual Inquiries (U11, U12) • Goal: • View all mistakes that students made, by frequency, steps and knowledge components MHCI PSLC Team | Summer 2005

  19. Error Report "Here we are. Errors by classification. Hmm, unanticipated? Oh, I can click it. Okay, the most common one is miles.“ - U30 MHCI PSLC Team | Summer 2005

  20. Learning Curves • Supporting Data: • Requirements Solicitation (Koedinger, Aleven, Ritter) • General Research Contextual Inquiries (U2) • LearnLab Research Contextual Inquiries (U13) • Goal: understand students’ performance over time particular knowledge components MHCI PSLC Team | Summer 2005

  21. Learning Curves/Problem Profiles • “Oh! I don’t need to see the [transaction table], it’s right here in the graph.” (-U28) • “I like being able to see the curves without punching in the formulas.” (-U1) • “I love that the a’s and the b’s come right up.” (-U36) • “I liked the Problem Profile. Leave it as it is.”(-U28) MHCI PSLC Team | Summer 2005

  22. Sample Selector • Supporting Data: • Contextual Inquiry Research: U3, U15 • Requirements Solicitation: Aleven, Koedinger • User Testing Pre-test Questions: U15, U16, U19-U24, U26, U28-U34, U36 • Goal: Define multiple groups of students and compare their performance throughout the standard reports within the Data Shop MHCI PSLC Team | Summer 2005

  23. Sample Selector • "so the only option I have is 'all students.' Ah, but I can edit this list.“ (U27) • “Oh, maybe I need to create a new sample.” (U31) • “making the samples was fairly easy.” (U29) MHCI PSLC Team | Summer 2005

  24. Data Export • Dave’s Goal is to: • Select a subset of his data • Export it to a file for further analysis • Supporting Data: • Requirements Solicitation (Koedinger, VanLehn) • General Research Contextual Inquiries (U5, U8) MHCI PSLC Team | Summer 2005

  25. Data Export Questions MHCI PSLC Team | Summer 2005

  26. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  27. Implementation plan Estimated Implementation Time (in Days) MHCI PSLC Team | Summer 2005

  28. Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

  29. Context Matters • Intimate Knowledge of Tutors Required • Dig a little deeper, right away MHCI PSLC Team | Summer 2005

  30. Inter-report Navigation • Reports are useful when they are connected. • Carry context between reports when possible. MHCI PSLC Team | Summer 2005

  31. Specialized Reports • More tailored than a stat package • Eliminates grunt work MHCI PSLC Team | Summer 2005

  32. Use Visual Communication • Get familiar with data quickly • Identify points of interest MHCI PSLC Team | Summer 2005

  33. Evolution of Error Report MHCI PSLC Team | Summer 2005

  34. Evolution of Error Report "I really don't understand what it [error report] means. When Ken was showing it to us [earlier in the day at the Data Shop demo] it made sense but on my own I wasn't sure." –User 13, Summer School MHCI PSLC Team | Summer 2005

  35. Evolution of Error Report "This wasn't very helpful... probably just the layout - it's hard to decipher….this is very difficult to read… I’m not sure what these errors mean." – User 13 MHCI PSLC Team | Summer 2005

  36. Evolution of Error Report MHCI PSLC Team | Summer 2005

  37. Evolution of Error Report MHCI PSLC Team | Summer 2005

  38. Evolution of Error Report MHCI PSLC Team | Summer 2005

  39. Evolution of Error Report MHCI PSLC Team | Summer 2005

  40. Acknowledgements Bob Kraut Andrea Knight Ken Koedinger Kurt Van Lehn Vincent Aleven Shipra Kayan Carolyn Rose Polo Chau Michael Bett Alida Skogsholm Peter Centgraf Braden Kowitz Bonnie John Ben Billings MHCI PSLC Team | Summer 2005

  41. The End “It’s like having my very own grad student!” – User 21 MHCI PSLC Team | Summer 2005

  42. Backup Slide: Data Visualization • Information Visualization (Card, 2003) says that Data Visualization improves cognition in 6 ways: • Increasing the memory and processing resources available to users • Reducing the search for information • Using visual representations to enhance the detection of patterns • Enabling perceptual inference operations • Using perceptual attention mechanisms for monitoring • Encoding information in a manipulable medium MHCI PSLC Team | Summer 2005

  43. Backup Slide - Error Report • Horizontal Stacked Bars • Option to take hints as errors of omission • Allows them to compare “down the line” • Error names fit better horizontally • Visualization provides better performance than tables MHCI PSLC Team | Summer 2005

  44. Problem Profile • Supporting Data: • General Research Contextual Inquiries (U1, U3, U7, U8, U9, U10) • LearnLab Research Contextual Inquiries (U11, U12) • Course Committee Survey (Chem) • Think Aloud Pilot (U2) • Goal: Understand students’ performance on a particular problem, and the problem’s context MHCI PSLC Team | Summer 2005

  45. Multi-Selection(in long lists) • A more standard method of indicating multiple-selection • Highlight helps users quickly spot which items are selected if scrolling the list. “…if there were check boxes on the side I would have known I could select more than one” – (U15) MHCI PSLC Team | Summer 2005

  46. “Scrubbing” A method to quickly compare across knowledge components “Oh interesting. It [the next curve] pops up.“ – (U35) “ooh…that's so cute….I'm going to click on that point to see why it jumped back up.” – (U36) MHCI PSLC Team | Summer 2005

  47. Why Not Just Add Condition? • Within-subject experiments require the capability for students to be assigned to multiple conditions • Sample Selector allows for multiple groupings based on individual researchers’ units of analysis MHCI PSLC Team | Summer 2005

  48. What About Behavior Characteristics? • The Sample Selector can build groups of students based on any characteristic in the database • Student behavior characteristics are not currently fields in the database • Once explicitly defined and included in the database, any behavior characteristic can be added and then used to build samples MHCI PSLC Team | Summer 2005

  49. Student Characteristics MHCI PSLC Team | Summer 2005

  50. Problem Characteristics MHCI PSLC Team | Summer 2005

More Related