290 likes | 505 Views
ITS Data Collection Framework. Capturing data based on agent communication standard Olga Medvedeva , Center for Pathology Informatics, University of Pittsburgh. Outline. Need for communication standard for Intelligent Tutoring Systems Existing standard for multi-agent communication
E N D
ITS Data Collection Framework Capturing data based on agent communication standard Olga Medvedeva, Center for Pathology Informatics, University of Pittsburgh Educational Data Mining Workshop 20th AAAI-05 Conference
Outline • Need for communication standard for Intelligent Tutoring Systems • Existing standard for multi-agent communication • Implementation in SlideTutor • Communication protocol • Data collection • Database query tool • Lessons learned • Comparison with recent standardization effort • Advantages of using the the existing standard Educational Data Mining Workshop 20th AAAI-05 Conference
Intelligent Learning Environment Common Base • Underlying theory • Cognitive tutors (Anderson et al. 1995) • Adaptive hypermedia (Brusilovsky et al. 1996) • Constraint-based (Mitrovic et al. 2001) • Modules • Expert, Student, Interface, Pedagogic • “Single-purpose” development approach Educational Data Mining Workshop 20th AAAI-05 Conference
Keystone – communication standard • Previous efforts: • Inter-tutor communication (Ritter, Koedinger 1996; Brusilovsky et al. 1997) one-to-one translators, strict channel, no real protocol • Shared resources (Koedinger et al. 1999) – limited use: lack of standard • DORMIN protocol (developed at CMU) – used in commercial product • Our approach • Multi-agent technology • Use existing inter-agent communication standard Educational Data Mining Workshop 20th AAAI-05 Conference
Foundation for Intelligent Physical Agents (FIPA) FIPA (www.fipa.org) - collection of standards for inter-agent communication: • Agent Management System – manages an agent life-cycle, maintains a registry with unique Agent Identifier (AID) • Transport – describes message exchange protocol: transport type and specific address for an agent • Agent Communication Language (ACL) –communication specifications FIPA was officially accepted by the IEEE as one of its standards committees on 8 June 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA Design Principals • Forms abstract basis for concrete architecture • Sets minimum required elements • Permits introduction of new elements • Permits arbitrary content language, uses Abstract Content Representation (ACR) for ACL as key-value pairs Envelope: Sender (locator) Receiver (locator) Timestamp Message (ACL): Sender (AID) Receiver (AID) Performative (String) Content: ( ACR) Reply-to(Message ID) Message (ACL): Sender (AID) Receiver (AID) Performative (String) Content: ( ACR) Reply-to (Message ID) Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA ACL Message Structure :sender – identity of the sender :receiver – identity of the recipient :content – the object of the action :performative – the type of the communicative act Optional: :reply-with:replay-to:in-replay-to:replay-by– replay constraints :language – encoding schema of the content of the message :encoding – encoding identifier :ontology – is used to give a meaning to symbols/concepts in the content :protocol – gives additional context for the interpretation of the message :conversation-id – identifies the ongoing sequence of communicative act, manages the conversation strategies Educational Data Mining Workshop 20th AAAI-05 Conference
Accept-proposal Agree Cancel Call-for-proposal Confirm Disconfirm Failure Inform Inform-if Inform-ref Not-understood Propagate Propose Proxy Query-if Query-ref Refuse Reject-proposal Request Request-when Request-whenever Subscribe FIPA Performatives Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA Implementation in Java • Java Agent Services (JAS) (www.jcp.org) defines a set of objects and service interfaces to support the deployment and operation of the agents. • Contains interfaces for building messages, directory services and a factory for message transfer services. • JAS is a base for multi-agent communication in our system Educational Data Mining Workshop 20th AAAI-05 Conference
SlideTutor Architecturehttp://slidetutor.upmc.edu SlideTutor - an agent-based model tracing ITS for visual classification problem solving insurgical pathology Educational Data Mining Workshop 20th AAAI-05 Conference
Educational Data Mining Workshop 20th AAAI-05 Conference
Generic Representation of Problem-Solving Space Educational Data Mining Workshop 20th AAAI-05 Conference
Collected Data • InterfaceEvent – low-level human-computer interactions • ClientEvent – collection of InterfaceEvents that represents an elementary subgoal, understood by tutor • TutorResponse – system response to a ClientEvent Educational Data Mining Workshop 20th AAAI-05 Conference
Message Example ClientEvent • Envelope indicates the locators of client and protocol agents • 4 required key-value pairs for a message • Performative defines a type of communicative act • List of preceding InterfaceEvent Ids: • click on Finding button • Click on image • Selecting 3 times down a tree of findings Envelope: Sender: Client_1 Receiver: PROTOCOL TimeStamp = 1114444377783 Message: Sender: Concept2 Receiver: PROTOCOL Performative: X-Created In-reply-with: 1114444378242 Content: Type = Finding Label = blister Id = Concept2 ObjectDescription = Finding.blister.Concept2 Parent = null Input: name = text value = blister name = y value = 11808 name = x value = 38048 name = z value = 0.03 InterfaceEventIDS = [1114444374333, 1114444375546, 1114444376304, 1114444376798, 1114444377444] Educational Data Mining Workshop 20th AAAI-05 Conference
Message in Depth ClientEvent • Widget object (agent) description parameters • Type(“Button”, “Finding”) • Label(“Next”, “Blister”) • Id – unique within a session • ObjectDescription – combination of Type+Label+Id (“Finding.blister.Concept2) • Parent – list of all parent ObjectDescriptions for hierarchical structures • Common for ITS user action triplet • Action = Performative • Selection = ObjectDescription+Parent • Input = list form Content Input • Message encoded in XML is easy to translate into other languages (RDF, KIF, SL, etc.) Envelope: Sender: Client_1 Receiver: PROTOCOL TimeStamp = 1114444377783 Message: Sender: Concept2 Receiver: PROTOCOL Performative: X-Created In-reply-with: 1114444378242 Content: Type = Finding Label = blister Id = Concept2 ObjectDescription = Finding.blister.Concept2 Parent = null Input: name = text value = blister name = y value = 11808 name = x value = 38048 name = z value = 0.03 InterfaceEventIDS = [1114444374333, 1114444375546, 1114444376304, 1114444376798, 1114444377444] Educational Data Mining Workshop 20th AAAI-05 Conference
TutorResponse Example • Student performance data • Performative: FAILURE – user took incorrect step • ErrorCode = 15 – user incorrectly located existing finding • Input: - contains a description of an error message to be presented to user • Tutor performance data • Best possible next step – action expert model would take in this problem state Envelope Sender: TutorEngine0 Receiver: PROTOCOL TimeStamp: 1114444379378 Message: Sender: TutorEngine0 Receiver: PROTOCOL Performative: FAILURE Conversation_ID: 1114444378242 Content: ErrorCode = 15 NextStepType = Evidence NextStepLabel = blister NextStepID = Concept2 NextStepParent = null NextStepAction = DELETE Input: name = Messages value = "[TEXT:There is BLISTER present, but not where you have pointed in the image. See if you can find where. POINTERS:[PointTo:Concept2 IsPermanent:false Method:setFlash Args:[true]]]“ name= TutorAction value = "PointTo:Concept2 IsPermanent:false Method:setBackgroundColor Args:[RED]" Educational Data Mining Workshop 20th AAAI-05 Conference
Database Schema • High-level static tables similar to Mostow et al. 2002 contains • Experiment, CaseList, Student, etc. • Low-level tables for captured events, including start/end of problem and • session closely follow the FIPA standard, generic with any number of • event parameters stored in corresponding Input tables Educational Data Mining Workshop 20th AAAI-05 Conference
Web-Based Protocol Query Tool • Allows the user to obtain data sets specific to a wide range of constraints • Outputs to HTML file that can be transferred to Excel • Query can be saved and viewed in SQL • Interface, Client and Tutor events data can be joined in different ways Educational Data Mining Workshop 20th AAAI-05 Conference
Query Tool Results for Identifying Blister InterfaceEvents ClientEvents TutorResponses Educational Data Mining Workshop 20th AAAI-05 Conference
Advantages of Event-Based Data Representation • Usability Perspective: InterfaceEvents linked to ClientEvents (Saadawi et al. 2005) • How many actions were performed • How much time was required to achieve a particular subgoal, such as identification of Blister • How many InterfaceEvents were unrelated to any ClientEvent • Student Performance over time: ClientEvents linked to TutorResponses • Number of hints requested • Depth of hints • Error frequency and distribution • Tutor Performance:NextStep fields in TutorResponses • Compare next student actions to those predicted by tutor Educational Data Mining Workshop 20th AAAI-05 Conference
SlideTutor Data Sharing Limitation • This paper and presentation have been approved by Institutional Review Board (IRB) • Researcher needs to sign a Limited Use Agreement • There might be one agreement with consortiums Educational Data Mining Workshop 20th AAAI-05 Conference
Lessons Learned For the past year our data collection framework was used in 4 small HCI studies and one large experiment with a total of 50 students. • Keep data clean: ended up maintaining ‘raw’ and ‘clean’ copies of database • Granularity of captured data: capturing of detailed data slows the system • Separate database for assessment: no explicit mapping of performance on tests and in the tutoring system Educational Data Mining Workshop 20th AAAI-05 Conference
Data Collection Framework Advantages • Advantages of relational database (Mostow et al. 2002) • Eases the analysis of the enormous volume of complex data • Generic framework that might be adapted to other model-tracing ITS • Adapted in the extension of SlideTutor – ReportTutor that teaches how to write the pathology reports • Flexibility of FIPA-based communication protocol • Flexibility to describe interaction events • Extendable set of performatives • Multiple messages in one envelope, unrestricted number of input parameters • Potential to reference ontologies within the message • Can be easily reused in the Data Shop Educational Data Mining Workshop 20th AAAI-05 Conference
Data Shop Project, Pittsburgh Science of Learning Center (http://www.learnlab.org ) • Logging and Analysis: Tools and reports to aid PSLC researchers and course developers • Log the activities of the experiments to a database • Provide the reports and queries on that experiments • Goal: Standardize the messaging format among tools, tutoring translators and agents • Message types: tool_message, tutor_message, curriculum_message, message • Data Shop Tutor Logging v3 released in June 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
meta (0 or 1) user_id session_id time time_zone tool_message attempt_id event_descriptor (0+) event_id selection (0+) id type action (0+) id problem_name (0 or 1) semantic_event id semantic_event_id name trigger input (0+) id (1+) step (0+) probability ui_event id Data Shop Tool Message and SlideTutor Interface/Client events Educational Data Mining Workshop 20th AAAI-05 Conference
meta (0 or 1) user_id session_id time time_zone problem_name (0 or 1) semantic_event id semantic_event_id name trigger ui_event id action_evaluation (0+) current_hint_number total_hints_available classification event_descriptor (0+) event_id step (0+) probability tutor_advice (0+) selection (0+) id type skill (0+) probability production (0+) step_interpretation (0+) action (0+) id input (0+) id custom_field (0+) name (1) name (1) value (1) value (1) Data Shop Tutor Message and SlideTutor TutorResponse Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA Advantages • FIPA as a information exchange underlying standard • Develop a set of performatives – a controlled vocabulary for ITS communication • Create sharable ontologies for domain knowledge, hint content, error categories and use ‘:ontology’ FIPA parameter to give a meaning to the message content • Use ‘:protocol’ parameter to identify the translator and to preserve the internal component structure • Syntactically aligned systems • Ease meta-analysis for tutors with the identical performatives • Reuse data for simulations • Shared services for real-time interoperability • Identifying particular help-seeking behavior • Calculating knowledge tracing probabilities Educational Data Mining Workshop 20th AAAI-05 Conference
Acknowledgements Grants: • National Library of Medicine • National Cancer Institute People: • Rebecca Crowley • Girish Chavan • Eugene Tseytlin • Elizabeth Legowski • Katsura Fujita • Maria Bond Educational Data Mining Workshop 20th AAAI-05 Conference
References • Anderson JR, Corbett AT, Koedinger KR, and Pelletier R. Cognitive Tutors: Lessons learned. Journal of the Learning Sciences 4(2): 167-207, 1995 • Brusilovsky, P., Kommers, P. & Streitz, N. (Eds.) (1996) Multimedia, Hypermedia, and Virtual Reality (LNCS Vol. 1077). Berlin: Springer-Verlag, 1996 • Mitrovic A, Mayo M, Suraweera, P and Martin, B. Constraint-Based Tutors: A Success Story. In Monostori, L. and Vancza, J. (Eds). Proceedings of the 14th International Conference on Industrial & Engineering Applications of Artificial Intelligence and Expert Systems, Budapest, Hungary, Springer, pp. 931-940, 2001 • Ritter, S. and Koedinger, K. R. (1996). An architecture for plug-in tutor agents. Journal of Artificial Intelligence in Education, 7, 315-347 • Brusilovsky, P., Ritter, S., & Schwarz, E. Distributed intelligent tutoring on the Web, Proceedings of AIEDâ97, the Eighth World Conference on Artificial Intelligence in Education. 1997 • Koedinger KR, Suthers DD, & Forbus KD. Component-based construction of a science learning space: A model and feasibility demonstration. International Journal of Artificial Intelligence in Education: 10, 392-31, 1999 • Mostow J, Beck J, Chalasani R, Cuneo A, and Jia P.Viewing and Analyzing Multimodal Human-computer Tutorial Dialogue: A Database Approach. Proceedings of the ITS 2002 Workshop on Empirical Methods for Tutorial Dialogue Systems, 75-84 • Saadawi G, Legowski E, Medvedeva O, Chavan G, and Crowley RS. A method for automated detection of usability problems from client user interface events. Accepted to Proceedings of the American Medical Informatics Association Symposium 2005 Educational Data Mining Workshop 20th AAAI-05 Conference