DesignWebs: Learning in Engineering Project Teams

1. DesignWebs: Learning in Engineering Project Teams Sharad Oberoi Carnegie Mellon University

2. Outline • Common issues in engg project teams • Need for DesignWebs • Background • Data source • Approach • Visionary scenario • Research Questions

3. Student Engineering Projects • Characteristics • Students bring together knowledge from different sources: • collaborate among themselves • share knowledge • negotiate with each other, faculty members and the client, in order to create engineering artifacts • Students reuse previous knowledge and create new knowledge within the context of the problem

4. Common Issues in Engg Project Teams • Knowledge isolation • Knowledge messiness • Transient team membership • Lack of knowledge synthesis

5. Collaborative Learning In Engg Projects • Goals • Develop an infrastructure that supports knowledge management for student engineering project teams • Assist members of engineering project teams to build and retain knowledge created through discussions, project documents and sharing of artifacts • Find evidence for learning in team communications constructed automatically from the project corpus

6. Designwebs • Knowledge is created in student interactions through multiple modalities: email, online discussion forums, collaborative documents, etc. • We visualize a knowledge management framework that: • Is shared: summarizing the team communications to track the emergence of the shared solution • Aids visualization & navigation: Enables the visualization and navigation of the ideas and their connections • Is searchable: Retrieves queries generated explicitly by the user or implicitly during browsing

8. Need for DesignWebs • DesignWebs will enable: • Within team knowledge-sharing • Across team knowledge-sharing • Global information access and integration

9. Background • Group memory systems • Scientometrics • Textual Analysis of Design Conversations • LDA Modeling • Cluster analysis • Co-word analysis

10. Group Memory Systems • Organizational knowledge • Semantic knowledge • Episodic knowledge • Group memory systems • act as a common repository to share information gathered by individuals or developed by the team • Issue Based Information Systems • Principle: “design process for complex problems is fundamentally a conversation among the stakeholders in which they bring their respective expertise and viewpoints to the resolution of design issues”

11. Scientometrics • Scientometrics • studies the cognitive as well as socio-organizational structures within a scientific community by analyzing the documents produced by that community • Uses bibliometrics to identify emerging research areas and the evolution of research areas

12. Latent Dirichlet Allocation (LDA) • LDA views every document as a mixture of various topics; each word in turn is attributable to one of the document's topics • A generative process that models each document as a mixture of topics, and models each topic as a multinomial distribution over words • Useful to model how collections of documents evolve over time

13. Cluster Analysis • Used to group text segments with the goal of maximizing intra-group similarity and minimizing inter-group similarity • Dimensionality reduction to derive useful representations of high dimensional data • Vector Space Model • suffers from the vocabulary mismatch problem • alternative techniques: LSA, Lexical Chaining and automatic discovery of vocabulary and thesauri

14. Co-word Analysis • Uses content analysis to map the strength of association between keywords in textual data • Basic principle: • it reduces a “space of descriptors (or keywords) to a set of network graphs” • These graphs do not display data like other statistical graphs, but construct multiple networks that highlight associations between keywords where possible

15. Background Review • Group memory systems: requirements • Scientometrics: modeling on large scale • Textual Analysis of Design Conversations • LDA Modeling: topic model • Cluster analysis: hierarchical clustering of topics • Co-word analysis: connections between terms

16. Capturing in-process data • We have 6 years of data from project classes that used the Kiva for team collaboration • Combines functions of e-mail, bboards and chat • Each year’s Kiva has hundreds of threads and thousands of posts and files

17. Capturing in-process data

18. Approach • DesignWebs • synthesize the knowledge from multiple sources of information used by team members during a project • enable users to see connections between the concepts used in an artifact at different levels • can reveal the current state of the project and the underlying structure of the artifact

19. Approach • Seed the DesignWeb with research papers • Extract the Latent Dirichlet Allocation topic model • Apply hierarchical clustering • Label clusters and extract phrases • Calculate connections between topics and terms • Co-word analysis • Make relevant documents accessible • Generate the web-based navigable DesignWeb

20. Visionary Scenario • A class of engineering graduate students are about to embark on a project involving an analysis of hybrid cars • At the start of such a class, students normally are asked to perform a literature review to understand the current state of the technology and the issues involved • To simulate a typical knowledge base at the start of such a project class, students were asked to perform a literature review from a corpus of 250 articles from Google Scholar

21. Demo

22. Challenges • Segment resolution • Levels of abstraction • Draft reports are often not well-structured • Alternate views for different users • Credibility of source • Identifying the structure of created knowledge, especially for different versions of the same document

23. Research Questions • Can the knowledge about an artifact be extracted and synthesized from multiple sources of information used and created by teams during an engineering project? • Can we identify evidence of the success/failure of a group from the team vocabulary? • Can we automatically detect inception and connectivity of key ideas in groups at various artifact milestones?

24. Research Question # 1 • Can student teams use the DesignWebs to navigate and search for artifact knowledge extracted and synthesized from multiple sources of information during an engineering project? • Perform user studies

25. Research Question # 2 • Can we identify evidence of the success/failure of a group from the team vocabulary? • Previous research has focused on convergence /divergence of vocabulary as sign of group cohesion • Advantage of having student data throughout the evolution of the design artifact • Churning vs. convergence • Assimilation of ideas within and across teams

26. Research Question # 3 • Can we automatically detect inception and connectivity of key ideas in groups at various artifact milestones? • Introduction of new ideas and propagation across teams • Data flow from end-to-end • Vocabulary at team boundaries • Can we detect detail?

27. Progress to date • Research Question # 1 • The first prototype of DesignWebs tool has been developed and some user studies carried out • The second prototype is currently being evaluated • Research Questions # 2 and 3 • Data already collected • Experiments will begin as soon as the prototype revison is completed

28. Questions?

29. Group Memory Systems • Drawbacks: • extra effort to document activities is perceived as having no immediate benefit • documentation is usually after-the-task and so any unsuccessful approaches are not documented • creators and eventual users may have different backgrounds and mental models of the knowledge structures, leading to difficulties in learning • emphasis in most group memory systems has been on the content aspect with little significance given to the process and context