ONR/NETC Planning Meeting 18 July, 2003 UCLA/CRESST, Los Angeles, CA

1. ONR Advanced Distributed Learning Intellectual and Social Capital: Opportunities Eva L. Baker ONR/NETC Planning Meeting 18 July, 2003 UCLA/CRESST, Los Angeles, CA UCLA CRESST National Center for Research on Evaluation, Standards, and Student Testing 2003 Regents of the University of California

2. Intellectual Capital • Fund of knowledge, skills, and cognitive propensities—analogous to ROI • CRESST focus for ONR • Content understanding • Problem solving • Teamwork • Communication • Metacognition

3. Intellectual Capital Content Understanding Learning Teamwork and Collaboration Problem Solving Communication Metacognition

4. Social Capital • Team performance and skills • Networking and information sharing • Trust • Efficacy (can do) • Adaptive communication • Environmental stress management • Leadership roles • Effort

5. Organizational Development and the Creation of Social Capital • Both intellectual and social capital needed • Social capital is the sizzle • Social capital interacts with individual differences • Individual differences in the past has been conceived as general traits or traits that come into play on certain occasions

6. Social Capital Trust Leadership LearningOrganization Networks Efficacy Adaptive Communication Effort Teamwork

7. Measurement of Social Capital • In institutions (e.g., schools), as a component of effectiveness • In programs, as the context and outcome of quality instruction • In organizations, to improve effectiveness

8. What’s New? • Not trait-, state-, personality-focused • Componential and therefore instruction-sensitive • Interactive with intellectual capital formation • Measurement should be iterative and focus on bottom-up cycles of reporting • Organizations put self forward like awardees

9. Program • Design analytic framework for evaluating social capital, development, duration, predictive elements • Contrast high-functioning with low-functioning systems to discriminate key predictors • Create measures for self-assessment and improvement of individuals, teams, and organizations

10. Components of Social Capital • Individual behaviors (motivated, persistent, resilient, altruistic) • Individual meta processes (intellectual and emotional awareness and control) • Knowledge of how to make social systems work

11. Opportunities • Measurement of Team Performance • Knowledge maps for team cognition • Content knowledge • Team process measured by pre-specified messages or online text analysis, e.g., latent semantic analysis E-Rater (ETS) • Automated generated and scored After-Action Reviews

12. Opportunities(cont.) • Dynamic Testing of Team Performance • Vygotsky (ZPD), feedback (O’Neil), help seeking (Webb) • Cognitive Demands of Psychomotor Tasks (rifle marksmanship and SEALS) • Brief SEALS August 14, 2003, for possible joint program

13. Opportunities(cont.) • Comparative Testing of Authoring Systems • Top Down • Ontology, task analysis, experts • Bottom Up • “Repurposing” simulations • Dependent measures • Outcome of instruction is learning • Outcome of assessment is reliable and valid decisions • Online Analysis and Reporting

14. Psychometrics of Simulation Individual Training Collective Training • Psychometric Issues • Validity and accuracy • Design economies • Technical comparability and equating • Estimation of difficulty • Generalization and characteristics across topics, contexts, prior knowledge • Sampling • Scaling • New functional approachesto norming • Online administration, diagnostic/ diagnostic/scoring, reportingprescriptive prescriptive (AAR)

15. Opportunities(cont.)Standardizing Quality • Alignment (R&D) • Linguistics • Search capacity • Evaluation (program assessment) • 5 Vector mixed models