Peter Jones, University of California, Irvine Anne Jones, University of California, Riverside

1. Teacher Education Integrated Information System (TEIIS)ConceptualizationImplementationResearch PotentialPresented at AERA, April 2011

2. Blueprint for Success: Conceptualizing Teacher Education Information Systems and Maximizing Resources to Improve Teacher Preparation Peter Jones, University of California, Irvine Anne Jones, University of California, Riverside

3. Objectives • The design and implementation of an integrated information system that addresses accountability issues • Describe how this system uses a common structure that can be adopted across programs and campuses • Describe the underlying conceptual construct

4. Theoretical Framework • MIS technologies are widely used by academic institutions to manage student records but • It is rare to find examinations of the adoption of these systems by teacher preparation programs • A recent national study found that 50% of teacher preparation programs surveyed had not moved beyond the use of paper-based systems and • Only 10% of the sample were using proprietary systems that were designed to meet their specific needs (Keil & Haughton, 2009)

5. Theoretical Framework (cont) • Most teacher preparation programs in the United States are organized around a common structure and practices: Candidates go through admissions processes, content methodology training, and placement in school settings. Candidates observe and record student behavior. Supervisors observe and record candidate behavior. • As a consequence, these activities are reflected in teacher preparation program accreditation standards. Accrediting institutions across the country are beginning to require that these programs systematically implement comprehensive data collection systems .

6. System Modules

7. Admissions • Applicant • Applications • Program • Reviewer • Reviews • Documents • Communication

8. Administration • Administrators • Students • Supervisors • Mentors • Placements • Observations • Schools • Communication • Credentials

9. Evaluations • Candidate Competence • Supervisor Observations • Mentor Candidate Evaluation • Supervisor Candidate Evaluation • Teaching Performance Assessment (PACT) • Program Evaluation • Candidate Mentor Evaluation • Candidate Supervisor Evaluation • Candidate Post Program Evaluation • Candidate Employer Post Program Evaluation

10. Performance Assessment for California Teachers (PACT) • Assessors • Calibration • Assessments • Predictions • Candidate Demographics

11. TEIIS Online Portals • Admissions • Applicant • Reviewer • Administration • Student • Supervisor • Mentor • PACT • Assessors • Student • Graduate / Alumni • Graduates

12. TEIIS Flow Chart

13. Evidence of Excellence: Into, Through, and Beyond Accreditation Standards Anne Jones, University of California, Riverside Peter Jones, University of California, Irvine

14. Objectives • Teacher preparation programs are more aware than ever of the importance of implementing an information system to collect and organize data . • The TEIIS data management system has been aligned with State accreditation requirements. The power of this system, however, lies in the institutional flexibility to extend this capacity to meet local and regional reporting requirements and national standards. • The overarching goal is to enable the institution to efficiently adapt to new and revised standards, as well as to inform continuous improvement and support a targeted research agenda.

15. Theoretical Framework • Three major governing documents describe operating requirements and outcome objectives for teacher preparation programs in California. They are the overarching California Teacher Preparation Program Common Standards, the new Preliminary Credential Program Standards adopted in January of 2009, and the Accreditation Framework for Educator Preparation in California • Common Standard 2: Unit and Program Evaluation System states: “The education unit implements an assessment system for ongoing program and unit evaluation and improvement. The system collects, analyzes and utilizes data on candidate and program completer performance and unit operations. Assessment in all programs includes ongoing and comprehensive data collection related to candidate qualifications, proficiencies, competence, and program effectiveness. Data are analyzed to identify patterns and trends that serve as the basis for programmatic and unit decision-making” (California Commission on Teacher Credentialing, 2008)

16. Theoretical Framework (cont) • If our aim as a profession is to promote the perception of teachers and teaching as a model similar to that of engineering, medicine or other highly specialized professions, then mere accreditation compliance is not an acceptable goal. • We need to focus on a research agenda informed by qualitative and quantitative data that reveal the strengths and areas needing improvement within each program. • This can be achieved through a careful analysis of the components of the data management system, identifying the items that provide evidence that accreditation requirements are being met, and then using the available data to formulate and answer appropriate research questions.

17. Methods and Data Key data sources in the assessment and evaluation module include: •Instruments that measure candidate competency (1) Supervisor observation evaluations/report (qualitative) (2) Supervisor teacher-candidate evaluation (quantitative and qualitative) (3) Mentor teacher teacher-candidate evaluation (quantitative and qualitative) (4) Performance Assessment for California Teachers (PACT) `(quantitative and qualitative)

18. Methods and Data (cont) •Instruments that measure program effectiveness (1)  Teacher-candidate mentor teacher evaluation (quantitative and qualitative) (2) Teacher-candidate supervisor evaluation (quantitative and qualitative) (3) Teacher-candidate program evaluation (quantitative and qualitative)

19. Methods and Data (cont) •Two post-program evaluations (1)  1 Year Teacher Alumni Survey (quantitative and qualitative) (2) 1 Year Teacher Supervisor Survey (quantitative and qualitative)

20. Methods and Data (cont) • All the modules were originally designed in order to more efficiently manage the credential programs and track students’ progress and to be able to quickly and flexible retrieve key information. • Data are then analyzed to identify patterns and trends that serve as the basis for programmatic and unit decision-making. As California adopted new and updated standards and implemented a new accreditation system, component pieces in the system were identified for their suitability to meet the standards and support accreditation requirements. • As accreditation requirements evolve, the data collected are compared with the standards to reveal areas of coverage, redundancy, and omission. This allows for flexibility without compromising the individual institutions’ capacity to focus on its articulated mission or specific research agenda.

21. Results and Significance • Wilkins et. al (Wilkins, Young, & Sterner, 2009) found that analyzing data for the purposes of accountability and accreditation resulted in the identification of similar problems or areas of need and the proposal of common solutions across teacher preparation institutions nationally, regardless of their size or type. • This information supports the notion that common instruments can be used to support accreditation, inform program improvement, and supporting flexible and efficient adaptation to new or revised standards . • One of the ways in which teacher preparation program data can be leveraged is to facilitate transparent operation in the community by identifying what needs its candidates are best – and least – prepared to meet.

22. Results and Significance (cont) • For those states that allow flexibility in choosing a national accreditation option, these data can be used to inform decisions about which model(s) may be most useful for the institution to adopt. • For example, programs that are committed to using data to inform ongoing programmatic changes may find that there is a natural alignment with a continuous improvement approach, while another institution that regularly uses the data to conduct research may be best suited to an inquiry-based model. • In either instance data used to meet any local accreditation requirements can be leveraged to support a goal of national accreditation.

23. Results and Significance (cont) •As the state of the teaching profession continues to evolve, institutions that systematically implement data driven management and assessment systems have the potential to become models for other teacher preparation programs, especially regional institutions that serve a similar constituency (Reusser, Butler, Symonds, Vetter, & Wall, 2007). • It therefore behooves us as a profession to identify those common elements that can inform a data management system which will support and fulfill accreditation requirements, but whose greater value lies in its potential to facilitate research that improves both teacher training and teacher performance.

24. Mining the Evidence: What Can Academic Audit Data Tell Us About the Teaching Profession? Peter Jones, University of California, Irvine George Farkas, University of California, Irvine Anne Jones, University of California, Riverside

25. Objectives • National and state teacher preparation program accreditation standards have been under review. • Many more demands on schools and colleges of education to meet more rigorous sets of standards have been legislated in the past few years. • California, for example, has completely redesigned their teacher preparation program accreditation process and now requires institutions collect data systematically, review the data annually, and submit formal reports biennially.

26. Objectives (cont) •Our goal is to suggest ways in which the data available to teacher preparation programs can be used to address a number of key questions about the functioning and outcomes of these programs. •Examples of pertinent questions are: 1) Who applies to teacher preparation programs? 2) What are the distinguishing characteristics of those applicants that are most likely to be accepted? 3) Of those accepted, which students actually enroll in the program? 4) Of those who enroll, what are the characteristics of students that do not successfully complete the program? 5) Which program elements are the most and the least valuable in preparing students for careers as teachers? 6) Which aspects of student performance within the program are most predictive of students’ success in their first year as teachers? And, how do the answers to the above questions evolve over time?

27. Theoretical Framework • There are no agreed upon constructs for what constitutes teacher quality (!) • Using student achievement data in an attempt to measure this unclear concept is not an option for those attempting to determine whether a pre-service teacher is prepared adequately. • What data are available to teacher preparation programs tend to be pre-program academic tests and grades (admissions measurements), teacher preparation program course grades, assessments conducted by supervisors and cooperating teachers which include observations of student teaching performance and fieldwork interactions and, as of late, summative teaching performance assessments designed to measure the beginning teacher’s ability to design, plan for, and then conduct a series of lessons.

28. Methods and Data • To begin to explore these questions we have taken a 6-year sample of historical data from the data table used to collect the results of the Performance Assessment for California Teachers (PACT) and applied basic statistical procedures, including regression analysis, to explore some of the questions.

29. Results Table 1 Percent of Pre-Service Teacher Candidates Passing Teacher Performance Assessment (TPA) by Year

30. Results and Significance (cont) Table 2 Percent of Pre-Service Teacher Candidates Passing Teacher Performance Assessment (TPA) by Program

31. Results and Significance (cont) Table 3 Percent of Pre-Service Teacher Candidates Passing Teacher Performance Assessment (TPA) by Subject

32. Results and Significance (cont) Table 4 Percent of Pre-Service Teacher Candidates by Gender and Subject

33. Results and Significance (cont) Table 5 Percent of Pre-Service Teacher Candidates by Age and Subject

34. Results and Significance (cont) Table 6A Percent of Pre-Service Teacher Candidates by Mean BA GPA and Subject Table 6B Percent of Pre-Service Teacher Candidates by BA GPA Category and Subject

35. Results and Significance (cont) Table 7 Regression Analysis of Pre-Service Teacher Candidates Average Teacher Performance Assessment Scores ** P>|t| = <=.01 ** P>|t| = <=.05

36. Results and Significance (cont) Table 8 Logistic Regression Analysis of Pre-Service Teacher Candidates Passing the Teacher Performance Assessment ** P>|t| = <=.01 ** P>|t| = <=.05

37. Results and Significance (cont) Percent of Pre-Service Teacher Candidates by Gender

38. Results and Significance (cont) Percent of Pre-Service Teacher Candidates by Age Percent of Pre-Service Teacher Candidates by BA GPA

39. Summary and Significance • The research questions posed at the beginning of this paper are all common questions any teacher preparation program would find salient. • Unfortunately, many programs are not well equipped to explore them in more than an anecdotal way due to their administrative structure. • In far too many cases data are collected in paper forms making any analysis very expensive and cumbersome to provide. When data are collected electronically many times the data are stored in a format, such as a spreadsheet, that does not allow internal relationships to be established, thus limiting the data analysis to the unit-of-analysis of the underlying electronic format.

40. Review of Structure Teacher Education Integrated Information System (TEIIS) Flow Chart

41. Thank You! Questions…? Anne Jones anne.jones@ucr.edu Peter Jones jonesp@uci.edu George Farkasgfarkas@uci.edu