Personnel Technological Proficiency

1. CDE’s Personnel Technological Proficiency Surveys: Assessing the Technological Proficiency of Teachers, Librarians/Media Specialists, and School Administrators Nazanin Mohajeri-Nelson, DeLilah Collins, George Thornton, and Uma Kedharnath Colorado Department of Education July 29, 2010

2. Personnel Technological Proficiency U.S. Department of Education – EDFacts Requirement • Title II, Part D of ESEA • Program performance reporting • Education Data Exchange Network (EDEN) - N071 • Personnel Skilled in Technology • The unduplicated headcount of school personnel assessed against the standards-based performance profiles of technology user skills as defined by the state. • Citation: • U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, N071 – Personnel Skilled in Technology (Headcount) File Specifications, Washington, D.C., 2010

3. Personnel Technological Proficiency Federal Requirement • Who must report data? • LEAs that receive Title II, Part D funds • Who has to be assessed? • All full-time school-based instructional personnel and school administrators • What methods of assessment are accepted? • Certifications • Endorsements • College Credits/Course Completion • Observations • Self-report checklist, test, or assessment (CDE option)

4. Personnel Technological Proficiency USDE’s definition of personnel • All teachers = Full-time professional school staff members who provide instruction to students and maintain daily student attendance records. • Librarians/Media Specialists = Professional staff members and supervisors assigned specific duties and school time for professional library and media service activities including selecting, acquiring, preparing, cataloging, and circulating books and other printed materials; planning the use of library and medial services by students, teachers and other members of the instructional staff; and guiding individuals in their use of media services and library materials. • School administrators = Staff members whose activities are concerned with directing and managing the operation of a particular school including principals, assistant principals, and other assistants; and persons who supervise school operations, assign duties to staff members, supervise and maintain records of the school, and coordinate school instructional activities with those of the LEA, including department chairpersons.

5. Personnel Technological Proficiency Historical Context • What CDE did to prepare the state • Collaboration with Technology Leadership Forum • Subcommittee – started on 5/9/08 • May 2009 learned that report due by 9/30/10 • Decision to include in HR/Sped • Many decisions already addressed in HR/Sped • - Job classification codes • - Full time as defined by the state (.85) • One year of data collection with option of “not yet assessed” • Feedback from TLF, CEA, and CoP • EDAC approval – September 2009 • HR/SPED data collection • Four Trainings + Webinar

6. Personnel Technological Proficiency Data Collection from Year 1 • Using District measures (if any available) • New Fields in the HR/Sped collection: • Technology Proficiency Level • Proficient, Not Proficient, Not assessed • Type of Assessment Used • Self assessment, performance-based assessment, peer assessment, supervisor assessment, multiple, and not assessed • Date of Technology Assessment • https://cdeapps.cde.state.co.us/appendix_hrdefs.htm

7. Personnel Technological Proficiency Year 1 Data ~ still being cleaned and prepared for analyses

8. Personnel Technological Proficiency Creation of the Assessment • International Society of Technology in Education (ISTE) Standards • National Education Technology Standards for Teachers (NETS·T) • National Education Technology Standards for Administrators (NETS·A) • Assessment Expert • Converted Evaluation Rubrics into Self-Assessment Surveys • Conducted Pilot Study to test reliability and validity of the surveys

9. Personnel Technological Proficiency Methods for Creating and Testing the Self-Assessments • Used ISTE standards to create 2 self-assessments • Teacher Survey [33  22] • Administrator Survey [48  27] • 5 cluster of questions aligned with the technology standards • Included demographics, list of definitions for terms within the assessments, and essential conditions • Pilot data collected online using SurveyMonkey • In addition to self-ratings on the survey, qualitative data on the survey (feedback) was collected from pilot study participants

10. Personnel Technological Proficiency Methods for Analyzing the Self-Assessments • Demographic data (e.g., years of experience, exposure to technology, school size and setting) • Reliability analyses • Relationships between self assessments and selected demographics • Qualitative analyses of the write-in comments

11. Personnel Technological Proficiency Pilot Study • Demographics of Respondents (Teacher Survey) • Job title (N=210) • Teachers = 172 • Librarians/Media Specialists = 18 • Technology Staff = 10 • Other (e.g., PD director) = 10 • Full time = 96% • Had some tech courses in past 5 years = 80% • Master’s degree = 66%

12. Personnel Technological Proficiency Pilot Study • Demographics of Respondents (Administrator Survey) • Job title (N=72) • Superintendents = 2 • Principals= 27 • Asst Principals= 3 • Manager/Directors (e.g., Sped, Instruction, Tech) = 15 • Program Coordinators = 8 • Others = 17 • Full time = 97% • Had taken tech courses in past 5 years = 86% • Master’s degree = 93%

13. Personnel Technological Proficiency Pilot Study Results • Reliability was moderate to high based on • Correlations between items within each cluster • Internal consistency (reliability coefficients above .86) • Change in coefficient alpha for a cluster if an item was removed • Means and Standard Deviations were used to determine if there was sufficient variability in responses (no ceiling effects, etc.).

14. Personnel Technological Proficiency Pilot Study Results Relationship between survey questions and demographic information was tested by regressing responses on demographic information. Low correlation

15. Personnel Technological Proficiency After Pilot Study • Survey items were revised based on • Frequency counts (how often a response was selected) • Range of responses (variability of responses to each item) • Results of reliability analyses • Qualitative feedback from survey participants

16. Personnel Technological Proficiency Subject Matter Expert Review • Revised survey questions were reviewed and critiqued by Subject Matter Experts to determine validity • Relevance to measuring tech proficiency • Relevance to tech standards • Importance of each item • Clarity and understandability of the items • Whether the item clusters sufficiently cover competency on that standard (additional questions needed?)

17. Personnel Technological Proficiency Questions in final form • Available on CDE website • Will be available as online survey in Fall

18. Personnel Technological Proficiency Online System for Data Collection • Personnel Technological Proficiency Self Assessment Survey • Production of the PTP Self Assessment Survey is underway • Survey will consists of Self assessment questions, definitions, and essential conditions questions • If LEA chooses to use this survey, this will eliminate the need to report proficiency data into the HR/SPED collection system • Survey tool should be available in the fall of 2010 • Survey tool will provide proficiency information at the end of the survey • Districts will have the ability to request data during the HR/SPED collection period to determine the number of staff members who have not taken the survey

19. Personnel Technological Proficiency Plans for the Data • Meet federal reporting requirements • Needs assessment for future PD plans

20. Personnel Technological Proficiency Alternatives to CDE Assessments • Districts can use many other options • District-made assessment tool • Technology endorsements • College transcripts • Performance on technology tasks • Supervisor evaluation (e.g., using rubrics from ISTE) • Peer evaluation

21. Personnel Technological Proficiency • Additional information regarding this reporting requirement, including the self-assessment survey questions, can be found at: • www.cde.state.co.us/edtech/PersonnelTechnologicalProficiency.htm

22. Personnel Technological Proficiency Personnel Technological Proficiency • Contacts: • Nazanin (Nazie) Mohajeri-Nelson ~ • mohajeri-nelson_n@cde.state.co.us • (303) 866-6205 • DeLilah Collins ~ • collins_d@cde.state.co.us • (303) 866-6850