Education research: Issues, sources and tools

1. Education research: Issues, sources and tools Prepared for the DLI Workshop Kingston, Ontario April 7, 2004 Maryanne Webber & François Nault Culture, Tourism and Centre for Education Statistics

2. Centre for Education Statistics • Mandate: deliver comprehensive program of national education statistics for policy development, educators and the public • Overarching conceptual framework • Close to 30 institutional surveys and household surveys • New directions in dissemination

3. Conceptual framework: work in progress Supply Demand Supply Demand Families E-S Organizations in the private and public sectors Labour Market Skills Market Individuals PSE Transition between markets Adult ed.

4. Conceptual framework: work in progress Supply Demand Supply Demand Families E-S Problematiques: labour market What is the level of skill among Canadians? How to facilitate the transition? How does skill influence life outcomes? What are the factors influencing skill gain and loss? Organizations in the private and public sectors Labour Market Skills Market Individuals PSE Transition between markets Adult ed. Problematiques: skills market How effective are learning systems in ensuring people have the necessary skills? How to increase high school graduation and access and persistence in pse? What are the most cost effective ways to achieve learning and skill development? Do all segments of society have equal opportunity? What can be done for those left behind? What is the role of ICT in education?

5. Types of surveys • Skills supply: Institutional surveys of enrolment, graduates, teaching staff and education finances; Information and Communication Technology in Schools Survey and Survey of Principals. • Skills demand: Post-secondary Education Participation Survey, Survey of Approaches to Education Planning, Adult Education and Training Survey. • Transition between markets: Youth in Transition Survey, National Graduates Survey. • Labour market supply: Estimates of educational attainment and activities of population from Census and Labour Force Survey. Direct assessment with the Adult Literacy and Lifeskills Survey.

6. Elementary/secondary institutional surveys • Description • Data on enrolments, graduates, finance and teaching staff • Collected from provincial/territorial governments, school boards and schools (for private schools) • Key strengths • Annual • Full coverage of K to 12 system • Census • Source of key indicators: graduation rates, cost per student, teacher-pupil ratio

7. Elementary/secondary institutional surveys • Limitations • Timeliness (at the moment) • Limited content • Some conceptual “grey zones” to be resolved

8. Cost per student at the e-s level, 1999-2000 Source: PCEIP

9. Student-educator ratio, 1999-2000 Source: PCEIP

10. ICT in School Survey • Description • Survey of ICT infrastructure (hardware & software) and use within elementary & secondary schools • Conducted in 2003 • Results in June 2004 • Sponsored by Schoolnet program • Key strengths • Census • Current data • Could serve as platform for further survey on impact

11. ICT in School Survey • Limitations • No information on impact • Response rate: 47%

12. Average number of students per school computer, 2000 Source: PISA

13. Principals’ Survey • Description • University of Montreal/SSHRC sponsored • Part of a large-scale project • Survey will address impact of change over past 10 years on role of principal • Survey will be conducted in the autumn of 2004 • Key strengths • Linkage to focus groups, other aspects of study • Possibility of linking to ICT in School Survey + other institutional data sources

14. Principals’ Survey • Limitations • Sample of about 6000 principals • Concern about burden due to number of surveys being done in schools • Also: http://crifpe.scedu.umontreal.ca/gtrc/

15. Enhanced Student Information System • Description • Data on enrolment and graduates by program • Will cover all public and not-for-profit PS institutions • Key strengths • Coherence – consistent results across PS institutions • Program information coded using Classification of Instructional Programs (CIP) • Data on student characteristics • Linked longitudinally, to follow student pathways through PSE

16. Enhanced Student Information System • Limitations • Implementation has been difficult, affecting timeliness • Privacy concerns (largely resolved) • For-profit postsecondary institutions not covered

17. Trends in Full-time Enrolment1985=100 Source: ESIS, CCSIS and USIS

18. Full-time Enrolment by Sex for 18 to 24 Year Olds Rate of Full-time Enrolment for 18 to 24 Year Olds Source: ESIS and USIS

19. Postsecondary faculty and finance surveys • Description • Full-time university faculty survey • Part-time university faculty survey • College faculty survey • University finance survey • Tuition and living accommodation survey • College finance survey • Strengths • Long-time series • Relevance

20. Postsecondary faculty and finance surveys • Limitations • Quality constraints • Difficult to compare across • PS universe (surveys not integrated)

22. Universities collected record-high revenues in 2001-02, and for the first time in 15 years, revenues from public sources increased at a faster rate than those from private sources. Constant 2001-02 billions dollars Revenues from public sources Revenues from private sources (including student fees)

23. Indices of full-time and part-time faculty and full-time equivalent students, 1990=100 Source: Part-time university and college academic staff survey

24. National Longitudinal Survey of Children and Youth • Description • Longitudinal sample of children selected from households. Original cohort 25,000 children aged 0-12 started in 1992, followed biennially to age 30. • Cross-sectional oversample of 0-5 year olds. • Key strengths • Data on broad array of motor, cognitive and social factors influencing the level and social distribution of outcomes. • Complete families selected to allow for analysis of intra-family effects. • Will address questions of impact of early experiences on outcomes later in life.

25. National Longitudinal Survey of Children and Youth • Limitations • Sample size limits ability to examine small sub-populations. • Household-based sample design precludes direct analysis of school effects.

26. Resilient Positive Development Long term Vulnerable While largely a transitory state, vulnerability touches a significant proportion of children 1994 1996 56.2% Not Vulnerable 14.9% 71.1% 71.9% 15.7% Vulnerable 28.1% 28.9% 13.2% Newly Vulnerable

27. Children with persistent low learning scores have characteristics associated with disadvantage Source : NLSCY, 1994-1995, 1996-1997, 1998-1999

28. Youth in Transition Survey • Description • Longitudinal survey of 15 year olds selected from within schools • 30,000 students, interviewed biennially starting in 2000 • Longitudinal survey of 30,000 18-20 year olds, biennial interviews to age 22-24 • Focus on social & educational factors that influence outcomes • Key strengths – 15 year olds • Link to PISA – direct skill assessment • Contextual information from student, school, home & neighbourhood

29. Youth in Transition Survey • Key strengths -- 18-20 year olds • Immediate data on factors influencing high school completion & transitions to PSE & labour market • Comparable to 1995 School Leaver Survey • Limitations • Sample sizes provide insufficient yield of some key sub-populations, such as those who eventually go on to graduate studies

30. Programme for International Student Assessment • Description • International school-based skill assessment of 15 year olds • In Canada, linked to 2000 YITS – same sample • PISA focus in 2000 was reading • New cohorts in 2003 (focus on math) & 2006 (science) • Key strengths • Direct proficiency measures • Repeated cohorts support trend analysis • Proficiency scale for reading skills can be linked to ALL • Can analyze impact of a range of school effects • International comparisons

31. Programme for International Student Assessment • Limitations • As in YITS

32. Average reading score 375 425 475 525 575 Alberta Finland British Columbia Quebec Canada Ontario Manitoba 95% Confidence interval Saskatchewan New Zealand Australia Ireland Korea United Kingdom Japan Nova Scotia Average score Prince Edward Island Newfoundland Sweden Austria Belgium Iceland Norway France United States New Brunswick Denmark Switzerland Spain Czech Republic Italy Germany Liechtenstein Hungary Poland Greece Portugal Russian Federation Latvia Luxembourg Mexico Brazil Canada rates near the top of the world in READING literacy Source: Programme for International Student Assessment, 2000.

33. 600 Finland 580 Canada 560 540 Japan 520 Reading achievement 500 480 460 United Kingdom France 440 United States Italy Canada Japan 420 Russian Federation Germany Finland 400 20 30 40 50 60 70 80 Socio-economic status Countries with high achievement in reading also have less variation in achievement by SES (based on PISA, 2000)

34. In Canada most of the variation in student reading performance is within schools Variation of performance within schools Variation of performance between schools Variation explained by institutional structures Source: PISA 2000

35. After controlling for parental SES-private school advantage disappears The apparent private school advantage Private Private Public Public Proportion of population Proportion of population 300 400 500 600 700 800 400 500 600 700 Reading achievement Reading achievement

36. Educational status of 18 to 20-year-olds no longer in high school

37. The literacy gap between disabled and non-disabled students varies across provinces, particularly for cognitively/emotionally disabled students No disability Physical disability Cognitive/ emotional disability

38. Immigrant students overall have lower literacy than other students, but this disadvantage disappears over time

39. Grade repetition and dropping out are much more frequent for aboriginal students than for other students, even after controlling for socio-economic status

40. Reading achievement of students in minority language schools

41. Survey of Approaches to Educational Planning • Description • First Statcan survey to collect detailed information on how Canadian parents prepare for their children’s postsecondary education • First conducted in 1999, collecting information on 20,000 children aged 18 and under • Conducted again in October 2002 • Key strengths • Single source for both financial (ex. current savings, expectations of costs) and non-financial (ex. parents’ expectations, practices concerning homework) parental factors that may affect child’s educational pathway

42. Survey of Approaches to Educational Planning • Limitations • Does not trace long-term outcomes.

43. For all income groups except the two highest, parents were most likely to expect around $10,000 in education savings (for parents who were “savers” when surveyed) Source: Survey of Approaches to Educational Planning, 2002

44. Media coverage • [National Post]Grants go to least needy: RESP survey: Parents' expectations too high, study finds • [The Vancouver Sun]Families saving more than ever for college education, but it's still not enough: Low-income families are unable to keep pace because federal program disproportionately favours wealthy, study says • [GLOBE AND MAIL]More parents dream of university for kids But few expect to be able to save enough to cover full cost of four-year degree

45. Postsecondary Education Participation Survey • Description • A new cross-sectional survey of 18-24 year olds (17-24 in Quebec) on postsecondary participation • Designed to provide indicators for the Canada Student Loans Program • Data collected February 2002 • Key strengths • Information on cost of attending a postsecondary program • Information on sources of revenue to pay for PSE (ex. student loans, family support, jobs)

46. Postsecondary Education Participation Survey • Limitations • Small sample. • Limited information on non-financial factors influencing postsecondary attendance.

47. Household income $

48. Access : No one factor fully accounts for PSE Participation 18-24 year-olds % Source: Post-secondary Education Participation Survey, 2002

49. Expenditures of CEGEP studentsvary less than those of college and university students Source: Post-secondary Education Participation Survey, 2002

50. National Graduate Survey • Description • Series of cohorts of postsecondary graduates designed to trace participation in advanced studies and labour market success two and five years after graduation • Samples are large enough to profile major fields of study by level and province • Recent cohorts include Class of 1995 followed up in 2000 and Class of 2000 followed up in 2002 • 1995 graduates who moved to the US followed up in 1997 and 2000