440 likes | 554 Views
PIAAC = Programme for the International Assessment of Adult Competencies. Through the assessments and background questionnaire, PIAAC situated respondents within a continuum of life-long learning. From Whence We Came. . PISA.
E N D
PIAAC = Programme for the International Assessment of Adult Competencies
Through the assessments and background questionnaire, PIAAC situated respondents within a continuum of life-long learning
PISA • The OECD also coordinates the Programmefor International Student Assessment (PISA) • Every 3 years, starting 2000 • 15 year old students from randomly selected schools worldwide • Assessments in reading, mathematics and science • Students and principals complete background questionnaires looking at family background, approaches to learning and school administration • In some cases, parents also complete a questionnaire
What PIAAC Assessed • Literacy • Single scale measuring reading of prose, document, digital and mixed format texts • IALSS reported literacy as two separate domains (prose and document comprehension on two separate scales)
What PIAAC Assessed • Numeracy • Essentially the same concept of numeracy as used in IALSS • PIAAC, however, gathered significantly more data for the construction of its numeracy scale
What PIAAC Assessed • Problem Solving in a Technology Rich Environment (PS-TRE)
PS-TRE • Newly introduced domain • PIAAC broadens the concept of literacy by including digital technology, communication tools and networks to acquire and evaluate information, communicate with others, and perform practical tasks • At a basic level, it represents the intersection between computer literacy skills and the cognitive skills required to solve problems • PIAAC assessed adults capacity to use computer tools (keyboards and mice, file management tools, applications, and graphic interfaces) to assess the capacity of adults to access, process, evaluate and analyze information effectively
PIAAC employed adaptive testing – a feature of computer-based testing that adjusts the difficulty of questions as participants take the test • Provides a much more nuanced distinction in scores than were provided by paper-based predecessors
If respondents could not (or would not) use the computer based instrument, they were provided with a paper-based instrument that measured literacy and numeracy • Respondents could score at levels 4/5 in literacy and numeracy using the paper-based instrument
Main Elements of PIAAC Direct assessment (skills measured by the survey) • Literacy • Numeracy • Problem solving in technology-rich environments Background questionnaire (information about respondents) • Demographic characteristics • Educational attainment and training • Employment status and income • Social and linguistic background Module on skills use (information on how skills are used every day) • cognitive skills (e.g., engagement in reading) • non-cognitive skills (e.g., capacity to work in a team) and communications, organization, and influencing skills • workplace skills (e.g., autonomy over key aspects of work)
The Background • 157,000 adults 16-65 in 24 countries • Second round with nine additional countries • Third round proposed for 2014 • Language of assessment was the official language(s) of the countries participating • Sample sizes were determined nationally • Canada chose a larger sample size to ensure PIAAC data could be compared to IALSS
PIAAC Sample in Canada • Aboriginal status was self-declared, as determined through other Stats Canada reporting • Hours and hours of pan-Canadian conference calls and meetings were required to achieve this level of partipation • Conference calls and meetings continue as we start to work with the data • Confidentiality assured. The Ministry of Education will have access to the full data set and this was communicated to respondents
Computer Based Assessment • At 86%, Saskatchewan had the third highest proportion of the population (in the world) that engaged in computer based survey – after Sweden and the Netherlands. • The next highest proportion in Canada was Nova Scotia at 84%
Computer Based Assessment • Japan had a relatively low rate of completion via computer, BUT scored highly in literacy and numeracy • PS-TRE could not be measured by the PIAAC paper-based assessment instrument
Saskatchewan is … • At the OECD average in literacy • At the OECD average in PS-TRE • Below the OECD average in numeracy
Saskatchewan … • Saskatchewan’s overall performance mirrors Canada’s performance • Matches the OECD average for people at the highest proficiency levels • Has a higher proportion of the population at the lowest proficiency levels in numeracy and PS-TRE compared to the OECD average
Saskatchewan … • Had 10% of respondents whose first language was not English and French • Matches the OECD average, but is 13% below the Canadian average • Youth (16-24) is performing below the OECD average in all three domains • In PS-TRE, Saskatchewan youth at levels 2/3 is 38% vs. the OECD average of 51%
Nationally, educational achievement for both Aboriginal and non-Aboriginal people affects their overall literacy attainment in similar ways • Saskatchewan’s results do not show the same correlation, and we are starting to work with Manitoba and other pan-Canadian partners to try and understand why the data shows this • The gap between Aboriginal and non-Aboriginal populations in Saskatchewan (at a 26 and 35 score point difference in literacy and numeracy respectively), is larger than in Canada overall at 14 and 22 points.
Proficiency Levels - Literacy • Below Level 1 • 3% OECD, 4% Canada, 3% Saskatchewan • Level 1 • 12%, 13%, 14% • Level 2 • 34%, 32%, 33% • Level 3 • 39%, 38%, 39% • Level 4/5 • 12%, 14%, 11%
Proficiency Levels - Numeracy • Below Level 1 • 5% OECD, 6% Canada, 6% Saskatchewan • Level 1 • 14%, 17%, 18% • Level 2 • 33%, 32%, 33% • Level 3 • 35%, 33%, 33% • Level 4/5 • 13%, 12%, 11%
Proficiency Levels - PS-TRE • Non-respondents • 24% OECD, 19% Canada, 15% Saskatchewan • Below Level 1 • 12%, 15%, 18% • Level 1 • 29%, 30%, 35% • Level 2 • 28%, 29%, 28% • Level 3 • 6%, 7%, 5%
So Brett, how does our performance compare to IALSS? • (and, wink, wink, I know we’re doing worse)
PIAAC and IALSS • It is not possible to make a direct comparison of the results from IALSS and PIAAC; the scales used are not the same • The data from the prose, document and numeracy measures of IALSS have been rescaled at the national level to fit with the literacy measure used in PIAAC – these can be compared, but are still do not represent a 1 for 1 correlation • Level 3 in PIAAC is not equivalent to Level 3 in IALSS. In IALSS the OECD identified Level 3 as a minimum threshold of the skills needed to manage the demands of work and life. IN PIAAC, Level 3 was not defined the same way.
PIAAC and IALSS • The OECD designed the PIAAC survey to help jurisdictions determine the degree to which their skills address their needs. As a result, the OECD proposes that results for each jurisdiction should be assessed in terms of the association between skill levels and other characteristics, such as job, level of education, and income specific to the populations in each jurisdiction. • Does that mean all of our existing benchmark work is now thrown out? No – at this stage, the results from PIAAC have limited bearing on existing benchmarks. Much further analysis and research will be needed before the results can be applied to literacy programs and the measures we all use.
Overall, the difference between the performance of people from different backgrounds within jurisdictions is much larger than the difference between jurisdictions themselves • e.g., The gap between Canadians (and Saskatchewanians) at the 95th and 5th percentiles is larger than the gap between OECD countries on average
Skills in the labour force Literacy ― Proficiency of population aged 16 to 65 by employment status, OECD average and countries, 2012
Skills in the Labour Force • Contrary to what might be expected, Canada and the United States have an almost equal proportion of working people at levels 4/5 and level 1 and below • Italy’s and Korea’s performance better fits expectations
Immigrants Literacy — Average scores and score point differences of population aged 16 to 65 by immigrant status, OECD average and countries, 2012
Immigrants • Canada and Australia have very diverse populations with a range of educational backgrounds, but immigration status does not factor as highly in determining overall literacy attainment in our two countries as in others
The skills of young adults PS-TRE ― Proficiency of population aged 16 to 65 at Level 2 or 3 by age groups, OECD average and Canada, 2012
Youth • Do we need to look at new ways of knowledge transfer between older and younger workers? • How do employment opportunities (primarily p/t vs. primarily f/t affect attainment)? • Other questions will be forthcoming …
Where Does Canada Go Next? • A series of thematic reports • Education • Official Language Minority Communities • Labour Market • Social Outcomes • Aboriginal Populations • New Canadians • PIAAC Online
Together • WE can look at building richer programs and services for individuals, families, employees and communities • WE can support individuals in a life long learning continuum that is reflective of their changing circumstances • WE can honour the holistic teachings of the communities we have all worked with
www.piaac.ca http://www.oecd.org/site/piaac/ http://piaacdataexplorer.oecd.org/ide/idepiaac/
Questions? Brett Waytuck Provincial Librarian / Executive Director Provincial Library & Literacy Office brett.waytuck@gov.sk.ca