Programme for International Student Assessment

Data from international standardised assessments can be useful in research on causal factors within or across education systems.[4] Mons notes that the databases generated by large-scale international assessments have made it possible to carry out inventories and comparisons of education systems on an unprecedented scale* on themes ranging from the conditions for learning mathematics and reading, to institutional autonomy and admissions policies.[6] They allow typologies to be developed that can be used for comparative statistical analyses of education performance indicators, thereby identifying the consequences of different policy choices. They have generated new knowledge about education: PISA findings have challenged deeply embedded educational practices, such as the early tracking of students into vocational or academic pathways.[7]

• 79 countries and economies participated in the 2018 data collection.

Barroso and de Carvalho find that PISA provides a common reference connecting academic research in education and the political realm of public policy, operating as a mediator between different strands of knowledge from the realm of education and public policy.[8] However, although the key findings from comparative assessments are widely shared in the research community[4] the knowledge they create does not necessarily fit with government reform agendas; this leads to some inappropriate uses of assessment data.

Emerging research suggests that international standardised assessments are having an impact on national assessment policy and practice. PISA is being integrated into national policies and practices on assessment, evaluation, curriculum standards and performance targets; its assessment frameworks and instruments are being used as best-practice models for improving national assessments; many countries have explicitly incorporated and emphasise PISA-like competencies in revised national standards and curricula; others use PISA data to complement national data and validate national results against an international benchmark.[7]

More important than its influence on countries' policy of student assessment, is the range of ways in which PISA is influencing countries education policy choices.

Policy-makers in most participating countries see PISA as an important indicator of system performance; PISA reports can define policy problems and set the agenda for national policy debate; policymakers seem to accept PISA as a valid and reliable instrument for internationally benchmarking system performance and changes over time; most countries—irrespective of whether they performed above, at, or below the average PISA score—have begun policy reforms in response to PISA reports.[7]

Against this, impact on national education systems varies markedly. For example, in Germany, the results of the first PISA assessment caused the so-called 'PISA shock': a questioning of previously accepted educational policies; in a state marked by jealously guarded regional policy differences, it led ultimately to an agreement by all Länder to introduce common national standards and even an institutionalised structure to ensure that they were observed.[9] In Hungary, by comparison, which shared similar conditions to Germany, PISA results have not led to significant changes in educational policy.[10]

Because many countries have set national performance targets based on their relative rank or absolute PISA score, PISA assessments have increased the influence of their (non-elected) commissioning body, the OECD, as an international education monitor and policy actor, which implies an important degree of 'policy transfer' from the international to the national level; PISA in particular is having "an influential normative effect on the direction of national education policies".[7] Thus, it is argued that the use of international standardised assessments has led to a shift towards international, external accountability for national system performance; Rey contends that PISA surveys, portrayed as objective, third-party diagnoses of education systems, actually serve to promote specific orientations on educational issues.[4]

National policy actors refer to high-performing PISA countries to "help legitimise and justify their intended reform agenda within contested national policy debates".[11] PISA data can be "used to fuel long-standing debates around pre-existing conflicts or rivalries between different policy options, such as in the French Community of Belgium".[12] In such instances, PISA assessment data are used selectively: in public discourse governments often only use superficial features of PISA surveys such as country rankings and not the more detailed analyses. Rey (2010:145, citing Greger, 2008) notes that often the real results of PISA assessments are ignored as policymakers selectively refer to data in order to legitimise policies introduced for other reasons.[13]

In addition, PISA's international comparisons can be used to justify reforms with which the data themselves have no connection; in Portugal, for example, PISA data were used to justify new arrangements for teacher assessment (based on inferences that were not justified by the assessments and data themselves); they also fed the government's discourse about the issue of pupils repeating a year, (which, according to research, fails to improve student results).[14] In Finland, the country's PISA results (that are in other countries deemed to be excellent) were used by Ministers to promote new policies for 'gifted' students.[15] Such uses and interpretations often assume causal relationships that cannot legitimately be based upon PISA data which would normally require fuller investigation through qualitative in-depth studies and longitudinal surveys based on mixed quantitative and qualitative methods,[16] which politicians are often reluctant to fund.

Recent decades have witnessed an expansion in the uses of PISA and similar assessments, from assessing students' learning, to connecting "the educational realm (their traditional remit) with the political realm".[17] This raises the question of whether PISA data are sufficiently robust to bear the weight of the major policy decisions that are being based upon them, for, according to Breakspear, PISA data have "come to increasingly shape, define and evaluate the key goals of the national / federal education system".[7] This implies that those who set the PISA tests – e.g. in choosing the content to be assessed and not assessed – are in a position of considerable power to set the terms of the education debate, and to orient educational reform in many countries around the globe.[7]

The students tested by PISA are aged between 15 years and 3 months and 16 years and 2 months at the beginning of the assessment period. The school year pupils are in is not taken into consideration. Only students at school are tested, not home-schoolers. In PISA 2006, however, several countries also used a grade-based sample of students. This made it possible to study how age and school year interact.

To fulfill OECD requirements, each country must draw a sample of at least 5,000 students. In small countries like Iceland and Luxembourg, where there are fewer than 5,000 students per year, an entire age cohort is tested. Some countries used much larger samples than required to allow comparisons between regions.

PISA test documents on a school table (Neues Gymnasium, Oldenburg, Germany, 2006)

Each student takes a two-hour computer based test. Part of the test is multiple-choice and part involves fuller answers. There are six and a half hours of assessment material, but each student is not tested on all the parts. Following the cognitive test, participating students spend nearly one more hour answering a questionnaire on their background including learning habits, motivation, and family. School directors fill in a questionnaire describing school demographics, funding, etc. In 2012 the participants were, for the first time in the history of large-scale testing and assessments, offered a new type of problem, i.e. interactive (complex) problems requiring exploration of a novel virtual device.[20][21]

In selected countries, PISA started experimentation with computer adaptive testing.

Countries are allowed to combine PISA with complementary national tests.

Germany does this in a very extensive way: On the day following the international test, students take a national test called PISA-E (E=Ergänzung=complement). Test items of PISA-E are closer to TIMSS than to PISA. While only about 5,000 German students participate in the international and the national test, another 45,000 take the national test only. This large sample is needed to allow an analysis by federal states. Following a clash about the interpretation of 2006 results, the OECD warned Germany that it might withdraw the right to use the "PISA" label for national tests.[22]

From the beginning, PISA has been designed with one particular method of data analysis in mind. Since students work on different test booklets, raw scores must be 'scaled' to allow meaningful comparisons. Scores are thus scaled so that the OECD average in each domain (mathematics, reading and science) is 500 and the standard deviation is 100.[23] This is true only for the initial PISA cycle when the scale was first introduced, though, subsequent cycles are linked to the previous cycles through IRT scale linking methods.[24]

This generation of proficiency estimates is done using a latent regression extension of the Rasch model, a model of item response theory (IRT), also known as conditioning model or population model. The proficiency estimates are provided in the form of so-called plausible values, which allow unbiased estimates of differences between groups. The latent regression, together with the use of a Gaussian prior probability distribution of student competencies allows estimation of the proficiency distributions of groups of participating students.[25] The scaling and conditioning procedures are described in nearly identical terms in the Technical Reports of PISA 2000, 2003, 2006. NAEP and TIMSS use similar scaling methods.

The results of PISA 2018 were presented on 3 December 2019, which included data for around 600,000 participating students in 79 countries and economies, with China's economic area of Beijing, Shanghai, Jiangsu and Zhejiang emerging as the top performer in all categories.[26] Note that this does not represent the entirety of mainland China.[27] Reading results for Spain were not released due to perceived anomalies.[28]

Mathematics	Science	Reading
1  China (B-S-J-Z)[lower-alpha 1] 591 2  Singapore 569 3  Macau 558 4  Hong Kong 551 5  Taiwan 531 6  Japan 527 7  South Korea 526 8  Estonia 523 9  Netherlands 519 10  Poland 516 11   Switzerland 515 12  Canada 512 13  Denmark 509 13  Slovenia 509 15  Belgium 508 16  Finland 507 17  Sweden 502 17  United Kingdom 502 19  Norway 501 20  Germany 500 20  Ireland 500 22  Czech Republic 499 22  Austria 499 24  Latvia 496 24  Vietnam 496 26  France 495 26  Iceland 495 28  New Zealand 494 29  Portugal 492 30  Australia 491 International Average (OECD) 489 31  Russia 488 32  Italy 487 33  Slovakia 486 34  Luxembourg 483 35  Lithuania 481 35  Spain 481 35  Hungary 481 38  United States 478 39  Belarus 472 39  Malta 472 41  Croatia 464 42  Israel 463 43  Turkey 454 44  Ukraine 453 45  Cyprus 451 45  Greece 451 47  Serbia 448 48  Malaysia 440 49  Albania 437 50  Bulgaria 436 51  United Arab Emirates 435 52  Brunei 430 52  Montenegro 430 52  Romania 430 55  Kazakhstan 423 56  Moldova 421 57  Azerbaijan 420 58  Thailand 419 59  Uruguay 418 60  Chile 417 61  Qatar 414 62  Mexico 409 63  Bosnia and Herzegovina 406 64  Costa Rica 402 65  Jordan 400 65  Peru 400 67  Georgia 398 68  North Macedonia 394 69  Lebanon 393 70  Colombia 391 71  Brazil 384 72  Argentina 379 72  Indonesia 379 74  Saudi Arabia 373 75  Morocco 368 76  Kosovo 366 77  Panama 353 77  Philippines 353 79  Dominican Republic 325	1  China (B-S-J-Z)[lower-alpha 1] 590 2  Singapore 551 3  Macau 544 4  Vietnam 543 5  Estonia 530 6  Japan 529 7  Finland 522 8  South Korea 519 9  Canada 518 10  Hong Kong 517 11  Taiwan 516 12  Poland 511 13  New Zealand 508 14  Slovenia 507 15  United Kingdom 505 16  Australia 503 16  Germany 503 16  Netherlands 503 19  United States 502 20  Belgium 499 20  Sweden 499 22  Czech Republic 497 23  Ireland 496 24   Switzerland 495 25  Denmark 493 25  France 493 27  Portugal 492 28  Austria 490 28  Norway 490 International Average (OECD) 489 30  Latvia 487 31  Spain 483 32  Lithuania 482 33  Hungary 481 34  Russia 478 35  Luxembourg 477 36  Iceland 475 37  Croatia 472 38  Belarus 471 39  Ukraine 469 40  Italy 468 40  Turkey 468 42  Slovakia 464 43  Israel 462 44  Malta 457 45  Greece 452 46  Chile 444 47  Serbia 440 48  Cyprus 439 49  Malaysia 438 50  United Arab Emirates 434 51  Brunei 431 52  Jordan 429 53  Moldova 428 54  Romania 426 54  Thailand 426 54  Uruguay 426 57  Bulgaria 424 58  Mexico 419 58  Qatar 419 60  Albania 417 61  Costa Rica 416 62  Montenegro 415 63  Colombia 413 63  North Macedonia 413 65  Argentina 404 65  Brazil 404 65  Peru 404 68  Azerbaijan 398 68  Bosnia and Herzegovina 398 70  Kazakhstan 397 71  Indonesia 396 72  Saudi Arabia 386 73  Lebanon 384 74  Georgia 383 75  Morocco 377 76  Kosovo 365 76  Panama 365 78  Philippines 357 79  Dominican Republic 336	1  China (B-S-J-Z)[lower-alpha 1] 555 2  Singapore 549 3  Macau 525 4  Hong Kong 524 5  Estonia 523 6  Canada 520 6  Finland 520 8  Ireland 518 9  South Korea 514 10  Poland 512 11  New Zealand 506 11  Sweden 506 13  United States 505 13  Vietnam 505 15  Japan 504 15  United Kingdom 504 17  Australia 503 17  Taiwan 503 19  Denmark 501 20  Norway 499 21  Germany 498 22  Slovenia 495 23  Belgium 493 23  France 493 25  Portugal 492 26  Czech Republic 490 International Average (OECD) 487 27  Netherlands 485 28  Austria 484 28   Switzerland 484 30  Croatia 479 30  Latvia 479 30  Russia 479 33  Hungary 476 33  Italy 476 33  Lithuania 476 36  Belarus 474 36  Iceland 474 38  Israel 470 38  Luxembourg 470 40  Turkey 466 40  Ukraine 466 42  Slovakia 458 43  Greece 457 44  Chile 452 45  Malta 448 46  Serbia 439 47  United Arab Emirates 432 48  Romania 428 49  Uruguay 427 50  Costa Rica 426 51  Cyprus 424 51  Moldova 424 53  Montenegro 421 54  Bulgaria 420 54  Mexico 420 56  Jordan 419 57  Malaysia 415 58  Brazil 413 59  Colombia 412 60  Brunei 408 61  Qatar 407 62  Albania 405 63  Bosnia and Herzegovina 403 64  Argentina 402 65  Peru 401 66  Saudi Arabia 399 67  North Macedonia 393 67  Thailand 393 69  Azerbaijan 389 70  Kazakhstan 387 71  Georgia 380 72  Panama 377 73  Indonesia 371 74  Morocco 359 75  Kosovo 353 75  Lebanon 353 77  Dominican Republic 342 78  Philippines 340

Finland, which received several top positions in the first tests, fell in all three subjects in 2012, but remained the best performing country overall in Europe, achieving their best result in science with 545 points (5th) and worst in mathematics with 519 (12th) in which the country was outperformed by four other European countries. The drop in mathematics was 25 points since 2003, the last time mathematics was the focus of the tests. For the first time Finnish girls outperformed boys in mathematics narrowly. It was also the first time pupils in Finnish-speaking schools did not perform better than pupils in Swedish-speaking schools. Minister of Education and Science Krista Kiuru expressed concern for the overall drop, as well as the fact that the number of low-performers had increased from 7% to 12%.[47]

India participated in the 2009 round of testing but pulled out of the 2012 PISA testing, with the Indian government attributing its action to the unfairness of PISA testing to Indian students.[48] India had ranked 72nd out of 73 countries tested in 2009.[49] The Indian Express reported, "The ministry (of education) has concluded that there was a socio-cultural disconnect between the questions and Indian students. The ministry will write to the OECD and drive home the need to factor in India's "socio-cultural milieu". India's participation in the next PISA cycle will hinge on this".[50] The Indian Express also noted that "Considering that over 70 nations participate in PISA, it is uncertain whether an exception would be made for India".

India did not participate in the 2012, 2015 and 2018 PISA rounds.[51]

A Kendriya Vidyalaya Sangathan (KVS) committee as well as a group of secretaries on education constituted by the Prime Minister of India Narendra Modi recommended that India should participate in PISA. Accordingly, in February 2017, the Ministry of Human Resource Development under Prakash Javadekar decided to end the boycott and participate in PISA from 2020. To address the socio-cultural disconnect between the test questions and students, it was reported that the OECD will update some questions. For example, the word avocado in a question may be replaced with a more popular Indian fruit such as mango.[52]

In 2015, the results from Malaysia were found by the OECD to have not met the maximum response rate.[53] Opposition politician Ong Kian Ming said the education ministry tried to oversample high-performing students in rich schools.[54][55]

Sweden's result dropped in all three subjects in the 2012 test, which was a continuation of a trend from 2006 and 2009. It saw the sharpest fall in mathematics performance with a drop in score from 509 in 2003 to 478 in 2012. The score in reading showed a drop from 516 in 2000 to 483 in 2012. The country performed below the OECD average in all three subjects.[56] The leader of the opposition, Social Democrat Stefan Löfven, described the situation as a national crisis.[57] Along with the party's spokesperson on education, Ibrahim Baylan, he pointed to the downward trend in reading as most severe.[57]

In 2020, Swedish newspaper Expressen revealed that Sweden had inflated their score in PISA 2018 by not conforming to OECD standards. According to professor Magnus Henrekson a large number of foreign-born students had not been tested.[58] According to an article of Sveriges Radio, poor immigrant children's scores are a significant cause of the recent decrease in Swedish Pisa scores.

In the 2012 test, as in 2009, the result was slightly above average for the United Kingdom, with the science ranking being highest (20).[59] England, Wales, Scotland and Northern Ireland also participated as separated entities, showing the worst result for Wales which in mathematics was 43rd of the 65 countries and economies. Minister of Education in Wales Huw Lewis expressed disappointment in the results, said that there were no "quick fixes", but hoped that several educational reforms that have been implemented in the last few years would give better results in the next round of tests.[60] The United Kingdom had a greater gap between high- and low-scoring students than the average. There was little difference between public and private schools when adjusted for socio-economic background of students. The gender difference in favour of girls was less than in most other countries, as was the difference between natives and immigrants.[59]

Writing in the Daily Telegraph, Ambrose Evans-Pritchard warned against putting too much emphasis on the UK's international ranking, arguing that an overfocus on scholarly performances in East Asia might have contributed to the area's low birthrate, which he argued could harm the economic performance in the future more than a good PISA score would outweigh.[61]

In 2013, the Times Educational Supplement (TES) published an article, "Is PISA Fundamentally Flawed?" by William Stewart, detailing serious critiques of PISA's conceptual foundations and methods advanced by statisticians at major universities.[62]

In the article, Professor Harvey Goldstein of the University of Bristol was quoted as saying that when the OECD tries to rule out questions suspected of bias, it can have the effect of "smoothing out" key differences between countries. "That is leaving out many of the important things," he warned. "They simply don't get commented on. What you are looking at is something that happens to be common. But (is it) worth looking at? PISA results are taken at face value as providing some sort of common standard across countries. But as soon as you begin to unpick it, I think that all falls apart."

Queen's University Belfast mathematician Dr. Hugh Morrison stated that he found the statistical model underlying PISA to contain a fundamental, insoluble mathematical error that renders Pisa rankings "valueless".[63] Goldstein remarked that Dr. Morrison's objection highlights "an important technical issue" if not a "profound conceptual error". However, Goldstein cautioned that PISA has been "used inappropriately", contending that some of the blame for this "lies with PISA itself. I think it tends to say too much for what it can do and it tends not to publicise the negative or the weaker aspects." Professors Morrison and Goldstein expressed dismay at the OECD's response to criticism. Morrison said that when he first published his criticisms of PISA in 2004 and also personally queried several of the OECD's "senior people" about them, his points were met with "absolute silence" and have yet to be addressed. "I was amazed at how unforthcoming they were," he told TES. "That makes me suspicious." "Pisa steadfastly ignored many of these issues," he says. "I am still concerned."[64]

Professor Svend Kreiner, of the University of Copenhagen, agreed: "One of the problems that everybody has with PISA is that they don't want to discuss things with people criticising or asking questions concerning the results. They didn't want to talk to me at all. I am sure it is because they can't defend themselves.[64]

Since 2012 a few states have participated in the PISA tests as separate entities. Only the 2012 and 2015 results are available on a state basis. Puerto Rico participated in 2015 as a separate US entity as well.

2012 US State results

Mathematics	Science	Reading
Massachusetts 514  Connecticut 506 US Average 481  Florida 467	Massachusetts 527  Connecticut 521 US Average 497  Florida 485	Massachusetts 527  Connecticut 521 US Average 498  Florida 492

2015 US State results

Mathematics	Science	Reading
Massachusetts 500  North Carolina 471 US Average 470  Puerto Rico 378	Massachusetts 529  North Carolina 502 US Average 496  Puerto Rico 403	Massachusetts 527  North Carolina 500 US Average 497  Puerto Rico 410