I’m in the middle of studying for my university exams, so in terms of writing for this blog I’ve been much slower lately. I mean, it’s not as if there’s huge political change taking place, no sir, not at all. But hence, I am writing another blog. I’ve come to the point now that every time Scottish Labour or any other political commentator spout “PISA” thinking it is accurate data, I feel numerous brain cells in my head die.
For those who do not know, PISA stands for “Programme for International Student Assessment“, which is a study on science, maths and reading for the OECD. The data includes studies on 540,000 children in 72 countries across the world, which are then all ranked from best to worst. And in Scotland unionists love it.
(Ruth Davidson certainly has a cheek, since the PISA figures found Scotland did better than the UK average and the Operation for Economic Co-Operation and Development found English pupils “are most illiterate in the developed world”, with the Tory government being responsible for English education.)
It’s odd that PISA seemed to show Scotland falling in the ranking tables, because it does not seem to add up. Just before the release of the recent figures, Scotland had seen a rise in pupils securing places at university, record exam passes, evidence showing the attainment gap is closing, 92% of leavers reaching positive destinations and leavers gaining more qualifications (note the change in trend was when the SNP took office):
And the ONS themselves said that “Scotland is the best-educated country in Europe” in 2014. Yet despite this, PISA represents Scotland as such:
There is a lot of conflicting data, and the Scottish education system is certainly not perfect. I myself have been critical in the past of the SQA in my years in high school, and I remember seeing my friends come out of one infamous 2015 maths exam which had people rubbing their temples in confusion. And whilst there are other issues people can bring up, it’s fair to say the SNP have overall been successful on their education record. So why is PISA showing such a vastly different picture?
Let’s be frank here, PISA is a very powerful study that has influenced how countries approach their education system. Such influence extends to the point that some governments and parties will even form education policy around the ranking system, showing how globalisation has a large effect. If we take the view of Professor Yong Zhao from the University of Kansas, he has made clear that countries that have tried to make changes to education and have seen a slip in their rank quickly backtrack:
“It is putting everybody at risk. Countries that do well are put up on a pedestal so that they are not really [willing] to reform their education for fear of losing their spot. Pisa is putting people in a very awkward position.”
Professor Zhao in the same article uses Japan as a clear example:
Professor Zhao has been a long term critic of PISA, doing a five part series titled “How Does PISA put the World at Risk”. It’s certainly something worth reading if you have the time. But he is not alone in such a view that PISA is not reliable.
PISA uses the “Rasch model”, which calculates a pupil’s latent ability. From a pupil’s answer, PISA makes the assumption that from the way the pupil has answered the question they will continue to use similar methods to other questions that had not been asked to them already. Every single pupil within the PISA studies face the same standards, without taking into consideration 1) their culture and 2) how difficulty for each question varies. Such a view is shared by Professor David Spiegelhalter of public understanding of risk at Cambridge University, as he argues “They are predicted conditional on knowing the difficulties of the questions – as if these are fixed constants”. But not only does he think there is “considerable uncertainty” behind such methods, but also argues that “Assuming the difficulty is the same for all students around the whole world” is a mistake.
Statistician Svend Kreiner from the University of Copenhagen also criticises PISA. He makes the point that the same question for one child in their language is different for another child in another country in terms of difficulty. The reason for this is mostly down to the functionality of the question, which is the point Svend made:
It should also be important to note the view of Times Educational Supplement, who hold very strong views when it comes to PISA. In their own words, they call the figures “useless”, as it produces “meaningless” rankings as the methodology to gather such data is “utterly wrong”.
If you’re still not convinced that PISA has a lot of issues then look no further than an open letter from over 80 academics and professionals all expressing deep concern about the data. Even the Swedish government, which has one of the most successful education systems in the world, questions PISA.
But if we want to look at a less critical source, then take the viewpoint of the OECD themselves. They make the point that the data they gather is not precise, and give an example of the UK:
So the United Kingdom has a ranking variable of nine places. But despite such caveats, unionists will never point this out at all, because it suits their agenda to see the SNP fail when it comes to education. They probably do not even realise that PISA is calculated through a computer running a statistical programme to predict the answers of pupils to questions they have not even been asked. And if unionists do not know how such data is actually calculated, does it not ultimately prove their attack on the Scottish government is ultimately lazy and poor?
Many critics may then decide to mention the Scottish Survey of Literacy and Numeracy (SSLN). The findings here are something more reliable, but once again can be skewed. For example, the SSLN findings report that only 49% of 13 to 14 year olds pupils were doing “very well” or “well” in writing in 2016. This left a large percentage “working within” the expected level. But the phrases “very well”, “well” or “working within” are not scientific ones. Just because someone is “working within” their expected level does not suddenly mean they are underperforming, and any such claim is nonsense.
We should also note, that the new Curriculum for Excellence was introduced after one of the biggest recessions in history, at least since the Great Depression. Anyone who denies that there isn’t any social or economic ties to education seem to miss the human factor of this debate. Just around one in four children in Scotland live in poverty as there is record use of food banks. Families have seen some of the worst wage growth in Europe, with future welfare cuts that will drive one million more children into poverty. These factors have huge consequences when it comes to a child’s education, and we know that most macroeconomic levers are controlled by Westminster. This is not me shifting the entirety of the blame to Tory austerity, but to deny it has no large impact is the highest of ignorance.
Is the Scottish education system perfect? No, it is not. Is it absolutely abysmal? No, it is not. There are issues with the current education system in such difficult times, and the Scottish government should be judged on their policies. But we must be sure to be reading data correctly instead of throwing around silly claims to make headlines. This is complex issue with lots of factors to be considered.