Roger Taylor, the former chair of Ofqual, who oversaw the grades fiasco last summer, has spent much of his life thinking about transparency and data. Here he says no algorithm would ever have worked ̶ and why BTECs were not the main focus
“An unsolvable problem.” That is how Roger Taylor, former chair of Ofqual, describes what the regulator was tasked with in 2020. To devise a statistical model for awarding grades when nobody was sitting exams that was accurate, did not cause inflation and was acceptable to the public. “An unsolvable problem,” he repeats.
Since resigning at the end of last year, Taylor has kept quiet. But from last week he has been talking. Because he was not an employee, he is not forbidden by contract from speaking out – a rare privilege among ex-top decision-makers.
It’s a power Taylor has wielded before. He publicly demanded that the education secretary Gavin Williamson stop taking credit for Ofqual’s decision to switch to teacher-moderated grades (which worked). Taylor also published the non-disclosure agreement Ofqual was asking algorithm experts to sign, after being criticised for its contents. In contrast to ex-chief regulator Sally Collier, who has been almost silent, Taylor has been free to speak up, and even call people’s bluff.
But why speak now? It’s six months since Taylor resigned. No one has been blaming him much for last year’s grading fiasco, including BTEC results being delayed for weeks. Williamson has taken most of the blame.
Yet Taylor has taken the rather unusual step of publishing an essay last week with the Centre for Progressive Policy, titled: “Is the algorithm working for us?”. Chapter One looks at “The 2020 exam debacle: how did it happen?”. Of course, by producing a defence, Taylor risks bringing criticism back on to his head. It’s a bold move.
When we meet virtually this week, Taylor is sitting at a desk behind which hangs a tasteful painting and two shelves packed with vinyl records. He has a lively, intelligent face and unpacks his ideas rapidly. He’s a PPE graduate from Oxford, with an MSc in economics, and is a former Financial Times journalist who reported on tech before moving into data technology businesses.
“It was very, very intense,” he says of the period when Ofqual started to design a grading model. “There was an incredible effort made by everyone to try to make something that was workable.”
It’s important to note that Ofqual said in 2020 the standardisation model it devised was not applied to most technical and vocational qualifications, including BTECs. Many VTQs could use evidence of learners’ work completed during the course to calculate results, and in other cases, learners had taken adapted assessments and so a moderation process wasn’t needed.
However, many students still faced grades they felt were lower than expected in the summer.
But the real fall-out was for half a million BTEC students, for whom results day brought unbelievable stress.
The uproar about the moderated results meant Ofqual switched, last minute, to school-based assessments for GCSEs and A-levels. This in turn meant that Pearson followed suit at the 11th hour with BTECs – pulling all their results the night before to bring them in line with the higher grades being handed to schools. BTEC students faced losing their places in further and higher education, as the clock ticked on for weeks until all results were issued.
Taylor’s point is that policymakers didn’t realise how much students would feel the government had risked their futures.
He points out Ofqual “is constitutionally obliged under law to prevent grades from inflating”. Education ministers were adamant: no grade inflation.
But “from the point of view of the individual citizen, the problem looks different. They see that the government has denied them the chance to demonstrate that they deserve a university place […] It has put their future at risk.”
With regards the BTECs delay, he adds: “We could have worked out early on that this approach was not an acceptable route and planned accordingly – which would have avoided the distress to students, including BTEC students, as well as the problems for university.”
Policymakers assumed they should offer the same number of higher education places as normal, and fill them as accurately as possible. Instead, Taylor says inflation (inaccuracy) should have been allowed, and more places made available.
His argument is essentially one about the difference between accuracy and legitimacy. “People are not willing to accept their lives being affected by a decision-making process driven by predictive algorithms,” says Taylor. “We risk missing this very basic lesson, if we comfort ourselves with the idea that the algorithm malfunctioned.”
In a way, Taylor is saying the mistake was basically a PR one; a failure to understand human psychology. “Teacher-assessed grades are in many ways more biased than the moderated grades,” he continues. “Their advantage is not that they are less biased; the advantage is that they allow for a significant amount of inflation.”
So why didn’t Ofqual spot the PR problem sooner?
“That is something everyone involved needs to reflect on.” He points out Ofqual’s consultation showed a degree of consensus. When asked about “the relative weight that the model should place on historical evidence of centre performance” (a bone of contention for many) 54 per cent agreed, with fewer (33 per cent) against.
Yet even if Ofqual didn’t spot the problem earlier, they were told about it later. The education select committee published a strongly worded warning in July, but still Ofqual persisted. Why not drop the model?
Taylor has a curious answer to this. “My view on that is you very quickly risk the regulator getting involved in what are properly political decisions. My own stance on that is quite conservative: politics is for politicians.” The answer is a tricky one, as Ofqual is an independent body, accountable to parliament – not a blind executioner of DfE will.
The question of Ofqual’s independence continues, as Williamson has appointed his own policy adviser to become chief regulator. Meanwhile, the government’s go-to person to lead expert reviews, Ian Bauckham, is now the chair.
Another big criticism levied at Ofqual was a lack of transparency. If Taylor draws a line on how “political” Ofqual should have been, he also draws a line on how transparent.
He himself is an author of a book on transparency, which he self-deprecatingly says “about three people have read”. Published in 2016, it is called Transparency and the Open Society. It makes the case, says Taylor, for transparency with certain limits. In a sense, it’s the same approach Taylor took with the algorithm itself.
Why didn’t Ofqual share the algorithm model?
“If you tell everyone about it, there is a risk of it leading to gaming,” responds Taylor. But surely sharing it with expert statisticians is not the same as sharing it with “everyone”? Yet Taylor holds that no standardisation model would ever have worked. They are simply unpalatable to the individual.
If you tell everyone about it, there is a risk of it leading to gaming
He is frank, meanwhile, about the focus on GCSEs and A-levels rather than BTECs. He says that at a political level in 2020 “there was a lot more focus on general qualifications” than on vocational qualifications “primarily because of the consequences around university admissions”.
Across government, “we are not on the whole focused enough on getting vocational qualifications working the way they need to”. It’s “not a problem that Ofqual can fix on their own.”
Then there was the delay in communicating what to do about qualifications for occupational competence, such as licences to practice. General qualifications are mostly used to “rank candidates”, explains Taylor, while licences to practice must ensure someone has an exact skill set. So estimating grades would have meant “exposing people to risks”. The regulator decided these exams should be sat later.
The licences to practice are an example of assessments that Ofqual saw as outside what the algorithm could responsibly do – although in the end, of course, this turned out to be the case for all qualifications: general, vocational and occupational.
Overall, Taylor deserves real credit for trying to make us think about the possibilities and limitations of algorithms, and the difference between accuracy and legitimacy.
He cares about digital technology in public services. He previously founded a company, Dr Foster, which drew data together about hospitals, and he has worked for the Careers and Enterprise Company. He becomes passionately frustrated as he explains the DfE should ensure every student has a “digitised record” of their achievements and qualifications.
It would allow students to keep their qualifications in one place (the biggest request from students to Ofqual are for copies of certificates, he says) while allowing others to “look at students in context”. It could particularly help disadvantaged learners, who often have a “thinner file”.
If people had richer individual education records … it might be a less pressurised situation
It might even alleviate the 2020 situations of the future, says Taylor, because “if people had richer individual education records and realised their fate didn’t hang on a single grade, but a more nuanced judgment, it might be a less pressurised situation”.
His belief in the smart use of data made the grading fiasco “quite painful to me personally”, Taylor reflects.
“I’ve spent most of my life looking at […] how do we use data that is fair to people and particularly in ways that empower individuals.” Instead the “government was using data in a way that was deeply and massively insensitive to individuals”. The son of a philosophy academic, Taylor seems to have been genuinely mulling the philosophical problems – and opportunities – of statistical modelling in education since he departed.
There are some holes in his answers. There is also a hole in his solution: this week, teachers warned that students with top grades had got no higher education offers, because universities had awarded too many places last year. In a way it goes to show, algorithm or no algorithm, every solution was deeply flawed.
I ask Taylor why he stepped down.
“Whatever you think about 2020, my view is that Ofqual is a world-class organisation. There’s not many organisations that understand assessment.” He laughs. “2021 is going to be a difficult year. It wasn’t going to help Ofqual’s case to have the same grey, old bloke in place.”
Perhaps, however, Ofqual has lost one of its most open communicators.
Ofsted’s chief inspector Amanda Spielman