Ofqual and the exams fiasco: 8 interesting things we learned

Senior staff from the exams regulator were hauled infront of the education select committee today where they were grilled on this year’s results fiasco.

Ofqual has been criticised for keeping quiet while chaos reigned on their now infamous grades-awarded-by-mutant-algorithm system, while ministers pushed blame their way. But the Ofqual team’s accounts seem to have shifted that blame squarely back onto ministers.

There was a hell of a lot to cover, but here are some of the key points:

 

1. Williamson’s claims he was unaware of problems look like bunkum

One of the more interesting developments post the exams fallout was the continued insistence from ministers, mostly education secretary Gavin Williamson, that he first knew there were issues with calculated grades a couple of days AFTER pupils got results.

MPs were told today that Ofqual was meeting with ministers on a weekly basis throughout the period. The exams regulator’s first advice to ministers was also it would be “challenging if not impossible to attempt to moderate estimates in a way that is fair for all students”.

They also briefed Number 10 on August 7th in a paper that was “very alert to the risks” the results posed to disadvantaged, outlier students, centre expecting improved grades and those with lower entry cohorts (who were actually advantaged by the algorithm).

 

2. Ofqual advised against calculated grades back in March

Interestingly, Taylor also revealed Ofqual’s initial recommendations to Williamson at the outset in March was to, first of all, try to hold exams in a socially distanced manner.

The second option was  to delay exams, and the third option was to look at some form of calculated grade, but Taylor suggested this should be some form of teacher certificate, rather than attempting to replicate exam grades.

However, Taylor said the decision soon came, from Williamson, to scrap exams and award calculated grades – without any further consultation with Ofqual.

 

3. How the 11th hour ‘mock appeal’ announcement started the downfall

Following the outcry in Scotland over their calculated results, Williamson announced his ill-fated “triple lock” appeals route – allowing pupils to appeal a grade if they had a mock that was higher.

It appears this was a key first link in a chain of chaos that led to pupils being awarded CAGs.

Up until this stage, it seemed Ofqual – while knowing its algorithm had flaws – was confident it could deal with the fallout. Its approach was thus: a system to contact schools and colleges who had pupils with “outlier” grade changes to make sure they knew they had a “good” case to appeal, and also the autumn resits (with universities holding open students’ places).

Taylor said he told Williamson he was “not confident” the new mocks appeals route “could be delivered … to ensure valid and trustworthy grades were issued”.

Nonetheless, Ofqual decided it should find a way to implement this, and after getting agreement from the DfE and Williamson’s office, published the guidance on Saturday afternoon.

But Ofqual was contacted by Williamson on Saturday evening at around 8pm saying this guidance wasn’t in line with government policy (and the guidance was swiftly removed).

A hastily-convened Ofqual board, meeting at 10pm that evening, then realised the situation was “rapidly getting out of control”, Taylor said, with policies “we felt would not be consistent with our legal duties”. (He said the mock appeals route would have led to around 85 per cent of pupils receiving their CAG, and a small number not receiving them – but this would have also taken an age to happen).

By Sunday evening, the opinion from Ofqual was there was no way to carry on “in an orderly manner” and the “only way out” was to move towards awarding CAGs.

 

4. Lessons learnt? Don’t try to replicate grades

As we’ve covered here, Ofqual has now admitted it was a “fundamental mistake” to believe that calculated grades would ever be accepted by the public – and said tweaks to the algorithm wouldn’t have made it any better.

Elaborating on this, Taylor said one of the lessons learned was “if you can’t replicate normal grades, don’t pretend you can” – pointing out that the objective was actually to enable progress for students onto their next step (rather than award GCSE grades) and that’s what seemed to have got lost in all this.

Note: this is important for decisions made for next year’s exam series – and it looks like any form of CAGs is totally off the table, with the focus instead on finding ways to deliver tests come what may.

 

5. College ‘concerns’ over ‘ill-equipped’ students

Julie Swan, Ofqual’s executive director, told MPs that the regulator is now hearing “some concern” from general FE colleges and sixth form colleges over students which received their teacher-assessed grades from schools that were over “optimistic” about what they would have realistically achieved in an exam.

The centre assessed grades now “indicate a level of ability that isn’t actually very accurate”.

“There are concerns that they [students] might progress onto a course for which they are now ill-equipped for,” she added.

 

6. Delaying exams next year has ‘enormous benefits’

Currently, government is proposing exams go ahead with some tweaks to assessments in some subjects.

But what about local lockdowns throwing this off-course? Swan said having an extra set of papers for pupils that may be ill, for instance, is something being considered.

Taylor also said another plan b could be using online tests.

The government has been consulting on plans to delay exams since June. While Taylor said there would be “enormous benefit” to delays, Ofqual still won’t commit to a date for when schools, colleges and pupils will have clarity. All they would say was a decision would be “weeks, rather than a period of months” away.

 

7. A whole new (transparent) world?

One of the big criticisms of Ofqual has been its secrecy over how the algorithm will work. This was touched upon a few times during the hearing.

Taylor said the “primary constraints” on transparency was not to publish information on how the system would work that could influence how teachers made their grades.

They also held onto publishing full details of the algorithm because that could lead to pupils identifying what grades they got before results day.

Michelle Meadows, executive director for strategy, added: “Our aim throughout has been to be transparent about the limitations of the model, and that’s why we did publish those metrics on results day so people could understand the limitations of what’s possible with statistical moderation like this.”

Meanwhile, Ian Mearns MP called for “deep, forensic” analysis on the CAGs submitted by schools and colleges to work out, for instance, how many schools and colleges “were gaming it”.

Taylor said he wouldn’t publish the full data – as it involves confidential informaiton – but said it’s “absolutely essential that independent researchers have access to this, in a secure way, so lessons can be learnt”.

And Taylor also committed to publishing Ofqual board minutes (the last of which was published in September last year).

 

8. But will Ofqual survive?

Committee chair Robert Halfon ended the hearing by asking whether Ofqual was still fit for purpose amid rumours the body could have a similar ending to Public Health England.

One of Halfon’s big issues was over a lack of communication from the regulator during the chaos (or “hiding away in the Ofqual attic”, as he put it). He highlighted Ofqual’s communications team included 11 people, with a job advert out for another comms person on £80,000.

Dame Glenys Stacey, who has stepped in as acting chief regulator after Sally Collier stepped down, said she has a “strong interest” in communications capacity, and that structure was “under active review”.

But she claimed the whole debacle had “shown the importance of an independent regulator”.

“We do have very important, very rare skills and expertise .. yes we’ve had an enormously difficult year… but we do have an important job to do,” she added.

Taylor said there had been a “major blow to confidence in Ofqual”, but claimed the regulator had a good track record over the previous decade on, for instance, delivering new GCSEs.

When asked about reform, Stacey added: “Yes I will look at the way Ofqual works, of course I will, and if changes are needed I’ll put those in play.”