Missing records thwart evaluation of covid tuition fund

Researchers could not judge whether catch-up tuition worked in colleges after more than 500 institutions that took the funding failed to log which students received it.

An independent impact evaluation of the 2021-22 year of the 16-19 tuition fund, published last week, found that so many schools and colleges omitted information from their data returns that no reliable comparison group could be built for FE and sixth form colleges.

The 16-19 tuition fund allocated £420 million over four academic years, with £102 million committed in 2021-22.

The data gaps forced researchers from Ipsos UK and the Education Policy Institute to abandon plans to draw “meaningful policy inferences” on the fund’s impact on students’ attainment.

The Department for Education said it has since introduced “clearer” reporting requirements and better validation for providers.

Researchers were only able to produce an impact estimate for school sixth forms, covering around a third of students who benefited from the fund in 2021-22.

Those students scored an average of 1.49 points less in their 16-19 qualifications than matched students who did not, equivalent to about 15 per cent of an A Level grade.

But researchers warned that the finding could not be relied on because school sixth forms are more selective than colleges and 85 per cent of tuition recipients in the sample already held grade 4s in GCSE English and maths. They were not the primary group the policy set out to help.

The report said those students were “unlikely to be representative of the broader population that benefitted”, and that “no meaningful policy conclusions can be inferred”.

Comparison group collapsed

DfE launched the fund in the 2020-21 academic year as part of the government’s education recovery programme. It paid for small group tuition in English, maths and other subjects disrupted by the pandemic.

Eligibility was widened in 2022-23 after complaints from the sector that the criteria were too restrictive, before the fund closed at the end of 2023-24.

The evaluation covered 2021-22, when £102 million was allocated. Of 2,506 eligible institutions, 1,567 opted in, 63 per cent, down from 77 per cent the previous year.

Researchers set out to compare exam results for students who received tuition with similar students who did not, using the summer school census and the individualised learner record.

But a substantial proportion of the data was unreliable for every type of institution, because providers that had taken the funding did not record who received the tuition. Students placed in the comparison group may in fact have had it. School sixth forms accounted for the largest numbers.

“This reporting issue meant that many students included in our matched sample may have benefitted from the 16-19 tuition fund even though the data suggested they had not, making the analysis unreliable,” the report said.

It added: “We advise caution in interpretation of this analysis, […]  no meaningful policy conclusions can be inferred.”

Once the affected providers were stripped out, only two FE colleges, seven sixth form colleges and three other providers that had not opted in remained. Researchers tried matching students repeatedly against that small pool, but the sample size was too small to use.

‘We have introduced clearer reporting requirements’

The evaluation recommended DfE collect monitoring data on the type, frequency and duration of tuition delivered for any future tuition or catch-up programmes for 16 to 19-year-olds.

It also advised officials to set minimum expectations for delivery, align eligibility criteria with intended outcomes and test different models of tuition within a sample of institutions before any wider rollout.

A separate process evaluation, published in 2023, relied on provider and student surveys and found more positive perceptions of the fund for the same 2021-22 academic year.

Nearly 9 in 10 (88 per cent) institutions agreed the fund had improved students’ educational attainment and 23 per cent of students said they achieved better results.

A DfE spokesperson said: “We accept the evaluation’s recommendation to strengthen data collection and monitoring requirements. We have introduced clearer reporting requirements alongside improved validation and support for providers, helping ensure more robust evaluation while retaining the flexibility needed to deliver support effectively.”