The Office for Statistics Regulation has reprimanded the Department for Education over the data published for its new Skills Toolkit.
The body, an arm of the UK Statistics Authority, wrote to the DfE’s chief statistician Neil McIvor earlier this month to raise issues with the figures.
OSR deputy director for regulation Mary Gregory expresses concern in the letter that the DfE is not clear that users of the toolkit could be coming from across the globe.
She also pulls the department up on its use of unpublished data about the Skills Toolkit in multiple parliamentary questions.
The DfE has now agreed to implement a series of improvements in the data’s publication and committed to following the statistics authority’s “Code of Practice” through its publication of “experimental statistics”. Compliance with the code is only a requirement for official statistics.
The letter follows multiple FE Week investigations which have led to the DfE admitting previous data claims were inaccurate.
Significant overcounting has led to revised estimates of “registration” claims in the published statistics which continue to include web hits and could be coming from anywhere in the world as the course providers do not filter for geographical locations.
Earlier this month, FE Week also revealed how some course “completions” were being counted when users spent three minutes looking at one of the online resources.
The course content has not been developed by the government, but more than £1 million has been spent to develop and promote the skills toolkit “platform”. It launched in April 2020 and consists of a web page on the National Careers Service with short course descriptions and links to the external websites.
The government says the free educational content being promoted aims to help people who are out of work to boost their digital and numeracy skills during the pandemic.
Education secretary Gavin Williamson described the free online courses as having a “transformational impact on so many people taking furlough” during a speech in October.
The DfE now publishes “experimental” Skills Toolkit data alongside its monthly apprenticeships and traineeships statistics release. The publication does point out the limitations of the data and makes clear that reporting of registrations and completions varies by each provider.
But it does not explain how the data for each course is counted, nor does it inform users of the geographical coverage.
‘We have been clear about the caveats and limitations of this data’
Responding to the OSR’s letter, a DfE spokesperson said: “The Skills Toolkit was set up in response to the pandemic to ensure people have the opportunity to learn new skills. Right from the beginning, we have been aware the providers that supply the courses apply different methods to collect and measure usage.
“For these reasons we have published ‘experimental’ statistics which reflects the need to publish data while we seek to develop them further as we establish what data our users need. We have been clear about the caveats and limitations of this data. The UK Statistics Authority provides this designation of ‘experimental’ for exactly those purposes.”
The OSR says in its letter that the Skills Toolkit data is published under the “additional analysis section” of the publication which “may not be clear to users looking for these data in the release”.
It also acknowledges that the section provides users with “some of the limitations of the Skills Toolkit data”.
However, “not all limitations of the data are included. For example, they do not currently inform users of what the geographical coverage is,” the letter reads. “It is important to clearly explain the relevant limitations of the data and statistics to aid user understanding, and these should be regularly reviewed as the statistics are developed.”
The DfE spokesperson said they will implement “further improvements” to address these concerns in its next release.
Your thoughts