‘Deeply concerning’: Minister’s ‘Skills Toolkit’ figures prove inaccurate, DfE admits

Skills Toolkit
Exclusive

The government has been urged to withdraw claims made in parliament about the take-up of the courses on its new ‘skills toolkit’ after FE Week started asking questions about the figures.

Significant over-counting has already led to revised estimates, and an investigation by this publication has found “course start” and “registration” claims in official statistics will continue to include web hits.

Despite this, education secretary Gavin Williamson described the free online courses as having a “transformational impact on so many people taking furlough” during a speech in October.

This raises major questions about the government’s ability to get a grip on the jobs crisis

The course content has not been developed by the government, but more than £1 million has been spent to develop and promote the skills toolkit “platform”, which consists of a web page on the National Careers Service with short course descriptions and links to the external websites.

The government says the educational content being promoted aims to help people who are out of work to boost their digital and numeracy skills during the pandemic.

However, little is known about who is accessing the content or whether they are completing the course, as almost half of the 79 available online courses require no registration.

Skills minister Gillian Keegan has publicly claimed these are “high-quality” courses but a former director of FE funding in the DfE, Susan Pember, questioned how this can be known in the absence of learner and completion data.

Many of the courses are simply short video tutorials or PDF documents that people can stop and start with no tuition and no external quality assurance from the likes of Ofqual and Ofsted.

Pember told FE Week she would advise ministers to be “cautious” as “website hits or even signatures on enrolment forms do not equate to learning taking place”.

The department has already had to row back on claims that 136,000 people started on the courses in the toolkit’s first month of operation but there are currently other incorrect records, including answers from Keegan to parliamentary questions (PQ) tabled by shadow chancellor Anneliese Dodds.

Google’s ‘Communicate Effectively at Work’ video tutorials on the Skills Toolkit (click to enlarge)

On October 5 the minister told Dodds that as of September 30, there had been 97,615 “course starts” and 16,219 “completions”.

In a separate PQ two weeks later Keegan said that as of October 4 there had been 118,011 “course starts” and 19,564 “completions”.

When FE Week asked if it was true there had been almost 20,000 additional starts on courses on the toolkit in that four-day period, the DfE admitted they could not confirm.

A spokesperson said the information was the “best data available to us at the time of response” but it has since identified “issues” with a number of courses that it was “working to resolve”. They added that the DfE will “update the answers as appropriate, once the examination has completed”.

Dodds, the Labour MP for Oxford East, told FE Week it was “really troubling” that government appears to have no idea how many people have registered for the skills toolkit courses and that it is “incumbent on them” to correct the parliamentary record.

She said she was also “deeply concerned” that the DfE has spent over £1 million on the resource “without knowing how it’s performing”, which raises “major questions about the government’s ability to get a grip on the jobs crisis that’s engulfing Britain”.

Pember, who worked as a top government skills official from 2000 to 2013 and is now the policy director of adult education network HOLEX, said that in order to “protect the reputation of FE” the courses need to go through the “same rigorous process as everyone else”, which includes audit and regulatory oversight.

She explained that after the individual learning accounts debacle in the early 2000s, it was “understood that we needed really good-quality criteria for FE courses – that is when you get your standards written, Ofqual and Ofsted involved” to ensure “good experiences for learners” and “good value for public money”.

FE Week’s analysis of the courses on the skills toolkit, offered by 14 providers which are mostly commercial businesses, shows that their duration can range from 25 minutes to 70 hours.

Just two of the 79 involve guidance from a live tutor. Almost all of them, 69, can include certification but only if the person registers for the course. Registration is, however, not required to view and use the content for 39 of them.

Examples include “Become a Sales Representative” from LinkedIn – a series of 25 short videos from a sales expert that allows you to download a certificate of completion at the end, with no testing.

LinkedIns Become a Sales Representative video course on the Skills Toolkit

Another is “Make It Click”, provided by the Good Things Foundation, which is a web page that directs visitors to other web pages, such as “A Beginners’ Guide to Excel” on YouTube.

Three of the providers that do not require registrations told FE Week they use unique URL data to capture the numbers of participants and then relay that information to the DfE to measure “starts”.

After being presented with the FE Week’s findings, the DfE said it was “in the process of developing an assured standardised system for collating data from different course providers”.

When asked how they know the courses have had a “transformational impact” and are “high quality”, as claimed by ministers Williamson and Keegan respectively, a spokesperson added: “The skills toolkit is a successful platform, delivering good value for money, which has encouraged tens of thousands of people to undertake online learning in courses that are valued by employers.

“It plays an important role in packaging up courses that are both free, high quality, and which people can access from home at this difficult time. We are reliant on providers for estimates of registrations and we continue to work with providers on improving this process further.”

Pember urged DfE to get a grip on the data: “We collect registration or enrolment data because that is the beginning of a performance pathway that allows you to monitor retention, completion and then how many people achieve.

“Without registration data you can’t do any of the rest and you can’t monitor whether that particular pathway for doing a subject is as good or better than the ones the state is already funding.”

She said the next questions DfE should be asking are how many learners have completed and been offered qualifications at the end. “Why treat those people taking the courses differently to other funded courses?

“I’m sure every provider in the land would like to be monitored just on registration – it makes it so much easier for people. So it seems there is an element of unfairness around it.

Good Things Foundation’s ‘Make It Click’ course material on the Skills Toolkit (click to enlarge)

“Other providers are wondering why there are now a set of very low benchmark rules for the stuff on the skills toolkit and a very high set of benchmarks for them.”

Having herself looked through the skills toolkit courses, Pember there is content that she “quite likes” but also found some worrying examples. “Take maths, for example – one of the courses says it is for level 1 learners, but the language around it is graduate level.

“Many people with level 1 maths also usually have level 1 English, so they’re not going to get past the first screen,” she said, adding that it could “demoralise” vulnerable people who in turn would be “lost to the system”.

In terms of costs for the skills toolkit, almost £800,000 has been given to a firm that the DfE refuses to name to carry out “user research and testing, IT security testing, content and user experience design, software development and performance testing”. Just over £325,000 has also been earmarked for advertising, such as paid-for social media on Twitter and Facebook.

Pember said the advertising “worries me”.

“When the DfE puts out its own advertising for the toolkit it always talks about good quality, but under whose judgment? It may be, but how do they know?” she told FE Week.

“You’ve got other outstanding providers out there but why is the state advertising a certain set of commercial products over and above another set of commercial products? It doesn’t even say there are others available.”

Aside from the PQs, the DfE is reporting the number of skills toolkit starts in their official apprenticeship data releases and labels them as “experimental”.

“These are experimental statistics and rely on website analytics from providers and the method of processing these is subject to change,” the department said.

The UK Statistics Authority said it welcomed the inclusion of these statistics to “support transparency and clarity” as long as the DfE acknowledges the data’s “limitations”.

Your thoughts

Leave a Reply

Your email address will not be published. Required fields are marked *

One comment

  1. Two things worry me, firstly the website is little more than a curated directory of courses already out there for free, and secondly, even assuming the courses are good quality, how effective are they at improving anybody’s employment prospects without a qualification?

    How will prospective employers react to a list of certificates of completion where no test or exam has been done? How does compiling a curated list and making a simple website cost £800,000?