Bootcamps evaluation rings spending review alarm bells

The government is oblivious to how many people from the first wave of its flagship skills bootcamps secured a job or received a pay rise due to unreliable data, researchers have found.   

Sector leaders have warned this will ring alarm bells in the Treasury ahead of next week’s spending review, which will decide whether more taxpayer cash is invested in the programme.   

A process evaluation report for the first digital skills bootcamps, which began delivery in six areas in England in autumn 2020 at a cost of £8 million, was published by the Department for Education today.   

Researchers found positive results from qualitative interviews and analysis of management information: 81 per cent of participants successfully passed their assessments.   

Most encouraging was that 48 per cent of people enrolled were women – a much higher proportion than seen in the digital workforce, which has reportedly sat at around 20 per cent for some time. The finding will help ease sector leaders’ fears about bootcamps becoming male dominated.   

However, the researchers warn that the data obtained by the DfE from training providers did not cover all the bootcamps and “more problematically” did not consistently cover all of the geographic areas involved.   

They therefore “cannot be viewed as an accurate picture of all bootcamp provision” and instead “offer an illustrative overview of the bootcamps that submitted full data”.   

The data obtained showed that over 2,500 people applied for bootcamp courses and of these, 820 gained a place.   

While characteristics of participants, attendance and completion data could be drawn from the sample, the researchers found incomplete data on job and salary outcomes, even though the DfE specifies this information must be captured.   

The report said: “It seemed likely that these data had not been completed consistently (if at all) by providers, since there was no data on starting salaries and three learners were recorded as gaining a job.”   

It added that employment outcomes could be reported up to six months following the start of training so this “may have represented outcomes not yet being achieved, or providers not yet being able to evidence them”.   

The researchers carried out surveys with some bootcamp learners to get a better idea of their progress in the absence of reliable management information.   

Concerningly, 81 per cent (56) of 69 respondents who had completed their course said they were not offered a job interview at the end of their course, even though this is supposed to be a guaranteed part of the programme.   

Just nine per cent had been offered an interview with an employer, while ten per cent said they “did not know”.   

The DfE previously told FE Week that bootcamp data will not be submitted through its individualised learner records (ILR) and instead providers will submit their own data through a form on a monthly basis.   

Stephen Evans, chief executive of Learning and Work Institute, said bootcamps are a “good idea” but without “proper data we don’t know if they’ve succeeded in their aim of helping people into work”.   

“That makes it more difficult to know if they should be rolled out further and to persuade the Treasury to provide further funding,” he told FE Week. “With the need for skills and jobs only increasing, the DfE needs a much more robust approach to gathering data so we know whether bootcamps are working.”   

Some mayoral combined authorities that managed bootcamp contracts for their area even failed to gain consent from learners for their data to be used.   

Simon Ashworth, director of policy at the Association of Employment and Learning Providers, said: “There are high expectations on providers to achieve good outcomes for unemployed adults.   

“It’s shame that the dataset is incomplete, especially with a critical decision on whether to expand bootcamps expected as part of the spending review. The Treasury will want evidence to support this decision.”   

The researchers’ report says the DfE must make “amendments to the data collection sheets given to the providers” to measure success of bootcamps, and officials must relay the “importance of clear, accurate and timely information”.   

A DfE spokesperson claimed “robust” data collection on outcomes for wave one has been “ongoing from March to September 2021”. This data is “currently being validated and will be published by the end of November 2021” – a month after the spending review.   

Prime minister Boris Johnson announced the first wave of skills bootcamps in September 2020, plugging them as part of his new lifetime skills guarantee. A further £36 million has been invested in the courses since.   

The courses run for up to 16 weeks and are free to unemployed people, or where employed, their employer would pay a 30 per cent cash contribution.   

Across the six areas in wave one, digital provision operated in all areas, and covered topics ranging from digital marketing, women in software engineering, cloud services engineer, computer-aided design (CAD), coding, cybersecurity, IT, social media and digital leadership.   

Of the management information available, only 16 per cent of participants were already qualified to level 2 or below and 63 per cent were already in work or self-employed.   

The DfE spokesperson pointed to the researchers’ learner survey which “indicated that nearly four-in-five (79 per cent) were satisfied with their course overall”.   

However, only a little over two-in-five (44 per cent) survey respondents agreed that their bootcamp training and provision would be sufficient for them to apply for a job in their industry.   

The DfE spokesperson said: “We are pleased that this early evaluation report shows that wave one skills bootcamps were well received by all stakeholders involved and that there is overwhelming support for skills bootcamps to continue.”