One of England’s largest apprenticeship providers was left out of this year’s qualification achievement rates (QARs) after a systems error caused the duplication of 700 learners, leaving its figures “unreliable”.
Several other big providers that were included in the government’s official performance tables told FE Week they experienced similar problems but did not spot the issues until after the deadline for challenges, adding that their true rates are higher than those published.
But officials have insisted the issue was not widespread and is understood to not have had a significant impact on national results.
ILR transitions create duplications
Apprenticeship achievement rates were published at national and provider-level on March 31 for the 2021/22 academic year.
The data showed that the achievement rate for standards stood at 51.4 per cent, down 0.4 percentage points on the year before and a long way off the 67 per cent achievement rate target that ministers have set for 2025.
Two providers were redacted from the formal QAR dataset – one of which was Paragon Education and Skills Ltd, a long-running apprenticeship firm that had 3,270 leavers in 2021/22.
The company’s commercial director Paul O’Hagan told FE Week the redaction was based on “an error in one field of our ILR (individualised learner record) upload” caused after Paragon migrated to a new systems provider.
“There was a duplication of 700 learners at R04,” he explained. “What that meant was our stated QAR was coming out 10 percentage points less than it actually was. The transition created the duplication, and this was not picked up by the ESFA data self-assessment toolkit or the software we use to check success rates from multiple year ILRs.”
O’Hagan said the ESFA was “very supportive in terms of redacting the data based on the premise that it wasn’t reflecting of actually where we are as a business from a QAR perspective”, adding that the firm’s systems provider was “very open and very supportive with taking all the actions, so it’s not an issue moving forward”.
When the data duplication was removed, Paragon’s QAR moved from 39 per cent to about 49.9 per cent.
Nine of the 20 largest apprenticeship providers in England are recorded as having overall achievement rates of below 50 per cent in the DfE’s official QAR data.
They include the likes of Lifetime Training (45 per cent), Babington Business College (48.1 per cent), and HIT Training (36.1 per cent).
Several other providers that spoke to FE Week said they experienced similar data issues to Paragon – there was a problem with learner identification during their migration to new systems, which counted some leavers twice.
Jill Whittaker, managing director of HIT Training, said this impacted her provider, leaving its QAR around five percentage points lower than what it truly is.
Some of those affected said they experienced the issue when moving to Bud Systems.
A spokesperson for Bud said: “Bud was made aware of a system error that affected a small number of our customers. Bud worked closely with our customers and the ESFA to diagnose the issue and the extent of the impact. The ESFA concluded that the error did not have a significant impact on national QAR production. This issue has now been rectified within the software.”
A DfE spokesperson added: “We have a clear and robust process for calculating QARs based on data submitted by providers, who are responsible for the quality of their own data.
“Every year, providers have had the opportunity to report any issues regarding QAR, and we have worked with them to address these concerns. Any concerns raised have been fed into the usual Quality Assurance processes supporting final publication.”
QARs are for a ‘different world’
Regardless of the system issues, there are sector-wide concerns that the way in which apprenticeship achievement rates are calculated is out of date and represents a regime prior to the introduction of an employer-led system.
Lifetime Training, for example, said its QAR reflects the Covid-19 period, when its core sectors of hospitality, care and retail were facing significant challenges which impacted learners and caused many withdrawals.
A spokesperson for the provider said the current QAR calculation is a “one-dimensional metric which does not consider sectoral differences around turnover and workforce dynamics”.
Whittaker echoed this. “When the ESFA measure QAR it is assumed that all leavers are the providers fault, when most leavers are down to the learner or employer. How can the provider be responsible for learners who change career or move to an employer who doesn’t want to continue the apprenticeship?” she told FE Week.
“QAR was developed to measure the success of apprenticeship frameworks. We are now in a different world, working under a different regime which is very much controlled by employers, not by providers. It is time to update the current outmoded measurement systems to reflect this.”
QAR methodology works better for qualification based programmes where you have a two way relationship between learner and provider.
In apprenticeships you have a three way relationship as employers are also involved. Achievement rates still has value as a measure, but should really only be treated as the start of the conversation and definitely not a stick to beat providers with. (less superficial judgement and more dialogue..)
The health of an economic sector (ie job stability & workforce churn) and even policy decisions flow through to QARs. For instance, the impact of incentives could well ripple through achievement rates soon.
For me, the real risk of a one size fits all QAR methodology is its capacity to distract from the real issue, which is whether the provision is right for the situation. For example, in some occupations an apprenticeship is the perfect type of training programme, whereas in other occupations something else might be a better fit.
My understanding is it was a problem at the QAR end of things, not provider end, where, if a learner had “previous year learner ref” in multiple years (primarily used where providers change MI systems), the QAR calculation couldn’t cope with it and it left dangling records. This has never been mentioned as a potential issue, although providers should have spotted it in the in-year QAR as it should be pretty easy to fix *once you know it’s a problem*…
“This type of issue is inevitable whilst the entire apprenticeship system remains unnecessarily complicated and overly bureaucratic. What this article also really reveals is the significant apprenticeship policy failure that results in a large provider such as Paragon Skills who only recently were awarded Training Provider of the Year 2023 at the AAC Apprenticeship awards, and yet they can only deliver an achievement rate of 49% (adjusted from 39%) ! “