AI to vet apprenticeship vacancies for discrimination

The Department for Education is developing an artificial intelligence system to vet apprenticeship vacancies for discrimination and decide whether they need to be reviewed by a human before they are published.

The proposed “recruit an apprentice AI vacancy quality assurance API” will use a large language model to assess vacancies for potential breaches of the Equality Act 2010, alongside checks for spelling and grammar, missing information, inconsistent training associated to the apprenticeship, geographic issues and duplicate adverts.

It is intended to reduce the number of vacancies requiring manual checks, with the DfE projecting a 40 per cent saving in manual review costs.

A notice published by the Cabinet Office, Department for Science, Innovation and Technology and Government Digital Service said the system will use GPT-4o via Microsoft’s Azure Enterprise OpenAI service to assess apprenticeship vacancy text.

About 40,000 apprenticeship vacancies are uploaded to the government’s Find an Apprenticeship service each year, with each undergoing a manual review that can take up to 24 hours.

The new system is expected to process between 5,000 and 6,000 draft vacancies a month.

The DfE did not respond to FE Week enquiries about the timeline for launch, how much manual checks currently cost or whether the AI system would reduce staffing.

Stephen Evans, the chief executive of the Learning and Work Institute, said that AI could help to increase the efficiency of checks given the volume of vacancies. The current system was “unlikely to be perfect in any case as people can make mistakes just as AI can”.

However, he added that use of AI “must not come at the cost of accuracy so it’s important to test the system thoroughly before rollout and review and update it thereafter”.

AI to screen for discrimination

The government’s notice said the apprenticeship vacancy model has one specific discrimination check, alongside a check for missing or inconsistent content and nine spelling checks.

AI assigns each vacancy a red, amber or green rating.

All red vacancies – those deemed high risk – will be sent for full human review.

Half of amber vacancies will be reviewed, while only 1 per cent of green vacancies will be randomly sampled.

Vacancies that receive a green rating and are not selected for sampling will proceed automatically to the Find an Apprenticeship service without a further human check.

The DfE said this approach would allow reviewers to concentrate on “higher-risk or ambiguous vacancies”.

Phoebe Moore, professor of management and the futures of work at the University of Essex, said using AI to identify discrimination was an “interesting and potentially progressive innovation”, but that allowing AI to decide which adverts did not need human checks raised questions about whether it could reliably spot discriminatory language.

“Usually AI is critiqued for having discriminatory properties, so I think it’s interesting to flip that on its head and say we’re going to use it to identify discrimination,” she told FE Week.

“But I would question how the system will be trained to spot discrimination. Usually, patterns are what help an algorithm identify something, so what training data is going to be used and how will it deal with vacancies that don’t already fit those patterns?

“You can’t necessarily use a blanket category to identify discrimination because of the specific requirements of each vacancy. You need to understand why a company is asking for particular criteria.

“I think the function will draw attention to the potential for discrimination, rather than automatically pronounce that a vacancy is discriminatory or not. I wouldn’t say the Equality Act allows you to simply make a list of prohibited words, as you might in content moderation. It’s a tricky one.”

The department tested the AI checks against two and a half months of manually reviewed vacancy assessments.

For discrimination specifically, it recorded a 100 per cent true positive rate and a 0 per cent false negative rate for discrimination in that historical testing – meaning it identified all cases spotted by human reviewers.

However, the model also produced a 16 per cent false positive rate, meaning it flagged some vacancies as potentially discriminatory when human reviewers had not identified an issue.

The DfE said false negatives were of greater concern because they could result in a problematic vacancy being published, while false positives mainly created additional work for human reviewers.

Human in the loop

Officials have classified false negatives and hallucinations as moderate risks.

The department said its model could hallucinate – producing a factually incorrect classification or recommendation – but a “human in the loop” would minimise the impact.

It also acknowledged that identifying fraudulent or malicious vacancies was “inherently challenging” because of the “subjective nature of judgment” and might require information outside the vacancy itself, such as financial or Companies House checks. It described this risk as unavoidable.

A spokesperson for the Equality and Human Rights Commission said: “Under the Equality Act 2010, all public bodies – including the Department for Education – must make sure their policies promote equal opportunities, good relations and don’t lead to unlawful discrimination.

“While artificial intelligence can help transform public services for the better, improving delivery and reducing costs, public bodies must take steps to prevent AI from perpetuating bias and discrimination.”

‘AI is not infallible’

AI is increasingly being adopted across various government departments, according to a House of Commons Library report.

The DfE is experimenting with another AI tool to assist teachers by pooling government documents such as curriculum guidance and lesson plans, and the department could soon use artificial intelligence to draft responses to as much as 80 per cent of its external correspondence.

Evans said: “Ultimately AI is not infallible, only as good as its design, and will need ongoing updating based on expert human input. Both the department and users of the service must have the chance to feed back and the government should take action if AI is missing things or getting things wrong.”