Speed read: The 8 apprenticeship assessment concerns according to Ofqual

Ofqual has today published the findings of their initial “technical evaluations” of apprentice end-point assessment organisations (EPAOs).

It covers the first wave of 50 EPAs that the exams regulator was the external quality assurance provider for over the past two years.

The aim of the evaluations is to provide EPAOs with guidance “as to how to make sure their assessment materials meet the criteria”.

Eight “issues” that Ofqual said are “a threat to assessment validity” were found, ranging from there being more than one correct answer in multiple choice questions to contradiction between apprentice and assessor versions of materials.

FE Week has pulled out the key findings.

1. ‘Insufficient’ assessor guidance – unclear task requirements

Ofqual said that two thirds of evaluated EPA materials had issues with the “clarity of the task requirements” in the guidance provided to assessors.

These types of issues “pose a risk that assessment requirements are left too open to interpretation, and that judgements, and ultimately grading decisions, are not comparable between different apprentices”.

Often they did not consider whether the apprentice showcase should or should not involve a discussion, whether extra time should be allowed if the assessment isn’t completed in the allocated time and how the assessor chooses which practical tasks to assess the apprentice on.

2. Little or no exemplification of grading criteria

Two thirds of EPA materials had issues with a “lack of exemplification of the grade criteria or performance requirements, by which assessors judge proficiency”.

Ofqual said a lack of “necessary detail” might result in variations of interpretation by assessors, inconsistent approaches to criteria and “incomparability” of results.

In some cases during the technical evaluations it was “unclear” from the material how assessors should differentiate between grades for timed assessments and what the evidence requirements were.

3. General errors, contradictions and inaccuracies

Half of the EPAs Ofqual reviewed had a “wide range” of different errors, from spelling and grammar mistakes to contradictions between apprentice and assessor versions of materials.

The regulator said such errors “affect the expectations of both apprentices and assessors of the assessment experience and may result in unfair practices for apprentices”.

In some materials they found that titles of assessor guidance documents were “unspecific” and the wording was difficult to follow, and there were “discrepancies” about how and when the EPA should be booked – for example, in one case there was guidance stating the EPA should be booked 10 days in advance and opposing guidance which said it should be booked 20 days in advance

Inconsistencies in approach included one set of guidance stating that invigilators should mark the test, whilst another stated that this is not permitted.

4. Assessment requirements differ from the grading criteria

Examples of this issue found in EPA materials included the grading descriptors for merit and distinction “didn’t match the criteria in the assessment plan”, as well as that a list of knowledge and behaviours for an assessment method that didn’t match the requirements of the assessment plan.

5. ‘Lack of clarity’ around the task requirements for the apprentice

In some apprentice guidance Ofqual said it “wasn’t clear” that assessors may use the reflective journal as part of the assessment if they don’t observe elements naturally during the observation itself.

Arrangements for the preparation and submission of the portfolio were also “not sufficiently clear” and included “conflicting information”. For example, the regulator found in one part of a document it stated that the portfolio should be submitted when an apprentice reaches gateway, and later on it stated that it should be completed during the three months leading up to the EPA.

6. Not all knowledge, skills and behaviours are covered

In some assessment guidance Ofqual found that “not all” elements of the standard were covered, which could “result in an apprentice passing a specific assessment without having demonstrated all of the requirements set out in the assessment plan”.

The criteria provided was also “not specific enough”, and the language used was “different from that used in the assessment plan, so it was not clear that all criteria from the plan were covered”.

7. More than one correct answer in multiple choice questions

On a “number of occasions”, Ofqual’s technical evaluation “identified that some multiple-choice tests had items where more than one of the possible options could be considered correct, but only one of the responses was rewarded in the mark scheme”.

The regulator said in such a circumstance an apprentice might provide a legitimate answer to a question, but may not be rewarded with a mark.

8. EPA not meeting the requirements of the assessment plan

In “some” materials, Ofqual found that the assessment plan stated that the assessor should “use a range of questions devised by the EPAO” for the professional discussion, but the assessor guidance stated that assessors should review log books to identify areas to explore during the assessment.

In another example, the plan stated there should be 20 questions in an assessment that lasts one hour but there were 15 questions in the assessment which lasted 90 minutes.

Ofqual said such “anomalies” might lead to a variety of issues, including “different assessment methods being used to assess the same things within a standard, incomparable experiences for apprentices and unfair practices, leading to inconsistent and unreliable results”.