Lessons to be learned following SFA data ‘episode’

An investigation into changes on FE Choices has found more than 2,500 data records were adjusted without public knowledge.

The Data Service, which is part of the Skills Funding Agency, has revealed that 2,639 changes were made to “unique values” in the Learner Satisfaction indicator of the performance comparison site after it went live on January 26.

As revealed by FE Week last month, FE Choices, formerly known as the Framework for Excellence, is under scrutiny for unannounced post-publication alterations.

Despite the changes, made to comparable data fields, neither the Agency nor the Service made a public announcement, which could leave them in breach of the Code of Practice for Official Statistics by the UK Statistics Authority.

The Code requires that prompt public announcements are made on errors which are discovered in statistical reports.

A statement, posted on the Service’s website, read: “The revisions related to the data from the survey of learners.

“The Learner Satisfaction national, provider and organisation type aggregate scores were not affected, but supporting scores on the sub-scales were affected for some organisation types and providers.”

The data changes, according to the Service, included comparative data at organisational type and national level, which was changed to exclude zeros from the calculations. This caused a change in the minimum value and medians.

This change was to background data and no headline score was affected, say the Service.

Changes were also made to correct errors whereby the code for question 7 was wrongly referring to question 6 data (for the 19+, all levels category) and the code for comparative data for question 2 was referring to data from question 1.

The Agency initially said the post-publication changes were made on January 30, but they now say the adjustments took place on February 6.

An Agency spokesperson said: “At the time of the initial message being uploaded onto the FE Choices website it appeared that a change had occurred on January 30 as this was the date stamp on the file which contained the changed data.

“The investigation has shown that the changed file was uploaded to the FE Choices website on 6 February.

“All data on the website from February 7 is accurate and correct.”

The changes, according to the Service, represent 0.7 per cent of the 371,987 total unique values. However, a closer look shows that 205 unique values for “Organisation Type detailed scores” were changed – some 12.7 per cent of the 1,620 total values. Meanwhile, 30 of the 405 unique values for “National detailed scores” were changed, equating to 7.4 per cent.

The Skills Funding Agency said they have “a duty to let users know as soon as possible” about what changes were made.

A statement released by the Agency said: “The Skills Funding Agency has published information on what changes were made to the data on FE Choices, in compliance with statistical protocols.

“The investigation process is ongoing and a full report will be published in due course.”

The Agency also said it is identifying lessons to learn from the episode.

“The Skills Funding Agency and the Data Service continue to abide by the Code of Practice for Official Statistics and have taken swift action to conduct a thorough investigation.

“The Agency and the Service are identifying lessons to learn and steps to take to ensure that there is no recurrence by reinforcing practice, quality management and testing procedures,” the statement said.

As well as the investigation by the Agency, the Department for Business, Innovation and Skills, under a request by the FE and skills minister, John Hayes MP, and the UK Statistics Authority have launched their own inquiries.

As well as the Learner Satisfaction changes, data has been revised for one provider’s Employer Satisfaction indicator.

The Service said: “The challenge is isolated to one provider and does not raise concern on other providers data or other performance indicators.”