Measuring impact fairly in the devolved areas

26 Jan 2020, 6:00

The mayoral combined authorities are developing impact and performance measures to assess whether the adult education budget is having a positive impact in their area. Setting the right targets and outcome measures, in a fair, simple and cost-effective way, is critical, warns Harminder Matharu

The end of January marks the end of the first six months of devolved delivery of the adult education budget (AEB) and the returns from colleges and providers may indicate whether the new arrangements are resulting in less underspend than has been the case nationally over the past few years. Of course, it’s not just about how much funding has been spent; having fought so hard to get it devolved, the mayoral combined authorities (MCAs) and the Greater London Authority (GLA) will want to assure themselves and stakeholders that local delivery has made a tangible difference to their local communities.

The combined authorities should ensure fairness across all provider groups

The whole point of devolution is that each of the combined authorities will eventually define for themselves their AEB priorities. Increasingly there is talk of assessing the impact on these, rather than looking solely at outputs and outcomes, although the latter will obviously help measure impact. Work is already under way, with the GLA commissioning the Learning and Work Institute to support the development of their impact-assessment proposal with a view to running a pilot.

The Liverpool City Region is piloting outcome-based delivery through their sector-based work academies, while the West Midlands combined authority is testing wider than traditional approaches.

All are also very conscious of finding the best way to measure the impact of AEB on social value, where the learning may not be centred around qualification and outputs.

AELP welcomes this work, and if stakeholders are supportive, we would like to see it used as a template for developments in other areas.

We hope that the devolved areas as a collective group will also keep a wider national picture in mind because providers deliver across multiple areas. Measuring impact is a complex challenge when faced with variances in local labour markets, and AELP has suggested to the nine devolved areas a set of principles to consider.

These start with setting the right targets and outcome measures from the outset, as this is critical for ensuring that adult education reaches the right individuals. The combined authorities should consider and compare approaches in neighbouring areas, along with any national approaches, to identify common aspects to align where there is value in doing so. 

Outcome measures should take into account the learner’s starting point to accurately assess distance travelled, impact on social mobility and real added value. This should be backed up by the measurement of planned progression outcomes to allow for longer term impact assessments. We should recognise that cohorts of learners may require differentiated outcomes supported by clear criteria and rationale for assessing impact.  This involves acknowledging the varying socio-economic factors affecting them.

It will undoubtedly help if there are clear definitions for all performance measures related to assessing the impact of delivery. At the same time, we should minimise the administrative burden and financial costs of data collection. 

Finally, the combined authorities should ensure fairness across all provider groups through a system that allows providers to invest and plan delivery which maintains value for money throughout.

As well as deciding what progression outcomes should be measured, it’s important also to agree on when to measure progression, because a learner may not benefit from it in the short term. Existing data collection systems should be reviewed before new systems are created, which means engaging with other stakeholders and partners that already collect measurement and destination data. For example, we recommend a look at the DWP’s new Data Lab and exploring whether this allows tracking of individuals over a longer period.

Despite the size of the challenge, we must work to keep the processes as simple as possible with cost considerations at the forefront of the emerging proposals.


Your thoughts

Leave a Reply

Your email address will not be published.

One comment

  1. John Broadhurst

    WMCA have done a wonderful job for devolution. Been taken to court for depriving hundreds of learners in its two most deprived regions. How would you measure the impact of that? Increased unemployment, more child poverty created, young people with fewer opportunities do we need to go on.