Who Evaluates the Evaluation? Grassroots knowledge and experience of evaluation

blog | Words Otto Wolf | 13 Dec 2023

Introduction

Innovation Unit is the Learning and Support Partner on The National Lottery Community Fund’s Health Equalities grants programme. The 12 grantees (meet them here) are all working to develop effective and sustainable partnerships between the voluntary and community sector and the statutory sector (including the NHS and local authorities) to improve health and wellbeing, reduce health inequalities and empower communities in their local areas. We bring the cohort together to generate and share learning across the programme. 

In our most recent cohort learning event, we dived into the world of evaluation. Above all, grantees reflected on the importance of context-driven evaluation, recognising how the decisions made when setting up the evaluation can impact the results generated.

Grantees’ top tips on evaluation:

1. Internal vs. External evaluators 

When you’re doing an evaluation, it can be a smart move to think about getting an outside expert involved instead of doing it all in-house. They bring a fresh perspective, expertise, and the potential to share your findings beyond existing contacts.

Of course, working with external evaluators can have its challenges. You might have to give up some control over what they focus on, how they do the evaluation, and subsequent outcomes.

So, if you’re thinking about collaborating with external evaluators, it’s vital to begin with comprehensive preparation meetings going over any specific jargon or processes frequently used within your practice alongside areas you believe require the most focus. This ensures everyone is on the same page about the evaluation’s scope and foci.

2. Measure what matters, not what’s easy 

Evaluations can throw you a curveball sometimes, revealing unexpected findings without clear solutions. 

One grantee shared that their service users had raised that measuring trust should be an important feature of their programme evaluation. Trust is neither easily measured nor built.  Measuring it requires sensitive, in-depth data collection and has the potential for a range of possible outcomes. In these situations, it is important not to shy away from these findings, but invest time to properly consider how to measure and capture crucial aspects of a programme, often using qualitative research methods such as interviews or case studies.

3. Keep it evolving 

Once an evaluation process has begun, it is important to keep asking yourself how it can be improved. At our learning event one grantee described their ‘DIY process’ where, as they were conducting their evaluation, they were constantly considering which parts worked and which sections needed reworking. Another shared their experience of bringing their ‘theory of change’ to different demographics to test before scaling it up. In both examples, grantees didn’t assume that because a method worked for one group or context that it would inevitably continue to. Instead they continued to interrogate their evaluation process to ensure that it succeeded in achieving the outcomes they needed.

4. Keep it appropriate, proportionate and manageable 

When conducting an evaluation, it is important that the scale of the evaluation matches the scale of the organisation or project. In the learning event, grantees discussed how they would make sure that for smaller, especially voluntarily run organisations the evaluation was pared down to make it less labour intensive. By doing so, the evaluation could continue to find important information on the areas that mattered whilst not overwhelming the organisations that conducted them.

Final thoughts

The learning event brought up a lot of interesting and challenging points and questions around evaluations. Whilst we have given four useful tips that were discussed, there is no one right way to conduct an evaluation.

It is essential that whoever is conducting the evaluation should be constantly reflecting on the evaluation itself. Committing to consistent and in depth self-reflection can help to ensure that any evaluation which takes place is meaningful to both the programme and the community involved in it.