The philosophy of policy evaluation and social impact
Measuring impact and outcomes
Philosophy is a key aspect of evaluation. Here we are referring to normative ethics, which looks at general moral questions such as ‘what is socially good and valuable?’ and ‘what should organisations do for society?’ This guides our thinking about what it is that is important to society and in turn, it determines what it is we should measure when evaluating policies and social impact.
The main policy evaluation and social impact measurement tools and frameworks in use today include:
Cost-benefit analysis (CBA)
Cost-effectiveness analysis (CEA)
Cost-utility analysis (CUA)
Social return on investment (SROI)
Multi-criteria analysis (MCA)
These methods are based on different philosophical foundations and so it is key to use the one that best fits with an organisation’s objectives.
We have experience and expertise in all forms of social impact analysis and policy evaluation.
An important aspect of our work is to develop the right philosophical approach for our client organisations that aligns with their organisational objectives and values because there is no one-size-fits-all solution to social impact and policy evaluation.
Causal inference – the task of estimating cause and effect relationships – is of utmost importance to policy evaluation and social impact analysis because we need to understand whether our interventions have impacted on the outcomes of interest.
There are two fundamental approaches to causal inference. The Rubin Model is based on the concept of counterfactual outcomes. It compares what did happen against what would have happened anyway in the absence of the programme or intervention (the counterfactual).
The Campbell Approach instead looks at ‘statistical threats’ that would prevent us from concluding whether a relationship that we observe between an intervention and an outcome has a causal interpretation.
Our approach to causal inference is unique because we combine methods from both frameworks in order to find the optimal approach for our clients given the data available. This allows for great flexibility in our approach.
Our statistical research is scientifically rigorous and we regularly publish our research in leading academic journals as well as advise OECD governments on statistical methods for policy evaluation.
Valuation is the process of converting impacts from a divergent range of outcomes on to a single measure or metric. This allows for comparisons of the effectiveness of different interventions in a consistent and systematic way and it is core to social value measurement – valuing outcomes tells us how much they are worth to society.
Valuation can be monetary or non-monetary and it will depend on which social impact framework is employed (monetary valuation is required for CBA and SROI, whereas CUA and MCA use non-monetary valuation techniques).
We are pioneers in the field of valuation and have written guidelines that have become the core of the HM Treasury Green Book guidance manual, the OECD valuation guidelines (2018), as well as guidelines issued by the Governments of Australia (2017) and New Zealand (2019). We have extensive experience in measuring monetary and non-monetary (QALY) values through preference methods, such as contingent valuation and discrete choice models, and through wellbeing valuation methods using the most up to date best-practice in the field. We have undertaken research on valuation in a wide range of areas including housing, community investment, nutrition, health, sports, international development, construction and infrastructure, transport, volunteering, employment, education, crime and arts and culture.
Our research on valuation is at the forefront of the field and many of the current social value approaches in use today by the UK Government and public sector organisations across the OECD were developed by us.