Laura Hache´

This fall, the Social Impact Analysts Association will hold its third annual conference in Toronto. Entitled “Talking Data: Measurement with a Message”, the event will bring evaluation professionals together for discussions on the impact and value of social policy evaluation methodologies. A number of the newer practices in this field – notably the ‘Social Return on Investment‘ – are re-shaping the way that policy-makers, the social sector, and the general public perceive and respond to social policies.

Increasingly, the evaluation of social policy is determined by the measurement of social outcomes – lending to a practice of evidence-based policy-making. It follows that the methods we use to measure these outcomes is significant to the direction of our policy objectives. Numbers are the foundation on which we propose new ideas, and policy-makers today are increasingly asked to show the hard, quantified evidence to justify the need for a new program – or perhaps, the need to eliminate one. In everyday life however, we tend to be more influenced by experiences rather than data sets.

This commitment to quantifiable evidence and the desire to measure the impact of policy interventions has taken hold in the non-profit and social policy sectors in recent years – sectors which are often driven by immeasurable goals. Non-profit organizations are increasingly expected to provide data for more and more of their work to justify funding, or in order to request funding from government, for-profit businesses, and/or everyday Canadians.

Quantification in social policy evaluation is not entirely without merit. Consider a program that aims to lift youth from vulnerable communities out of poverty. Funders will want to know exactly how many disadvantaged youth received job-search training, for example, and how many are expected to apply to post-secondary institutions. The expectation to develop and manage new performance measurement indicators and provide large amounts of data constrains the already limited resources of non-profits. And when funders begin to expect quantitative data for hard-to-measure outcomes such as social cohesiveness, sense of belonging, or cultural diversity, these demands start to tip the scale of feasibility.

Sure, to measure something like cultural diversity at an event, you could count the proportion of Canadian-born individuals to non-Canadian-born individuals in the room – but does this rate really get at the root of the goal of cultural diversity? Are those people really connecting with each other? The supposed validity and objectivity of this type of measurement is suspect.

Encouraging social purpose organizations to think in terms of tangible outputs and breaking down their mission into measurable goals is by no means detrimental, but we should tread lightly when seemingly abstract standards of evaluation such as a degree of “cultural diversity” are used to determine the allocation of funds. The desire to measure such “soft” outcomes is what has inspired the development of new outcome measurement frameworks such as Social Return on Investment and Social Accounting and Audit. These methods are predominantly practiced in the non-profit and social enterprise sectors, and attempt to systematically assign financial proxies to social outcomes to determine social impact in comparison to cost. Although intriguing, these processes may overstate their validity in their attempt to quantify social impact.

The premise that we can boil complex social problems down to one golden ratio to assess social benefits against costs is without doubt naïve. The social world is far messier and far less governed than numbers would allow us to express.

No one would contest that basing policies on facts is a sound practice, but the design and selection of those facts can easily be skewed. Nancy Cartwright and Jeremy Hardie’s 2012 book Evidence-Based Policy: A Practical Guide to Doing it Better provides a formidable list of ways that assumptions of transferability can go wrong. Cartwright is not against evidence-based policy making, but warns that we are not taught or expected to consider seriously a theory of evidence. Stories, anecdotes, and qualitative research methods are crucial to ensuring effective evaluation.

The best-case scenario in uncovering “evidence” may in fact be to adopt a holistic approach to decision-making that balances both qualitative and quantitative methods with sound judgment of that data. We should seek to develop a deeper understanding of the communities and social groups that are the subject of our social policies. This approach is currently practiced, for example, in the Network Environments for Aboriginal Health Research  (NEAHR), which aims to enhance the research environment between universities and First Nations, Metis and Inuit communities by developing a trusting relationship between researchers and the intended users of the research – policy makers, organizations, and communities.

Our social world is fraught with phenomena that defy categorization and order, and it has become increasingly clear that we have an over-inflated sense of what we can measure. We have also come to define causal relationships much too quickly. Cartwright would argue that our experience of life is entirely subjective – and like her, we should be critical of any apparatus that claims to “objectively” measure the world. As we are increasingly instructed to make evidence-based policy decisions, the scope of what qualifies as “evidence” must be broadened to reflect our lived experience.

—

Laura Haché is a 2015 Master of Public Policy Candidate at the School of Public Policy and Governance and has worked on many collaborative projects in the non-profit sector, most recently with Jane’s Walk and Civic Action. Laura is passionate about social policy, gender equity, and encouraging civic literacy.