Opinion: The Limits of Evidence-Based Policy-Making

Laura Hache´

This fall, the Social Impact Analysts Association will hold its third annual conference in Toronto. Entitled “Talking Data: Measurement with a Message”, the event will bring evaluation professionals together for discussions on the impact and value of social policy evaluation methodologies. A number of the newer practices in this field – notably the ‘Social Return on Investment‘ – are re-shaping the way that policy-makers, the social sector, and the general public perceive and respond to social policies.

Increasingly, the evaluation of social policy is determined by the measurement of social outcomes – lending to a practice of evidence-based policy-making. It follows that the methods we use to measure these outcomes is significant to the direction of our policy objectives. Numbers are the foundation on which we propose new ideas, and policy-makers today are increasingly asked to show the hard, quantified evidence to justify the need for a new program – or perhaps, the need to eliminate one. In everyday life however, we tend to be more influenced by experiences rather than data sets.

This commitment to quantifiable evidence and the desire to measure the impact of policy interventions has taken hold in the non-profit and social policy sectors in recent years – sectors which are often driven by immeasurable goals. Non-profit organizations are increasingly expected to provide data for more and more of their work to justify funding, or in order to request funding from government, for-profit businesses, and/or everyday Canadians.

Quantification in social policy evaluation is not entirely without merit. Consider a program that aims to lift youth from vulnerable communities out of poverty. Funders will want to know exactly how many disadvantaged youth received job-search training, for example, and how many are expected to apply to post-secondary institutions. The expectation to develop and manage new performance measurement indicators and provide large amounts of data constrains the already limited resources of non-profits. And when funders begin to expect quantitative data for hard-to-measure outcomes such as social cohesiveness, sense of belonging, or cultural diversity, these demands start to tip the scale of feasibility.

Sure, to measure something like cultural diversity at an event, you could count the proportion of Canadian-born individuals to non-Canadian-born individuals in the room – but does this rate really get at the root of the goal of cultural diversity? Are those people really connecting with each other? The supposed validity and objectivity of this type of measurement is suspect.

Encouraging social purpose organizations to think in terms of tangible outputs and breaking down their mission into measurable goals is by no means detrimental, but we should tread lightly when seemingly abstract standards of evaluation such as a degree of “cultural diversity” are used to determine the allocation of funds. The desire to measure such “soft” outcomes is what has inspired the development of new outcome measurement frameworks such as Social Return on Investment and Social Accounting and Audit. These methods are predominantly practiced in the non-profit and social enterprise sectors, and attempt to systematically assign financial proxies to social outcomes to determine social impact in comparison to cost. Although intriguing, these processes may overstate their validity in their attempt to quantify social impact.

The premise that we can boil complex social problems down to one golden ratio to assess social benefits against costs is without doubt naïve. The social world is far messier and far less governed than numbers would allow us to express.

No one would contest that basing policies on facts is a sound practice, but the design and selection of those facts can easily be skewed. Nancy Cartwright and Jeremy Hardie’s 2012 book Evidence-Based Policy: A Practical Guide to Doing it Better provides a formidable list of ways that assumptions of transferability can go wrong. Cartwright is not against evidence-based policy making, but warns that we are not taught or expected to consider seriously a theory of evidence. Stories, anecdotes, and qualitative research methods are crucial to ensuring effective evaluation.

The best-case scenario in uncovering “evidence” may in fact be to adopt a holistic approach to decision-making that balances both qualitative and quantitative methods with sound judgment of that data. We should seek to develop a deeper understanding of the communities and social groups that are the subject of our social policies. This approach is currently practiced, for example, in the Network Environments for Aboriginal Health Research  (NEAHR), which aims to enhance the research environment between universities and First Nations, Metis and Inuit communities by developing a trusting relationship between researchers and the intended users of the research – policy makers, organizations, and communities.

Our social world is fraught with phenomena that defy categorization and order, and it has become increasingly clear that we have an over-inflated sense of what we can measure. We have also come to define causal relationships much too quickly. Cartwright would argue that our experience of life is entirely subjective – and like her, we should be critical of any apparatus that claims to “objectively” measure the world. As we are increasingly instructed to make evidence-based policy decisions, the scope of what qualifies as “evidence” must be broadened to reflect our lived experience.

Laura Haché is a 2015 Master of Public Policy Candidate at the School of Public Policy and Governance and has worked on many collaborative projects in the non-profit sector, most recently with Jane’s Walk and Civic Action. Laura is passionate about social policy, gender equity, and encouraging civic literacy.

13 Comments Add yours

  1. Ian Clark says:

    Dear Matteo and Lindsay,

    Congratulations again on the look and feel of the PPGR. The banner on the home page with the featured posts is fantastic.

    The only semi-urgent to-do that I can see is making the articles of Volume 5 Issue 1 and Issue 2 available on the past issues page https://ppgreview.ca/past-issues/

    From my computer here on the west coast, nothing comes up when I click those two buttons.

    I look forward to seeing you next week around the School.



    Ian D. Clark
    School of Public Policy and Governance
    University of Toronto http://www.ian-clark.ca @IanClark9
    Room 316, 14 Queen’s Park Cres. W., Toronto, ON, M5S 3K9
    Phone 416.978.2841 Fax 416.978.5079
    Cell 416.727.1226 id.clark@utoronto.ca

    1. PPGR says:

      Hi Ian,

      We are currently working to fix the issue with Volume 5, Issues 1 and 2. We hope to have those available online within the next few days.



  2. Patricia says:

    It’s difficult to find knowledgeable people for this topic, however, you seem like you know what you’re talking about!

  3. Ralph says:

    Appreciating the dedication you put into your site and
    in depth information you present. It’s nice to come across a
    blog every once in a while that isn’t the same outdated rehashed information. Wonderful read!
    I’ve saved your site and I’m adding your RSS feeds to my Google

  4. m88 says:

    You actuallly make it seem so easy with your presentation but I ind this topic too be
    really something which I think I would nevr understand. It seems
    tooo complex and very broad for me. I am looking forward
    for your next post,I’ll try to get the hang
    of it!

  5. Antoine says:

    Once I realized what it waas doing, it was simple to just double everything accordingly which made the figures more in keeping with the Charge and Fllex trackers.

    I am going to discuss 3 big points, wireless networking in laptops, desktops, and also
    tthe software on these machines. Some off thhe most important VIP lounges in the airport
    are Cathay Pacific Airlines, Thai Airways
    and Singapore Airlines.

  6. This function was initially component of the packet radio system, but subsequently became a separate plan in its own suitable.

  7. Obejrzyj says:

    For hottest information you have to pay a visit world-wide-web and
    on internet I found this site as a most excellent web site for newest updates.

  8. We built this Web site Speed Test to aid you analyze the load speed of your sites and study how
    to make them quicker.

  9. Thank you for another informative site. Where else may just I get that kind of information written in such a perfect manner?
    I’ve a challenge that I am just now working on, and I’ve been at the glance out
    for such information.

  10. Mellissa says:

    Hi, every time i used to check blog posts here in the early
    hours in the break of day, since i love to find out more and more.

  11. Why users still use to read news papers when in this technological
    world everything is presented on net?

  12. Woah! I’m really digging the template/theme of this blog.
    It’s simple, yet effective. A lot of times it’s tough to get that “perfect balance” between usability and visual appearance.

    I must say you’ve done a superb job with this. In addition, the blog loads very fast for
    me on Chrome. Excellent Blog!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s