We’re excited to share that we are starting a new series on the blog. Our Data By Design series will include deep dives from our data team here at Social Solutions and a “behind the scenes” look at how we use data.
To kick off our Data By Design series, our Lead Data Scientist, Adriana Beal, is sharing the fundamental and profound difference between output and outcome.
Adriana Beal is a native of Brazil and has lived in the U.S. since 2004. Her background is in Electrical Engineering, Strategic Management of Information Systems, and Data Analytics. Prior to joining Social Solutions, she worked as a product manager for analytics products in healthcare, social media management, and cloud management, and as a data scientist in charge of machine learning models.
On the Fundamental and Profound Difference Between Output and Outcome
In a recent TED talk, the cofounder of an organization that offers a kindergarten readiness program for 4-year-old children and their families described their success with the following statement:
“Now, we know this is working because we have a 90-percent completion rate for the program.”
Can we accept the “completion rate” as evidence that the program is meeting its goals? Clearly the answer is no. Graduating from a program (with diplomas, even!) doesn’t mean the program “worked” — i.e., generated the desired outcome. In the excellent book The Nonprofit Outcomes Toolbox: A Complete Guide to Program Effectiveness, Performance Measurement, and Results, author Robert Penna provides the following definition:
An outcome is the direct, intended beneficial effect on the stakeholders or interests our organization exist to serve. For example, the outcome of a smoking cessation program is the number or percentage of those completing the program that actually stop smoking.
The proof that a job training program “is working” is not a high percentage of participants completing the program (a measure of output), but rather a high percentage of program graduates who get a desired position and keep the job (a measure of outcome). Likewise, the proof that an at-home kindergarten readiness program “is working” is not a high completion rate, but rather a high rate of program graduates who go on to start school with the same skills to succeed displayed by children who attended traditional PreK programs.
Granted, measuring the outcome or effect is not as easy as measuring how many children graduated from a program. But while not as easy to track, it’s the former what provides tangible evidence that a nonprofit is delivering on its promises.
To be fair, the kindergarten readiness project mentioned in the TED Talk is measuring not only outputs like completion rates, but outcomes as well. As part of their assessment of direct outcomes, they measure the percentage of children who complete the program the year before kindergarten and arrive at school ready to learn compared to the average nationwide and the average for low-income children. And as part of their assessment of indirect, long-term outcomes, they also measure how the children they served perform compared to their peers on state standardized tests in literacy skills all the way through fourth grade.
Still, too many nonprofits lack reliable data on whether the work they are doing is having the desired impact in the populations they serve. In fact, the inability to gather usable outcome data is remarkably common in the social sector.
And this is what gets us in the Data Science team up in the morning every day. Our core job is to help the nonprofits using Social Solutions products to go beyond measuring activities and outputs, to measuring and managing actual outcomes. We’re going to talk in more detail about what this work entails in future articles in this series, but for now, we invite everyone reading this article to reflect upon this fundamental and profound difference between outcomes and outputs:
- Outcomes are the difference made by the outputs: healthier life, fewer students failing to graduate high school, fewer families experiencing homelessness.
- Without outcomes, there is no need for outputs. Outputs, such as number of participants completing a program, are only a means to an end. If 100% of the participants successfully complete a personal finance workshop, but continue to make the same bad financial choices that prevent them from getting out of debt and building an emergency fund, what is the point of continuing to fund the workshop?
Every day at Social Solutions we’re thinking of the best tools, techniques, and processes to recognize and measure outcomes. Collaborating with the nonprofits we serve to better understand their needs is a big piece of that puzzle. And we couldn’t be any prouder of how, despite all the challenges, issues, and constraints they face, our nonprofits remain eager to go beyond the use of data for reporting and compliance to embrace outcome data and insights into how they can create more value for the populations they serve.