News: Data, Research & Evaluation

Impact Survey Project Compares Members’ Outcomes, Costs

Friday, January 10, 2020  
Posted by: Bill DeBaun, Director of Data and Evaluation
Share |

There are a wide variety of college access and success programs, but the professionals working at them often wonder how their program or organization stacks up against their peers. A recent project provides new insights into the success rates of college and career programs and their cost per student outcome.

NCAN members had an 81.3% efficacy rate, on average, in helping students attain the knowledge and skills (both cognitive and non-cognitive) that prepare students to be ready for success in college. The average cost for attaining that outcome among participating members was $2,263, and the median cost was $600. All three of these figures are in line with other programs in the college access and readiness genome portfolio (which also includes the NCAN participants).

These are three findings in a treasure trove of data points resulting from a group of 44 NCAN member programs who participated in the Impact Survey Project, a partnership with Mission Measurement’s Impact Genome Project® (IGP) that occurred over the past six months. This work is generously supported by the Nielsen Foundation.

Figure 1. Key Metrics for NCAN Member Programs from the “College Access and Readiness” Outcome

NCAN and Mission Measurement released the project’s findings during a Nov. 20 webinar.

Announced last May, the Impact Survey Project demonstrates the range of activities NCAN member organizations employ to achieve their outcomes. It also helps to build the capacity of participating programs by providing a standardized framework to organize activities and outcomes, which can be used for communication to funders and among peer programs. Interested members should explore the full portfolio analysis. NCAN members and other organizations in the college access and success field can use this analysis to benchmark their own outcomes and see where they differ in terms of practice from others in the field.

Overall, 44 programs with budgets of totaling over $148 million achieved nearly 169,000 successful outcomes according to the Impact Genome Project’s® framework. Unsurprisingly, the outcome most often pursued by participating programs was “college access and readiness,” (93%), but college persistence (5%) and financial literacy, high school completion, improved academic performance, issue awareness, organizational effectiveness, and organizational reach all make appearances in the portfolio.

The Impact Survey Project’s portfolio analysis also considers the strength of the evidence that participating programs use to measure their outcomes. Evidence’s strength is measured according to RCT or quasi-experimental (Type 1), pre-post or cross sectional (Type 2), point-in-time study (Type 3), performance metrics/stats (Type 4), and anecdotal evidence (Type 5).

Among participating NCAN programs, 27% measure their impact using Types 1-3, the most rigorous levels, compared to 31% of all programs across the Impact Genome Project who do so. The portfolio analysis isn’t representative of all NCAN members, but it does suggest that, where possible, members should consider more rigorous approaches. That said, in the college access and readiness outcome, specifically, the level of evidence used by NCAN members was slightly higher than that of the portfolio as a whole.

The portfolio analysis also compares participating programs on other performance metrics like contact hours and cost-per-outcome per contact hour. All of these are ways of thinking about how efficient a program is at achieving its goals in a timely, cost-effective fashion.

The distribution of programs across cost and performance is ripe for exploration, but so too are the importance programs themselves place on specific activities versus the program’s overall efficacy and cost per outcome. Where a program is placing high emphasis on an activity and yielding high levels of efficacy at low costs per outcome, that program’s practices and implementation may be a valuable model for replication elsewhere. This represents a next step for this work where NCAN and Mission Measurement can drill down and examine specific programs and their practices.

Overall, the participating NCAN member programs greatly expanded the size of the College & Career Readiness Portfolio. These programs’ outcomes will be kept (anonymously) inside of that portfolio so that subsequent participants can compare their own outcomes to those other programs working in this space.

The IGP is a publicly-funded initiative to curate the world’s social impact evidence base, create standardized measures, and analyze benchmarks. The Genome is designed to help organizations improve the way outcomes are measured and reported. By creating a standard terminology, we can systematically compare programs and identify the most effective program activities (aka “genes”) for a given beneficiary and context. These data are then integrated into the Impact Genome platform, which is used by social sector professionals to measure, benchmark, and forecast the impact of social programs. NCAN covered more of the platform’s background in a November blog post.

Readers can explore the IGP evidence base, which consists of hundreds of journal articles, research reports, and other artifacts that have all been carefully coded by the IGP team. Think of it as an incredibly robust literature review that can be narrowed down and sorted. Readers can also can visit the Impact Genome website where they’ll find tools to benchmark their program’s impact and discover activity genes. Readers can even register through the Login button to take the full Impact Genome Survey for free. This will produce a full program scorecard with data readers can use to understand and communicate their program’s impact.

The Impact Survey Project is another in a long line of collaborations and endeavors aimed at helping NCAN member programs better understand how to use data to scale program capacity and improve program performance. Other products in this vein include NCAN’s Common Measures, a set of research-backed, member-developed college access and success indicators, and the Benchmarking Project, which examines the postsecondary outcomes of member-served students using data from the National Student Clearinghouse.