Reflections From NCAN’s Idea Incubators on Data and Evaluation

August 23, 2016

By Bill DeBaun, Director of Data and Evaluation

This summer, NCAN held a series of four “Idea Incubators” to tackle tough problems in data and evaluation, which we developed based on a strong belief in the cross-pollination of problem solving, resources, and practices that occurs whenever NCAN members come together. The sessions were extremely successful in bringing together college access and success programs from around the country to identify the need for guidance in key areas and to provide a roadmap for future resources NCAN can provide. Interest from members was high, with the number of applications greatly exceeding the number of scholarships we were able to offer.

Degrees of Change hosted the first incubator, “Incorporating Fit and Match Into College Advising and Program Data.” Seven NCAN members attended to tackle questions such as, “What are the implications for staff training and development for incorporating fit and match?” and “What are the implications for program-institution partnerships for promoting good fit and match?” “Match,” which is generally regarded as the degree to which a student’s academic capacity is well-reflected in the academic rigor and offerings of the institution to which they matriculate, is relatively easier to capture than “fit,” which tends to encompass a number of qualitative dimensions. Members discussed the ways in which they classify campuses as good matches or fits for particular students, with a specific focus on what robust on-campus student supports look like and how these supports could be codified to better assess how well-suited a given campus is to receive low-income, first-generation students. Attendees suggested an NCAN toolkit on fit and match and continued research on how other members handle this question as avenues for future resources.

“Managing the Database Lifecycle and Ensuring Quality Data” took place at College Forward in Austin. Eight members talked about how to make data systems meet the needs of various levels of staff, how to determine which features of a data system are the most essential for an organization, and how to promote staff buy-in of data entry and analysis, among other topics. Many of these points dovetail with NCAN’s Data and Evaluation Toolkit. Of particular note from this incubator is how difficult making the transition between data systems is for many members. The financial, technical and personnel aspects of making that move are some that NCAN will look to provide more guidance on in the future.

The SEED Foundation in Washington, D.C. held the third incubator, “Leveraging Data and Analysis for Program Improvement, Inquiry, and Growth.” This session focused heavily on issues that have been covered in the three-year Common Measures Learning Community project, in which 20 members have participated. Key topics of discussion at SEED were how to overlay metrics on top of an organization’s logic model to identify gaps in data collection and use. Designing data systems to be able to collect and analyze dosage data was also a focus; these data can help to identify how many “touches” students need to have a good chance of achieving a desired outcome (e.g., college enrollment, persistence or completion). Finally – and this is a big question – members asked, “Once you have data, how do you make decisions with it?” Members in this incubator were at all stages of data collection and expertise, which made for a robust conversation that spanned the entire breadth of member capacity.

The Idea Incubator series concluded with a trip to Iowa College Aid in Des Moines in mid-August. Its focus was “Collecting and Reporting Metrics across Organizations, Networks, and Collaboratives.” With so many NCAN members engaging external partners and participating in collective impact efforts, getting multiple organizations on the same page in terms of how to define metrics, which metrics to collect, and when, how and what to report is incredibly difficult. Beyond discussing best practices in data collection (hint: start small, with whatever metrics everyone is willing to report, and report consistently and build from there), there was also substantial conversation around adding prescriptions on collecting and reporting enrollment, persistence and completion rates to NCAN’s Common Measures to promote standardization and cross-comparison among organizations. This incubator came with a bonus: Democratic presidential nominee Hillary Clinton made a campaign stop nearby, and attending NCAN members snagged selfies with her!

NCAN’s annual conference and Spring Training series suggested that there was a lot of potential value in gathering members together to ponder complex topics. This Idea Incubator series did not disappoint. NCAN members who were not able to attend but who are interested in the topics covered can expect to see white papers this fall with resources and insights from each of these four sessions.

Back to Blog

Leave a Reply:

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License