NPI Evaluation

Brock University is excited to announce its partnership with the Niagara Region to evaluate the impact of ongoing prosperity enhancing initiatives in the Region. This partnership is an example of the University’s growing number of community engagement activities. It is funded by the Local Poverty Reduction Fund (LPRF), which is a $50 million, 6-year initiative created by the Government of Ontario to support innovative, community-driven projects that measurably improve the lives of those most affected by poverty. The transdisciplinary research team is led by Anteneh Ayanso (Goodman School of Business), Darlene Ciuffetelli-Parker (Education), Dawn Prentice (Nursing), Dennis Soron (Sociology), Felice Martinello (Economics), Kevin Gosine (Sociology), Jeff Boggs (Geography), Joyce Engel (Nursing), Michael Busseri (Psychology), Princely Ifinedo (Goodman School of Business), Tiffany Gallagher (Education), and Zachary Spicer (Political Science). Together, these Brock faculty members have a shared expertise in the methods and ethics of community program evaluation and shared commitments to contributing to social justice in Niagara and to creating employment and mentorship opportunities for Brock graduate and undergraduate students. Brock’s Social Justice Research Institute (SJRI) provided pivotal support to the research team by initiating the partnership and facilitating the grant application process.

This LPRF award will be used to evaluate the Niagara Region’s Niagara Prosperity Initiative (NPI)–a program established in 2008 to provide annual investments to poverty reduction and prevention activities. To date, the NPI has funded over $13M to 339 projects delivered by over 83 agencies and organizations to over 100,000 individuals and families across Niagara, with particular focuses on women, single parents and their children, youth, people with disabilities, newcomers, indigenous people, and seniors. Until now evaluation efforts have been limited to individual project outputs. This award allows for a more systematic evaluation of the impact and outcomes of all of NPI projects, as a whole.

In awarding this money, the province is recognizing the uniqueness and significance of the NPI to Niagara. The research will explore the potential of NPI to serve as a model for other regions as more becomes known about the nature of NPI’s impact. As importantly, the research will consider how other poverty reduction approaches may enhance the model of the NPI. Beyond the desired short-term outcome to assess the effectiveness of NPI as a response to poverty, this funding will allow the research team to identify which techniques are most appropriate for evaluating a given program type as well as the NPI administrative and funding process. Thus, the research will help develop the capacity of local NPI-funded groups to evaluate projects that address poverty in Niagara in addition to making recommendations for NPI and similar initiatives elsewhere in Ontario.

Process evaluation explores whether program design has been implemented as it was intended. The NPI process evaluation involves three sub-phases:

Scoping review

This phase will provide baseline data to assess the reliability of indicators and to locate trends in larger context of poverty in Canada, Ontario and Niagara. The scoping review will begin with a review of relevant scholarly and professional literature. After this, we will begin mapping the field of poverty and poverty reduction in Niagara with statistical data, a qualitative environmental scan of publicly available textual and visual data, and social network analysis. The insights gleaned from the literature review and quantitative and qualitative analysis will be used to evaluate NPI indicators.

NPI history, rationale, and objectives

This phase will clarify how NPI responds to context, especially in the form of funding programs for 1-2 years. Qualitative data will be collected from internal and external policy and legal documents and key informant interviews with NPI program developers and administrators.

Long-term consequences of NPI participation

This phase will document intended and unintended impacts on poverty reduction as observed by NPI-funded project leads. Some in-depth interviews with project leads will be conducted by faculty members of the research team, while others will be conducted by fourth-year undergraduate students from Brock University, for whom participation in this evaluation will be an experiential learning exercise.

Impact evaluation investigates the change effected by a program, including intended impacts and unanticipated effects. Impact evaluation tells us whether a program is living up to its mandate, but also delves deeper to determine what is working and what isn’t and how its costs compare to its benefits. The NPI impact evaluation involves four sub-phases:

Quantitative comparison of expected and actual project outputs 

This phase will make use of NPI’s internal quantitative data, such as funding allocations, numbers and types of projects funded, and project outcomes. This quantitative data will be used to model and test NPI’s performance between 2008 and 2020 and mined to generate insights that may be leveraged to optimize program operations in years to come.

Qualitative comparison of expected and actual project outputs

This phase will make use of NPI’s internal qualitative data, especially narrative testimonials collected from service users of NPI-funded projects. Narrative analysis and text-mining methods will be applied to illuminate the meanings NPI projects have for service users and to inform recommendations for shaping the symbolic and interpersonal dimensions of NPI-funded projects.

Quasi-experimental subjective well-being study

This phase will employ a quasi-experimental non-equivalent control group design to investigate the effects of participation in NPI-funded programs on the subjective well-being scores of service users. Surveys measuring the material, communal, and emotional subjective well-being will be administered to service users before and after NPI interventions, and to a random sample of low-income Niagara residents that will be used to control for the influence of other variables.

Qualitative evaluation of literacy programs

This phase is a focused evaluation of literacy programs funded by NPI. In-depth interview with project leads and focus group interviews with project staff members and service users will explore their needs, the extent to which those needs are being met, and how NPI-funded projects might better meet their needs.

Photovoice Project Evaluation

This phase entails an examination of NPI’s photo reporting procedures. Photovoice methodology will be combined with ethnographic field work to evaluate current and develop and test new photo reporting practices that are informed by current research and encourage client participation.

Reporting Evaluation

This phase is devoted to assessing the qualitative and quantitative reporting and impact measurement practices of NPI and NPI-funded projects. Mixed methods and comparative analysis will be employed to identify the strengths and weaknesses of current practices, as well as means of augmenting them to create effective feedback loops.

RFP and Funding Process Evaluation

This phase examines the ways NPI allocates funding, starting from the RFP and application instructions, through the application process, to review meetings where in funding decisions are made. This analysis is multidimensional, drawing on qualitative methods to determine how these processes are experienced by participants, structured observations of practices, business process analysis, and comparisons to other organizations that fund poverty reduction.

Needs assessment entails systematically comparing the way things are to the way we’d like them to be to identify the needs that must be met or gaps that must be filled to get from ‘here’ to ‘there.’ The NPI needs assessment consists of one multi-dimensional phase:

Review and assess existing poverty reduction strategies

After synthesizing research findings about best practices in local poverty reduction, data collected in other phases, and in-depth interviews with administrators from other Canadian poverty reduction programs, we will use comparative and other analytical approaches to test and refine findings from phase 2 and provide suggestions on how to modify NPI and its larger policy environment to augment its efficacy.

Evaluation of Niagara Region poverty reduction strategy

Leverage data collected during the evaluation to model the nature and distribution of poverty in Niagara to inform the Niagara Region’s poverty reduction strategy