The NPI Evaluation team is excited to announce the release of Connecting the Pieces: An Evaluation of the Niagara Prosperity Initiative and Call for a Broader Poverty Reduction Strategy for Niagara (2021).
This report is the culmination of a partnership with the Niagara Region to evaluate the impact of ongoing prosperity enhancing initiatives in the Region. This partnership was an example of the University’s growing number of community engagement activities. It was funded by the Government of Ontario’s Local Poverty Reduction Fund (LPRF), which was a $50 million, 6-year initiative (2014 to 2019) to support innovative, community-driven projects that measurably improve the lives of those most affected by poverty.
The transdisciplinary research team was led by Anteneh Ayanso (Goodman School of Business), Jeff Boggs (Geography), Michael Busseri (Psychology), Darlene Ciuffetelli-Parker (Education), Joyce Engel (Nursing), Tiffany Gallagher (Education), Kevin Gosine (Sociology), Princely Ifinedo (Goodman School of Business), Felice Martinello (Economics), Dawn Prentice (Nursing), Mary-Beth Raddon (Sociology), Dennis Soron (Sociology), and Zachary Spicer (Political Science). Together, these Brock faculty members have a shared expertise in the methods and ethics of community program evaluation and shared commitments to contributing to social justice in Niagara and to creating employment and mentorship opportunities for Brock graduate and undergraduate students. Brock’s Social Justice Research Institute (SJRI) provided pivotal support to the research team by initiating the partnership and facilitating the grant application process.
This LPRF award was used to evaluate the Niagara Region’s Niagara Prosperity Initiative (NPI)–a program established in 2008 to provide annual investments to poverty reduction and prevention activities. At the outset of the evaluation in 2018, the NPI had funded over $13M to 339 projects delivered by over 83 agencies and organizations to over 100,000 individuals and families across Niagara, with particular focuses on women, single parents and their children, youth, people with disabilities, newcomers, indigenous people, and seniors. Evaluation efforts had been limited to individual project outputs. This award allowed for a more systematic evaluation of the impact and outcomes of all of NPI projects taken as a whole.
In awarding this money, the province recognized the uniqueness and significance of the NPI to Niagara. The research explored the potential of NPI to serve as a model for other regions. As importantly, the research considered how other poverty reduction approaches may enhance the model of the NPI. Beyond the desired short-term outcome to assess the effectiveness of NPI as a response to poverty, this funding allowed the research team to identify which techniques are appropriate for evaluating a given program type as well as the NPI administrative and funding process. Thus, the research sought to develop the capacity of local groups to evaluate projects that address poverty in Niagara in addition to making recommendations for NPI and similar initiatives elsewhere in Ontario.
The evaluation research design is described in detail below.
Process evaluation explores whether program design has been implemented as it was intended. The NPI process evaluation involves three sub-phases:
Scoping review
This phase will provide baseline data to assess the reliability of indicators and to locate trends in larger context of poverty in Canada, Ontario and Niagara. The scoping review will begin with a review of relevant scholarly and professional literature. After this, we will begin mapping the field of poverty and poverty reduction in Niagara with statistical data, a qualitative environmental scan of publicly available textual and visual data, and social network analysis. The insights gleaned from the literature review and quantitative and qualitative analysis will be used to evaluate NPI indicators.
NPI history, rationale, and objectives
This phase will clarify how NPI responds to context, especially in the form of funding programs for 1-2 years. Qualitative data will be collected from internal and external policy and legal documents and key informant interviews with NPI program developers and administrators.
Long-term consequences of NPI participation
This phase will document intended and unintended impacts on poverty reduction as observed by NPI-funded project leads. Some in-depth interviews with project leads will be conducted by faculty members of the research team, while others will be conducted by fourth-year undergraduate students from Brock University, for whom participation in this evaluation will be an experiential learning exercise.
Impact evaluation investigates the change effected by a program, including intended impacts and unanticipated effects. Impact evaluation tells us whether a program is living up to its mandate, but also delves deeper to determine what is working and what isn’t and how its costs compare to its benefits. The NPI impact evaluation involves four sub-phases:
Quantitative comparison of expected and actual project outputs
This phase will make use of NPI’s internal quantitative data, such as funding allocations, numbers and types of projects funded, and project outcomes. This quantitative data will be used to model and test NPI’s performance between 2008 and 2020 and mined to generate insights that may be leveraged to optimize program operations in years to come.
Qualitative comparison of expected and actual project outputs
This phase will make use of NPI’s internal qualitative data, especially narrative testimonials collected from service users of NPI-funded projects. Narrative analysis and text-mining methods will be applied to illuminate the meanings NPI projects have for service users and to inform recommendations for shaping the symbolic and interpersonal dimensions of NPI-funded projects.
Quasi-experimental subjective well-being study
This phase will employ a quasi-experimental non-equivalent control group design to investigate the effects of participation in NPI-funded programs on the subjective well-being scores of service users. Surveys measuring the material, communal, and emotional subjective well-being will be administered to service users before and after NPI interventions, and to a random sample of low-income Niagara residents that will be used to control for the influence of other variables.
Qualitative evaluation of literacy programs
This phase is a focused evaluation of literacy programs funded by NPI. In-depth interview with project leads and focus group interviews with project staff members and service users will explore their needs, the extent to which those needs are being met, and how NPI-funded projects might better meet their needs.
Photovoice Project Evaluation
This phase entails an examination of NPI’s photo reporting procedures. Photovoice methodology will be combined with ethnographic field work to evaluate current and develop and test new photo reporting practices that are informed by current research and encourage client participation.
Reporting Evaluation
This phase is devoted to assessing the qualitative and quantitative reporting and impact measurement practices of NPI and NPI-funded projects. Mixed methods and comparative analysis will be employed to identify the strengths and weaknesses of current practices, as well as means of augmenting them to create effective feedback loops.
RFP and Funding Process Evaluation
This phase examines the ways NPI allocates funding, starting from the RFP and application instructions, through the application process, to review meetings where in funding decisions are made. This analysis is multidimensional, drawing on qualitative methods to determine how these processes are experienced by participants, structured observations of practices, business process analysis, and comparisons to other organizations that fund poverty reduction.
Needs assessment entails systematically comparing the way things are to the way we’d like them to be to identify the needs that must be met or gaps that must be filled to get from ‘here’ to ‘there.’ The NPI needs assessment consists of one multi-dimensional phase:
Review and assess existing poverty reduction strategies
After synthesizing research findings about best practices in local poverty reduction, data collected in other phases, and in-depth interviews with administrators from other Canadian poverty reduction programs, we will use comparative and other analytical approaches to test and refine findings from phase 2 and provide suggestions on how to modify NPI and its larger policy environment to augment its efficacy.
Evaluation of Niagara Region poverty reduction strategy
Leverage data collected during the evaluation to model the nature and distribution of poverty in Niagara to inform the Niagara Region’s poverty reduction strategy
