Two recent publications discuss how to effectively assess the impact of transdisciplinary (TDR) research and apply these methods to 5 case studies.
The creation of the CGIAR Research Programs (CRP) was aimed to increase the social, economic, and environmental impacts of research. These programs have intentionally developed broader and deeper partnerships with a wide range of policy and development actors (i.e., international conservation and development organizations, NGOs, policy actors, other stakeholders), as well as with other researchers and research organizations. These efforts mirrored a shift in the broader research environment toward more engaged, problem-centred research. Such research, known variously as Transdisciplinary Research (TDR), Mode 2 Research, and Sustainability Science, among other terms, actively involves stakeholders to help ensure the relevance of the research, incorporate a broader range of expertise in the research process, and promote the co-generation of knowledge with research users.
In theory, engaged TDR approaches should help address complex sustainability problems and contribute to more and better outcomes. However, the increased complexity of these approaches makes impact assessment even more challenging than for traditional research approaches. Research impact assessment is chronically challenged by the fact that the uptake and use of research-based knowledge is incremental, with multiple steps and other intervening factors, often with long time-lags. Measuring and attributing impact are difficult. CGIAR research impact assessment has typically attempted to measure the benefits of improved technologies generated by CGIAR research; this assumes that the main contributions of the research are bundled within an improved plant variety or other technology package. TDR deliberately aims to contribute to several impact pathways simultaneously, by supporting capacity-building and empowerment among partners, facilitating dialogue and political processes, co-generating knowledge that will be implemented directly by partners, as well as through more conventional research products. However, empirical evidence of whether and how transdisciplinary approaches contribute to (more) effective scientific and social outcomes remains limited.
CIFOR Senior Associate Scientist Brian Belcher and his team in the Sustainability Research Effectiveness Program (SRE) at Royal Roads University have developed methods to assess TDR. The SRE Program has also conducted a series of case studies of completed FTA research projects to investigate the link between transdisciplinary research and societal effects. They recently published two papers to share lessons from their work.
“A refined method for theory-based evaluation of the societal impacts of research” (Belcher et al., 2020) provides a detailed description of concepts and a method for assessing the relationship between research processes, outputs, and outcomes. The Outcome Evaluation Approach uses an actor-centred Theory of Change as the analytical framework, and accounts for complexity by recognizing the role of other actors, context, and external processes in change. The article provides stepwise guidance on how to:
- document a theory of change;
- determine data needs and sources;
- collect data;
- manage and analyze data; and
- present findings.
The paper responds to the need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes, in an effort to make research more effective in addressing complex sustainability challenges.
“Linking Transdisciplinary Research Characteristics and Quality to Effectiveness: A Comparative Analysis of Five Research-for-Development Projects” (Belcher et al., 2019) reports lessons from outcome evaluations [1] of five FTA projects. The five projects:
- Brazil Nut Project (BNP)
- Sustainable Wetlands Adaptation and Mitigation Program (SWAMP)
- Fire and Haze Indonesia (F&H)
- Global Comparative Study on Forest Tenure Reform-Peru (GCS-FTR), and
- Support to the Development of Agroforestry Concessions in Peru (SUCCESS)
represent a wide range of research approaches, social and policy contexts, and outcomes. Each case study used the Outcome Evaluation Approach described in Belcher et al. (2020) to document the project’s Theory of Change and assess whether and how outcomes were realized. The analysis also used Belcher et al.’s (2016) Transdisciplinary Research Quality Assessment Framework (QAF) to characterize each project by the degree to which its design and implementation conformed with transdisciplinary criteria.
Each project had a deliberate focus on moving beyond knowledge production to influence policy and practice. To do that, the projects employed a variety of strategies that crossed disciplinary bounds and engaged a range of partners and stakeholders at different levels. The results demonstrate that projects employing more transdisciplinary characteristics make more diverse contributions as they tend to leverage more diverse mechanisms of change. The participation of various system actors contributed to projects’ relevance and strongly contributed to the uptake and use of the research. Projects that invested most in developing and facilitating participation (e.g., the Global Comparative Study on Forest Tenure Reform-Peru and the Support to the Development of Agroforestry Concessions in Peru projects) were the most successful in generating social learning and building coalitions. Projects that employed the most traditional scientific models (e.g., the Brazil Nut Project and the Sustainable Wetlands Adaptation and Mitigation Program) but still invested in outreach and engagement, were able to realize significant outcomes. Research project efforts to support social processes helped translate and broker knowledge outputs and made substantial additional contributions through capacity-building, initiating and supporting discourse, and relationship-building.
Given the results, it is clear that research aiming to influence policy and practice change should consider integrating and reflecting on TDR characteristics more intentionally from the early planning stages and throughout the whole research process. This new Outcome Evaluation Approach will help linking outcomes, outputs and TDR more effectively, justifying the need for more transdisciplinary science, with an increase in overall results and global benefits.
[1] Two individual project outcome evaluation reports have been published (Brazilian Nut, SWAMP), while the others are forthcoming (F&H, GCS-FTR, SUCCESS).
FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with Bioversity International, CATIE, CIRAD, INBAR, ICRAF and TBI. FTA’s work is supported by the CGIAR Trust Fund.
Share your thoughts with us