Introduction and Purpose
USAID’s Monitoring, Evaluation, and Learning (MEL) Activity implemented by CAMRIS International, Inc. has played an instrumental role in advancing USAID/Nepal and its implementing partners’ (IP) monitoring and evaluation system to the next level. The MEL Activity has been mindfully integrating a collaborating, learning and adapting (CLA) approach in all the tasks it carries out. The mid-term performance evaluation of the HC3 Nepal Project is a demonstrative model of how the CLA approach remained instrumental in understanding the real needs of USAID and its implementing partner for this project, The John Hopkins Center for Communication Programs. The purpose of the evaluation was to assess the project’s performance at the mid-point to determine if any course corrections would be necessary based on the findings to ensure achievement of project objectives. In tracking whether the evaluation recommendations were implemented, the MEL Activity learned that sixteen months following the evaluation, approximately 94 percent of the recommendations had been implemented.
The HC3 Project (2012 – 2017) worked to improve reproductive health outcomes in Nepal through targeted Social Behavior Change Communication (SBCC) activities for youth, adolescents, migrants, and marginalized and disadvantaged groups. The project was implemented through two key partnerships: Nepal’s National Health Education Information and Communication Centre (NHEICC), which manages national SBCC efforts; and Nepal’s Family Health Division (FHD), which oversees supply-side service delivery in family planning. It also sought to strengthen the institutional and technical capacity of NHEICC and FHD to design, implement, and evaluate SBCC for family planning.
Evaluations Questions and Methodology
USAID/Nepal developed the following questions to guide the evaluation:
The team used a mixed-methods data collection approach that included document review and semi-structured interviews with 134 respondents. It conducted 26 focus group discussions with female beneficiaries, male beneficiaries, female community health workers, Health Facility Operation and Management Committee (HFOMC) members, and mothers-in-law. Additionally, the evaluation team observed five counseling sessions between providers and clients. Quantitative data collection focused on reviewing HC3’s monitoring data. The evaluation team limited data collection to four districts (Siraha, Dhading, Kaski, and Banke) and eight village development committees (VDCs) (See the evaluation report for the criteria used to derive the sample: https://goo.gl/i6PTGw).
Map of the HC3 Project Target Districts
Of the many challenges the evaluators faced, the key challenge for this evaluation was to strike a balance between ensuring independence of the evaluation and allowing for meaningful participation of the stakeholders in the evaluation process to generate a feedback loop in every phase.
Key Findings by Evaluation Question
Evaluation Question #1:
Evaluation Question #2:
Evaluation Questions #3:
Evaluation Recommendations and Feedback Loop
The key to getting USAID and the implementing partner to accept the evaluation and its use for adaptive management is to provide realistic and practical recommendations. This was possible due to a rigorous and iterative process adopted during and following the data collection phase. The MEL team collected data throughout the evaluation from interviews with key informants from the Department of Health Services – Family Health Division, NHEICC, and district and village development committees.
Following the data collection, the evaluation team conducted a series of debriefing sessions with the pertinent stakeholders to validate the data, identify the gaps, and reach out to other stakeholders to address the data gaps. This feedback loop not only helped to identify and resolve data gaps and disseminate the learning, but also increased the trust and confidence of stakeholders in the evaluators and the evaluation process. The CLA approach gave power to the evaluators to speak the truth with confidence and increased the chances that the data would be used for learning and program adaptation.