09.07.2018

Integrating Collaboration, Learning, and Adapting (CLA) Approach in Project Evaluation: Health Communication and Capacity Collaborative (HC3) Evaluation

A close-up photo of a child sitting in her mother's lap

Introduction and Purpose

USAID’s Monitoring, Evaluation, and Learning (MEL) Activity implemented by CAMRIS International, Inc. has played an instrumental role in advancing USAID/Nepal and its implementing partners’ (IP) monitoring and evaluation system to the next level. The MEL Activity has been mindfully integrating a collaborating, learning and adapting (CLA) approach in all the tasks it carries out. The mid-term performance evaluation of the HC3 Nepal Project is a demonstrative model of how the CLA approach remained instrumental in understanding the real needs of USAID and its implementing partner for this project, The John Hopkins Center for Communication Programs. The purpose of the evaluation was to assess the project’s performance at the mid-point to determine if any course corrections would be necessary based on the findings to ensure achievement of project objectives. In tracking whether the evaluation recommendations were implemented, the MEL Activity learned that sixteen months following the evaluation, approximately 94 percent of the recommendations had been implemented.

Project Background

The HC3 Project (2012 – 2017) worked to improve reproductive health outcomes in Nepal through targeted Social Behavior Change Communication (SBCC) activities for youth, adolescents, migrants, and marginalized and disadvantaged groups. The project was implemented through two key partnerships: Nepal’s National Health Education Information and Communication Centre (NHEICC), which manages national SBCC efforts; and Nepal’s Family Health Division (FHD), which oversees supply-side service delivery in family planning. It also sought to strengthen the institutional and technical capacity of NHEICC and FHD to design, implement, and evaluate SBCC for family planning.

Evaluations Questions and Methodology

USAID/Nepal developed the following questions to guide the evaluation:

  1. How well have SBCC materials and activities reached youth, migrants, and MDAGs? Has there been enough supply to meet demand?
  2. To what extent have HC3’s activities contributed to improved capacity to coordinate, design, implement, monitor, and evaluate SBCC within the NHEICC and FHD and among USAID partners?
  3. What evidence has HC3 generated that contributed to the design and implementation of their SBCC campaign and demand for FP, and how is the evidence being used by stakeholders?

The team used a mixed-methods data collection approach that included document review and semi-structured interviews with 134 respondents. It conducted 26 focus group discussions with female beneficiaries, male beneficiaries, female community health workers, Health Facility Operation and Management Committee (HFOMC) members, and mothers-in-law. Additionally, the evaluation team observed five counseling sessions between providers and clients. Quantitative data collection focused on reviewing HC3’s monitoring data. The evaluation team limited data collection to four districts (Siraha, Dhading, Kaski, and Banke) and eight village development committees (VDCs) (See the evaluation report for the criteria used to derive the sample: https://goo.gl/i6PTGw).

Map of the HC3 Project Target Districts

Key Challenge

Of the many challenges the evaluators faced, the key challenge for this evaluation was to strike a balance between ensuring independence of the evaluation and allowing for meaningful participation of the stakeholders in the evaluation process to generate a feedback loop in every phase.

Key Findings by Evaluation Question

Evaluation Question #1:

  • Social norms remain a barrier to increasing the demand for FP.
  • The peer facilitator (PF) model is unsustainable as the HC3 Project covers 100 percent of the PFs’ salaries.
  • Information, education, and communication (IEC) materials are not reaching all campaign messengers.
  • Among adolescents, there is a high number of unplanned pregnancies and low level of FP knowledge.
  • The supply of contraceptives is unable to meet demand.

Evaluation Question #2:

  • District-level implementing partners (IPs) stated that the campaign’s top-down approach leaves them out of the campaign’s design and planning stages, as the HC3 central office develops the annual work plan without input from district level IPs who could help ensure local needs are addressed during project implementation.
  • Central and district level staff do not provide sufficient technical support to district health offices (DHOs) to effectively build their capacity to implement SBCC activities.

Evaluation Questions #3:

  • The national formative assessment excluded minorities and hard-to-reach audience segments, which resulted in a critical information gap regarding the district-level realities facing those groups.

Evaluation Recommendations and Feedback Loop

The key to getting USAID and the implementing partner to accept the evaluation and its use for adaptive management is to provide realistic and practical recommendations. This was possible due to a rigorous and iterative process adopted during and following the data collection phase. The MEL team collected data throughout the evaluation from interviews with key informants from the Department of Health Services – Family Health Division, NHEICC, and district and village development committees.

Following the data collection, the evaluation team conducted a series of debriefing sessions with the pertinent stakeholders to validate the data, identify the gaps, and reach out to other stakeholders to address the data gaps. This feedback loop not only helped to identify and resolve data gaps and disseminate the learning, but also increased the trust and confidence of stakeholders in the evaluators and the evaluation process. The CLA approach gave power to the evaluators to speak the truth with confidence and increased the chances that the data would be used for learning and program adaptation.

Recommendations

  • Amplify the voice of satisfied users and PFs on local FM stations; shine a light on potentially harmful traditional beliefs and practices through discussions via the local media.
  • Consider organizing men’s groups and/or groups of elders/in-laws, modeled after the mothers’ groups to help break down cultural barriers to FP.
  • Expand social mobilization for FP through HFOMC members, forest user groups, citizen awareness centers, ward citizen forums, cooperatives, and other influential groups.
  • Continue to advocate with interested VDCs for a budget allocation for PFs and explore with VDCs other low-cost options for replicating the PF model.
  • Continue working with government and community-based partners to improve efforts to distribute IEC materials.
  • Strengthen and increase school-based SBCC programs.
  • Continue to coordinate with FP supply projects, DHOs, and FHD counterparts on family planning promotion.
  • Improve the capacity of local DHO staff to design, plan, and implement SBCC activities
  • Consider a discretionary district fund to provide more autonomy at the district level to design and deliver SBCC activities.
  • Conduct an additional formative assessment of minority and hard to reach audiences and use the findings to develop more tailored local strategies targeting hard-to-reach populations.