Consultancy: Final evaluation of phase 3 (2006-2010) and overall impact evaluation of the Vietnam Red Cross community based mangrove reforestation/disaster preparedness programme (1994-2010)

1. Summary

1.1 Purpose

The purpose of the evaluation is three fold:
(i) To assess the long term impact (outcomes) of the programme in the communities. It will assess the extent to which the programme has contributed to building more sustainable safety and resilience among the targeted communities.
(ii) To assess the performance and progress achieved (outputs) with respect to the objectives of the Community based Mangrove/Disaster Preparedness Programme Phase
(iii) To analyse the return on investments for both outputs and outcomes through a cost-benefit analysis, aiming to strengthen the cost efficiency of on-going and future programming by providing evidence-based lessons. In particular, it will provide important recommendations for the development of the Vietnam Red Cross Society’s (VNRC) Community Based Disaster Risk Reduction (CBDRR) plan from 2011 to 2015 or further.

1.2 Audience

The primary audience of the evaluation is VNRC and its partners in the community based disaster risk reduction (CBDRR) programmes in the country. The cost benefit analysis will provide evidence based lessons impact analysis of CBDRR approach and would be useful for the Red Cross Red Crescent Movement as a whole to better argue the case for expanded CBDRR programming with institutional donors.

1.3 Commissioners

The final evaluation is a programme requirement, funded under Phase 3 of the Vietnam Red Cross Mangrove Reforestation/Disaster Preparedness Programme in the Red River Delta. It is therefore commissioned by IFRC on behalf of the three parties – VNRC, Japanese RC and IFRC.

1.4 Reporting Line

The evaluation will be guided by a Technical Working Group comprised of key stakeholders from the Japanese RC, VNRC, International Federation of Red Cross and Red Crescent Societies (IFRC) and two Participating National Societies (PNS). The IFRC Vietnam country office will provide, day to day management, budgetary support and manage all contractual relationships.

1.5 Duration

The consultancy will constitute 34 paid days (4 days for preparations, 20 days in Vietnam for meetings/field visits and another 10 days to finalise the report).

1.6 Timeframe

The consultancy will be implemented November - December 2010.

1.7 Location

For phase 3 evaluation, the project provinces of Quang Ninh, Thai Binh, Nam Dinh, Hai Phong, Ninh Binh, Thanh Hoa, Nghe An and Ha Tinh would be considered. For the impact evaluation, a random selection of provinces will be made from entire period.

2. Background

The Community Based Mangrove/Disaster Preparedness Programme started out as ‘The Coastal Environmental Preservation Programme’ funded by Danish Red Cross in Thai Binh from 1994 to 1996, following a request from the Red Cross chapter of Thai Binh province. An external evaluation of the programme in August 1996 confirmed the need and effectiveness of the project in Phase 1 extension from 1997 to 2000 in the provinces of Thai Binh and Nam Dinh. During the same period, the project was extended to six other provinces: Quang Ninh; Hai Phong; Ninh Binh; Thanh Hoa; Nghe An and Ha Tinh, with support from the Japanese Red Cross through the IFRC country office.

Further evaluation of the programme in 2000 recommended that for Phase 2 of the project from 2001 to 2005, the support from the Danish Red Cross and Japanese Red Cross should continue as one integrated project on Mangrove Disaster Preparedness for the VNRC, with a Project Management Board (PMB) established at VNRC headquarters. A joint working group for planning, monitoring and evaluation was created. Project activities during this phase were also expanded from mainly planting mangroves to include other protection trees, as well as providing safer housing and water and sanitation. The focus of capacity building also changed from project management to capacity building on disaster preparedness for the targeted communities and other relevant stakeholders such as teachers, children and government staff. Activities to raise public awareness were carried out from community to national level. A mid-term and a final evaluation were carried out during this phase. The project was well recognized and received by the central government of Vietnam and integrated into their five million hectares reforestation programme.

The Phase 2 evaluations recommended that the VNRC shift from just ‘learning by doing’ to creating learning systems and from ad-hoc decision-making to decisions based on vulnerability and capacity assessments, and strategic disaster preparedness planning. The evaluations also highlighted the need for better coordination and cooperation with the government at central and local level and for stronger fund raising from private sector, government and general public for the continuation of project activities. The Danish Red Cross support ended in December 2005 in Thai Binh and Nam Dinh provinces. The effectiveness of the Mangrove Reforestation/Disaster Preparedness Programme was successfully demonstrated when Typhoon Damrey struck the Red River Delta of Vietnam on 27 September 2005 with wind strength of around 133 kilometres per hour. Only a few deaths were recorded. Several dyke sections with no mangrove protection were broken in Thanh Hoa, Nam Dinh, Hai Phong provinces. Based on this very positive visible impact and the positive evaluation of Phase 2 of the project, Japanese Red Cross extended its assistance through the IFRC for Phase 3 from 2006 to 2010. This phase covered eight provinces: Quang Ninh; Hai Phong; Thai Binh; Nam Dinh; Ninh Binh; Thanh Hoa; Nghe An and Ha Tinh.

A Memorandum of Understanding was signed for Phase 3, together with a detailed project proposal and annual plan and budget (see annexes for details). The project proposal for Phase 3 mandated a mid-term review and a final evaluation. The mid-term review was cancelled because of intensified typhoons and floods occurrence in Vietnam from 2007-2008.

These terms of reference therefore cover both the requirements of the final evaluation and a longitudinal study on the overall impact of the programme since it started in 1994. Outcomes of this evaluation will be used to design a comprehensive VNRC CBDRR programme for the period 2011-2015 or later. This future programme will be part of the wider national effort in Vietnam to improve community level resilience. It will also integrate community actions for climate change adaptation as part of wider disaster risk reduction efforts.

3 Purpose and Scope

3.1 Purpose

The purpose of the evaluation is to evaluate the implementation of phase three (2006-2010) of the Vietnam Red Cross Community Based Mangrove Reforestation/Disaster Preparedness Programme (CBMR/DPP) together with a longitudinal evaluation of the impact of programme over its entire period (1994-2010). The results of the evaluation will be used two primary purposes. One, it would inform the future design of the long term CBDRR programming of the VNRC and two, it would be used as part of International Federation wide evidence and knowledge base on impacts of CBDRR programming over long periods.

3.2 Scope

The evaluation will cover two periods: Phase three (2006-2010) for end of phase evaluation on implementation and will measure a) relevance and appropriateness, b) efficiency, c) effectiveness and d) coverage. The second period (1994-2010) of the evaluation will be for the longitudinal study on a) impact, b) efficiency and c) sustainability of the programme objectives. The phase three evaluation and the wider programme evaluation will be conducted for the eight provinces, where the programme has been implemented since 1994. There will be full involvement of the beneficiaries and VNRC staff during the evaluation at different levels.

4 Objectives and Criteria

4.1 Specific Objectives

The evaluation will have the following specific objectives:
(i) To assess the performance and progress achieved (outputs) with respect to the objectives of the Community based Mangrove/Disaster Preparedness Programme Phase 3 (2006-2010).
(ii) To assess the long term impact (outcomes) of the programme in the communities. It will assess the extent to which the programme has contributed to building more sustainable safety and resilience among the targeted communities during the period 1994-2010.
(iii) To analyse the return on investments for both outputs and outcomes through a cost-benefit analysis, aiming to strengthen the cost efficiency of on-going and future programming by providing evidence-based lessons. The evaluation will document the achievements and shortcomings of the programme as lessons learned for future programming. In particular it will provide important recommendations for the development of VNRC Community Based Disaster Risk Reduction plan from 2011 to 2015 or further.

4.2 Criteria

As defined in the scope above the following criteria will be used during the evaluation.

Relevance & Appropriateness

The evaluation will focuses on the extent to which the CBMR/DPP objectives were suited to the priorities of the addressed target group (local population) and the needs of the donor. It will also consider if other approaches may have been better suited to address the identified needs.

Key questions for relevance & appropriateness:
• Were the programme objectives realistic and consistent with national strategy for natural disaster prevention, response and mitigation to 2020?
• Did the objectives remain relevant over the full implementation period?
• Was the programme design appropriate for achieving the core programmes objectives?
• Did the programme design consider the inputs and needs of key stakeholders including government and the expected beneficiaries?
• Were the targeted vulnerable groups adequately covered and whether they benefited from programme activities (including gender aspects)?
• What are the main factors that contributed to a positive or less positive assessment of relevance?

Efficiency

A cost benefits approach to efficiency will be used to measure the extent to which CBMR/DPP’s results have been delivered in the least costly manner possible. Evaluation will measure whether the results justified the cost, and if alternative approaches to achieving the same results could have been adopted.

Key questions for efficiency:
• How has the programme activities resulted in economic benefits in dyke repair/maintenance, disaster losses and strengthened livelihoods in the targeted communities?
• What were the costs of investments to develop specific programme outputs (e.g. what is the cost of per hectare mangrove plantation and its maintenance)?
• Is the cost benefit ratio of inputs to outputs comparable to other national or international benchmarks?
• What are the administrative costs per beneficiary and how do they compare with other programmes in the country?
• Was there any extensions to the implementation period, if yes, what were the additional administrative costs that were incurred during the extension period?
• What is relative economic contribution of different programme components?
• What key factors help account for project efficiency performance?

Effectiveness

Effectiveness of the phase three will measure the extent to which it has achieved its intended results. This will take into account the CBMR/DPP’s objectives and related indicators. Measure of effectiveness will also include identification of key lessons to inform future CBDRR programmes.

Key questions for effectiveness:
• Were the selected communities better prepared and thus more resilient to disasters after all the vulnerability and capacity assessments, DP training to teachers and children and disaster management planning?
• What is the effectiveness of programme management system at provincial, district and commune levels including the functioning of Programme Management Board (PMB), Programme Working Group (PWG), co-ordination at HQ and eight RC chapters of VNRC?
• What factors in programme design and implementation contributed to its effectiveness?

Coverage

Given that the CBMR/DPP was targeted to reduce disaster risk at the community level, an evaluation of coverage will also be conducted to indentify how the districts and communes were selected and if this meets the stated objectives of the programme.

Key questions for coverage:
• Were the communities selected according to a needs analysis? If yes, how was the final selection made?
• Did the coverage of the programmes cover communities at risk from natural hazards such as flooding, tropical storm and cyclones?

Impact

The long period of the CBMR/DPP allows for the possibility of measuring outcomes, i.e how has the programme outputs over the years increased the resilience of communities directly or indirectly, intended or unintended? Evaluators will design a methodology to assess attribution of programme to observed changes in the communities.

Key questions for impact:
• What has been the long term impact of the programme on disaster risk reduction and capacity building since 1994, in particular the impact of the programme on the coastal environment and socio-economic situation of targeted vulnerable groups to flood and typhoons, poor people selected to involve in project activities?
• Did the household or community level disaster preparedness change?
• Does the livelihood of the people improve from the mangroves?
• To what extent does the community use the early warning information to improve their disaster preparedness and response?
• Has the improved awareness lead to changed behaviour for disaster risk management?
• What external factors promoted or inhibited the achievement of programme objectives and expected results at all levels?

Sustainability

Finally sustainability of the CBMR/DPP will also be measured to check if the benefits of the programme are continued after programme is finished in a particular area. It will pay particular attention to longer-term sustainability of the local capacity and ownership of the programme without donor funding.

Key questions for sustainability:
• How is the organisational capacity built by the programme, including VNRC human resource capacity at all levels for planning, monitoring and evaluation?
• Have the benefits generated by the programme continued after programme closure? If yes, what factors contribute in maintaining the benefits?
• Do programme activities (like maintaining the mangrove plantation) still benefit from the engagement, participation and ownership of local communities?

5 Evaluation Methodology

This evaluation will be participatory in nature and community will be made aware of the evaluation objectives to facilitate their maximum participation. The evaluation will use a range of quantitative and qualitative methods and will not be limited by the suggested options below:
(i) Desktop review - Collect all the key programme documents such as programme design, annual plans, reports, programme agreements, previous evaluations, case studies etc.
(ii) Secondary data collection and review – reports on impacts of disaster events over the programme period will be collected for review. This would include all reports available at national, provincial, district and commune level statistical impact on the resilience of the targeted communities.
(iii) Household survey – Using randomised sampling a household survey (for at minimum 400 households) will be conducted to collect data on the impact of the programme activities. Where baseline household data is absent, the survey will also collect retroactive baselines.
(iv) Focus group discussions – The evaluation will hold focus group discussions in selected (using random sampling) targeted communities to build on the data collected from household survey.
(v) Key informant interviews – Interviews with key partners such as staff in VNRC, IFRC, PNSs, relevant Ministries and institutions (incl. CCFSC, MERC), UNDP Disaster Management Unit, DMWG working in sectors relevant to the programme and selected people in the communities for in-depth understanding of the impact of the programme outputs.
(vi) Direct site visits – The team will take opportunity to make observations of the programme achievements through visits to the communities, planted mangroves and checking disaster risk reduction plans in school and communities. This physical verification of both outputs and visible outcomes (where possible) will be used to supplement other data gathered.
(vii) Case studies – The evaluation will also prepare a set of case studies from beneficiaries to highlight impact, effectiveness and efficiency. The data for the case studies can be collected from other methods mentioned above, especially desk top review, focus groups and key informant interviews.

6. Deliverables

The evaluation will result in the following outputs:
i) A final evaluation of the phase three that is no more than 30 pages (excluding annexes) in length, describing the findings and include the recommendations described earlier.
ii) A lessons learned report that captures the full programme since its inception and is no more than 40 pages (including annexes). This report will also focus on the cost benefit analysis of the programme over time.
iii) A one-day workshop to be organised to share the findings and lessons with relevant stakeholders. The draft report will be presented to the IFRC disaster preparedness manager in Vietnam, who will then forward the draft report to VNRC and other relevant stakeholders for comments. All feedback will be shared with the Evaluation Team Leader who will finalise the report and present the report to the IFRC for approval by the Mangrove/DP PMB. The final reports will be edited and laid out for publication ready quality.

7. Timeline

The study will commence on 1st November 2010 and finish by 31st December 2010 with a total of 45 consultancy days. Detailed timeline available upon request.

8. Team & Qualifications

The evaluation will be lead by an experienced team leader who will have at least 10 years experience of conducting programme evaluations. In addition to the team leader at least following capacities will be requested:
a) A specialist on cost benefit analysis
b) An environmental specialist with minimum 10 years of experience and previous exposure to mangroves plantation programmes.
c) A CBDRR specialist with minimum 8 years of experience in implementing or evaluating CBDRR programmes.
d) Translation facility or ability to speak in Vietnamese.

Additional resource persons may include representatives (to be identified but not limited to) from:
• VNRC PMB/PWG members
• IFRC country delegation/Secretariat
• PNSs working in Vietnam
• Japanese RC HQs representative

9. Management

The evaluation will be guided by a Technical Working Group comprised of key stakeholders from the Japanese RC, VNRC, IFRC and two PNS. The team leader and other consultants will however officially report to Bhupinder Tomar, Country Representative IFRC Vietnam and to Tao Van Dang, DP Manager IFRC Vietnam for practical and technical day-to-day management. The VNRC HQ and IFRC country office will support the evaluation logistics.