An evaluation is an essential way to use monitoring data to measure whether your disaster risk communication has met its overall objectives and goal, usually at the end of your initiative.
It can also inform learning and improvement — your project's strengths, weaknesses and best practices — ensuring stakeholder transparency, accountability and continuous improvement in your future disaster risk communication and that of others.
Objectives
Did the project meet its objectives?
The following research questions can be adapted for different communication projects.
Reach: How many people/what proportion of the target audience did the content reach? Who was not reached and why?
Understand: Did people understand the content in the way it was intended?
Relevance: Was the content relevant, timely and useful?
Engagement: Did people want to watch/read/listen to the content? Did they tune in/follow regularly? Why or why not?
Trust: Did people trust the content? Why or why not?
Effectiveness and impact: Was the content effective? This will depend on your objectives, but might include:
- Did the content help people feel better informed? Did it help people understand key risk information, including weather forecasts?
- Did the content help people feel better informed? Did it help people understand key risk information, including weather forecasts?
- Did it increase their knowledge? What did people learn about how to prepare for a hazard?
- Did it strengthen people’s perception that there is a real risk?
- Did it lead to discussion about the risk? Did people share information from the content with others?
- Did people take preparatory action as a result of the content?
- Did the content have any unintended impacts at the individual or population group level?
Evaluations can use various quantitative and/or qualitative research methods. Choose a research method that can best answer your research question, and measure progress against your indicators. Also consider the time you have to conduct and analyse the research, your budget, and the complexity of the research — who has the expertise or skills to conduct it, a research agency, an academic or a researcher.
It is useful to look at the existing literature in your country and context to understand what evidence already exists on disaster risk communication. Seeing how other researchers have framed research and survey questions, and their findings and learnings, might help shape your evaluation. In turn, it is good practice to share your evaluation results and analysis to help guide others.
Quantitative research
Randomised controlled trials (RCTs)
Participants are randomly assigned to a control group and a treatment group exposed to the content, then compared. Randomisation reduces bias in examining cause-and-effect.
Quantitative surveys
Best way to learn what proportion of the target audience was reached. Compares exposed vs. unexposed groups and can disaggregate by sex, age or ethnicity.
Baseline, midline and/or endline surveys
Compare results — such as audience understanding — at different points over time.
Longitudinal designs
Repeated measurement of outcomes with the same individuals to assess change over time.
Quantitative research
In-depth interviews & focus group discussions
Detailed perspectives from target audiences — what engaged them, what they recalled and learned, and what encouraged action.
Key informant interviews
Expertise on a topic or field, plus deeper understanding of the community and issues people face.