Qualitative and Quantitative Evaluation
Qualitative methods help you understand shifts in perceptions, beliefs, behaviours and are most often collected through interviews, observations and focus groups.
Quantitative methods emphasise objective measurements and statistical or numerical data analysis to understand outputs and outcomes of your initiative. These data are most often collected through polls, questionnaires or surveys.
Depending on what you need to know, both qualitative and quantitative methods are useful and most evaluators use a combination of the two.
With more people consuming and sharing video online, gathering quantitative data has become increasingly easy. Tracking likes, hits, clicks and more can be provided by social media platforms or for your own website by systems such as Google Analytics or the open source alternative Matomo.
But be careful that you are not seduced by data alone. It can be manipulated to say whatever you want it to say, and it largely tells you “what” happened but is somewhat limited in explaining “why” or “how” it happened.
Shifts in beliefs and behaviours are also more relevant to questions of human rights and social impact than the question of how many people “viewed” or “liked” your film.
For these reasons the Video for Change approach tends to prefer qualitative methods, or ensures that quantitative data is well informed by other qualitative information.
There’s been much debate in the documentary and impact field regarding the right balance of quantitative versus qualitative measurement. We surveyed Video for Change practitioners to find what their mix and match looks like.
There are quite a few different methods used and most practitioners use at least two or three. You can read the analysis of the survey results here.
The various methods outlined below provide opportunities for further knowledge exchange and the development of individual and collective capacities. This capacity development keeps the momentum going from the impacts of the initiative and prepares the ground for new collaborations that may emerge.
An impact story aims to communicate a deep understanding about what has happened over the course of your Video for Change initiative, and how and why it happened. Drawing on the documentation you’ve collected during each stage of your initiative, the impact story combines your evaluation results to create a compelling narrative about how the initiative has affected the various stakeholders and communities involved.
An impact story can make use of a wide variety of media, including infographics, survey results, photos, writing, audio and, of course, video. Your narrative should highlight moments where real change took place, or where individuals changed their behaviour or mindsets.
An impact story should also take into consideration your original objectives: Did you meet them? How? Where did the initiative fall short of expectations or plans? Were there unexpected positive or negative outcomes?
These are some examples of impact stories. To learn how to create effective impact stories, see Resources.
A written or online survey allows you to ask for individual impressions and detailed data through specific questions addressed to community members, audience members, partners and stakeholders. Responses can be confidential or anonymous if appropriate. To create surveys, see Resources.
Focus Group Discussions
A focus group discussion brings together key thinkers and stakeholders on your issues in order to discuss the initiative. They help you come up with ideas, give feedback on tactics and provide original insights grounded in their own knowledge and experience.
This method helps you collect a range of varied perspectives, which can contribute to richer design or evaluation stages, or both. Focus groups are also an effective way to build community and to create buy-in and ownership.
Assessing Participation in the Video-Making Process
This method incorporates the experiences of people who’ve engaged in your project, ensuring their feedback is counted and considered in your evaluation. Worksheet PDF
‘Most Significant Change’
A good impact story incorporates the stories of participants and people who’ve been affected by your video project. ‘Most Significant Change’ is a well-recognised qualitative method that creates and analyses personal stories of change and evaluates which was most significant to a given set of stakeholders and why.
It lets people tell their own story of how your initiative affected them, instead of only measuring impact with a set of limited indicators devised without their participation. Rick Davies created this method, which is also used to understand the impacts of an initiative, including those that may have been unexpected or unplanned.
See the Resources section for the user’s guide he created with Jess Dart, an explanatory video by InsightShare and more tools for using this methods.