Evaluation Methods

Qualitative and Quantitative Evaluation

Qualitative methods help you understand shifts in perceptions, beliefs, behaviours and are most often collected through interviews, observations and focus groups.

Quantitative methods emphasise objective measurements and statistical or numerical data analysis to understand outputs and outcomes of your initiative. These data are most often collected through polls, questionnaires or surveys.

Depending on what you need to know, both qualitative and quantitative methods are useful and most evaluators use a combination of the two.

With more people consuming and sharing video online, gathering quantitative data has become increasingly easy. Tracking likes, hits, clicks and more can be provided by social media platforms or for your own website by systems such as Google Analytics or the open source alternative Matomo.

But be careful that you are not seduced by data alone. It can be manipulated to say whatever you want it to say, and it largely tells you “what” happened but is somewhat limited in explaining “why” or “how” it happened.

Shifts in beliefs and behaviours are also more relevant to questions of human rights and social impact than the question of how many people “viewed” or “liked” your film.

For these reasons the Video for Change approach tends to prefer qualitative methods, or ensures that quantitative data is well informed by other qualitative information.

There’s been much debate in the documentary and impact field regarding the right balance of quantitative versus qualitative measurement. We surveyed Video for Change practitioners to find what their mix and match looks like.

There are quite a few different methods used and most practitioners use at least two or three. You can read the analysis of the survey results here.

Method Examples

The various methods outlined below provide opportunities for further knowledge exchange and the development of individual and collective capacities. This capacity development keeps the momentum going from the impacts of the initiative and prepares the ground for new collaborations that may emerge.

Impact Stories

Impact Story: Empowering Activists

Title: Empowering Activists Through Training

Location: Cambodia

Issue: Expose human rights abuses and support grassroots people’s movements

Aim/purpose of the video: To teach others how to use their mobile phone for video-making.

Organisation: LICADHO, WITNESS and EngageMedia

Featured methodologies: Video for Advocacy, Community Correspondent, Impact Video

Video length: Series of video’s [Cambodian language]

Year: 2015

WITNESS first encountered EngageMedia’s Cambodian partners at LICADHO over a decade ago while working together on a campaign against forced evictions. LICADHO is a grassroots organisation working to expose human rights abuses and support grassroots people’s movements throughout Cambodia.

Since then, thanks to multiple WITNESS trainings and collaborations, the organisation has gone on to train many more human rights defenders across Cambodia and share resources on how to use video for justice and accountability.

LICADHO has translated several of WITNESS’ training videos into Khmer and has begun to use them in its own trainings.

Impacts

Inspired by WITNESS and equipped with high-level production and editing skills learned from WITNESS, Chaly Johnson has started to produce his own video series on filming with mobile phones.

This is the ultimate testament to the strength of our model, proving not only that our individual resources are effective but also that we can empower activists to carry this work forward independently. LICADHO now regularly develops its own materials, some adapted from WITNESS and others not, with far less support.

Thanks to the credibility the organisation has built with its local partners, LICADHO is able to organically grow its own capabilities and share the WITNESS methodology broadly within Cambodia.

The Venerable Loun Sovath (otherwise known as the ‘multimedia monk’), the leader of LICADHO and recipient of the Martin Ennals Award for Human Rights Defenders, states: “LICADHO is so grateful for all the support and guidance we have received from WITNESS, [which] has played a crucial role in forming LICADHO into video activists.”

Impact Story: Pakorpa Susangho

Title: Pakorpa Susangho (Widow’s Cry)

Location: Ghana, West-Africa.

Issue: Widows fight land corruption in Ghana through a participatory video project

Aim/purpose of the video: Achieving changes in unjust laws that deny women their land rights.

Organisation: Joint partnership between the Ghana Integrity Initiative (GII), local partner organisation Widows and Orphans Movement (WOM) and InsightShare

Featured methodologies: Participatory Video, Advocacy Video

Video length: 12 mins.

Year: 2012

In the small village of Kulbia, in the Upper East region of Ghana, 10 widows joined a participatory video project to explore land issues affecting bereaved women in the community. In this village, customary laws dictate that when a woman’s husband dies, the land they both worked on is given to male-headed households.

Film Background

The group of widows documented these customary practices and the impact of landlessness on widows, their dependents and the community as a whole. The widows came together over four months to plan and make their film, which shares their stories and highlights the corruption and abuse that denies the women their livelihood.

The film also shows how the local land custodian, or Tindaana, eventually agrees to support the women’s rights. The video was screened to the public at the local women’s resource centre so everyone could see the commitments made by the Tindaana.

The small village of Kulbia was selected as the location for the participatory video process for the following reasons: the Ghana Integrity Initiative’s (GII) long-standing and trusted relationship with its local partner organisation Widows and Orphans Movement (WOM), the noted prevalence of corrupt practice affecting widows, and the willingness of the village chief to accept and welcome the project taking place within the community.

The three-stage project took nearly four months to complete (July-October 2016), including three weeks of workshops/ fieldwork facilitated by InsightShare during two trips from the UK. GII and WOM undertook a careful selection project, which recruited 10 widows as participants.

The group of widows represented a broad age group (29-59 years), diverse backgrounds and a range of experiences and outcomes following the deaths of their husbands. All of them faced intense pressure on their time and energy — from subsistence farming, income-generating activities, childcare and other factors — however, the project achieved a remarkable 100 per cent attendance across all stages, which gives some indication of the enjoyment and value the participants derived from joining the process.

Process

During intensive workshop sessions, the participants explored how corruption affected widows in their community and resulted in many being left landless and destitute. Many hours were spent telling and retelling personal stories within the safe space of the workshop, which helped build a picture of the different experiences in the group.

The participants worked together to determine the focus and content of their video, to plan sequences showing the challenges they face and, crucially, to demonstrate their awareness of corruption and its role in their landlessness.

Power-mapping exercises helped the women determine who is responsible, who contributes, and who could assist them in combatting corruption. The participants then carefully planned specific messages to reach their target audiences.

Over the following months, participants worked in small groups and travelled to various locations, including their homes and farmlands, to record their personal stories. They recorded discussions with several women from the community and interviewed many customary leaders (including chiefs and Tindaanas) and locally elected officials.

Draft edits of the video were screened and discussed locally, which not only helped to guide and inform the group’s investigation of the issue but also raised awareness of the project and the profiles of the participants within their community. The resulting video titled Pakorpa Susangho (Widow’s Cry) was created during a final workshop incorporating footage from across all stages.

On the final day of the project, a day-long event was organised to premiere the video and to engage key stakeholders in a discussion, including representatives from the House of Chiefs, local government, civil society organisations, local customary leaders, journalists and community members.

The participants also presented the video by describing their personal experiences as widows and the impact on their lives. The screening was followed by discussions, group-work and presentations, Q&A sessions, etc. The participants helped facilitate the process and used the video cameras to document everything, including the many pledges of support made by various officials.

Impacts

Since the video was launched, numerous screenings and discussions have taken place within the community and across the country, helping to raise awareness of the issues and contributing towards broader anti-corruption advocacy campaigns. GII is tracking the commitments by various officials and customary leaders, who are being pressed to realise the changes they agreed to.

GII has trained two community members as paralegals to support women in retaining control over their land, and further training for more women is planned in the near future.

The participants have continued to lobby their traditional leaders to stop the practice of widowhood rites, and are actively supporting women recently widowed to refuse to undergo the degrading rituals. Married women (non-widows) have also been sensitised to the issues — through screenings and informal discussions — and many have asked to be involved in future dialogue around the issues.

The women of Kulbia have reported some successes, including several widows who have avoided the practice of losing their land to male-headed families, and a significant improvement in their relations with fellow community members and overall standing within the community.

“Participatory Video will yield more results than legislative reform when it comes to addressing harmful traditional practices and engaging local leaders and communities to achieve behavioural change,” says Okai Michael Henchard, Ghana Integrity International.

Impact Story: B’Tselem Camera Project

Title: B’Tselem and the Impact of video in turbulent environments

Location: Israel, occupied territories

Issue: Human rights, military violence

Aim/purpose of the video: To bring to justice a soldier who murdered a Palestinian activist.

Organisation: B’Tselem

Featured methodologies: Witness Video, Video for Advocacy

Video length: 3 mins.

Year: 2016

B’Tselem, founded in 1989, is the Israeli Information Centre for Human Rights in the Occupied Territories. It uses video documentation to expose human rights violations and the daily reality under Israel’s military occupation.

In 2007, B’Tselem started its Camera Project in the West Bank and East Jerusalem, which gave legal and technical training to Palestinian volunteers and equipped them with cameras. Over the years, the volunteers’ footage has helped B’Tselem expose the violent nature of Israel’s control and advocate for an end to the occupation.

Film Background

One of the volunteers, Imad Abu Shamsiyeh, lives in Hebron, a Palestinian city where about 800 Israeli settlers live in the midst of its Palestinian population. This reality creates constant friction between the local population and the Jewish settlers and Israeli security forces.

On March 24, 2016 Abu Shamsiyeh documented an Israeli soldier extrajudicially killing a Palestinian who had carried out a knife attack and was lying incapacitated on the road for 11 minutes after being shot and injured by Israeli soldiers.

After verifying the footage was real and seeking the consent from the volunteer and the victim’s family members, B’Tselem published the video, which immediately created a media storm both in Israel and abroad.

It is extremely rare for Israeli security forces personnel to be indicted for offences committed against Palestinians, but the clear cut video left Israeli authorities with no other option.

The media storm continued for many months while the highly unusual trial took place. The soldier, Elor Azaria, was found guilty of manslaughter and sentenced to 18 months of imprisonment. He was released after serving nine months.

Risks and Impacts

For Abu Shamsiyeh there were consequences as well. He and his family have suffered from violence and harassment, a common fact of life in Hebron’s old city centre. After shooting the viral video, Abu Shamsiyeh received death threats with increasing frequency.

When he tried to file a complaint with the Israeli police officers in Hebron, the officers threatened him with arrest. B’Tselem had to temporarily relocate Abu Shamsiyeh and his family to a different area at times when tensions were especially high.

The discourse in Israel about the trial, where large segments of the population sympathised with the shooter, strengthened existing tendencies in Israeli politics to persecute human rights groups and activists.

B’Tselem has been accused of provocation, with some politicians claiming they are purposely setting up incidents in order to film them.

A draft law was recently introduced, seeking to forbid the documentation of Israeli security forces altogether. Whether the draft law will be formally introduced still remains to be seen, but the proposal is a testament to the effectiveness of video documentation.

The widespread international attention of the video contributed to increased exposure of other materials by B’Tselem, which also shed light on the daily violent routine of the occupation. Before March 2016, the Youtube channel of B’Tselem received around 20,000 viewers on a monthly basis. In 2017, this increased to around 500,000 viewers on a monthly basis.

But in the aftermath of Abu Shamsiyeh’s video, B’Tselem has become even more aware of the importance of the environment in which a film is published.

Ehab Tarabieh, B’Tselem’s video department director, explains that if you want to use video to document and publicise human rights offences you “must understand the law of the region you are working in”.

In some cases, the justice system forms the only way a human rights organisation can protect its volunteers or other community members. Therefore connections with lawyers or people that understand the law are vital components within risk management.

This B’Tselem’s case study illustrates that the political and legal environment can greatly affect the impact of a video and that one should be prepared to face these risks. Using video can also cause unexpected changes in this environment, such as the potential new law that forbids taking of footage of any Israeli soldier.

The filmmaker and his family have returned home safely after hiding. Abu Shamsiyeh has become an even more active volunteer of B’Tselem and continuous to capture footage of life inside the Occupied Territories.

An impact story aims to communicate a deep understanding about what has happened over the course of your Video for Change initiative, and how and why it happened. Drawing on the documentation you’ve collected during each stage of your initiative, the impact story combines your evaluation results to create a compelling narrative about how the initiative has affected the various stakeholders and communities involved.

An impact story can make use of a wide variety of media, including infographics, survey results, photos, writing, audio and, of course, video. Your narrative should highlight moments where real change took place, or where individuals changed their behaviour or mindsets.

An impact story should also take into consideration your original objectives: Did you meet them? How? Where did the initiative fall short of expectations or plans? Were there unexpected positive or negative outcomes?

These are some examples of impact stories. To learn how to create effective impact stories, see Resources.

Surveys

A written or online survey allows you to ask for individual impressions and detailed data through specific questions addressed to community members, audience members, partners and stakeholders. Responses can be confidential or anonymous if appropriate. To create surveys, see Resources.

Focus Group Discussions

A focus group discussion brings together key thinkers and stakeholders on your issues in order to discuss the initiative. They help you come up with ideas, give feedback on tactics and provide original insights grounded in their own knowledge and experience.

This method helps you collect a range of varied perspectives, which can contribute to richer design or evaluation stages, or both. Focus groups are also an effective way to build community and to create buy-in and ownership.

Assessing Participation in the Video-Making Process

This method incorporates the experiences of people who’ve engaged in your project, ensuring their feedback is counted and considered in your evaluation. Worksheet PDF

‘Most Significant Change’

A good impact story incorporates the stories of participants and people who’ve been affected by your video project. ‘Most Significant Change’ is a well-recognised qualitative method that creates and analyses personal stories of change and evaluates which was most significant to a given set of stakeholders and why.

It lets people tell their own story of how your initiative affected them, instead of only measuring impact with a set of limited indicators devised without their participation. Rick Davies created this method, which is also used to understand the impacts of an initiative, including those that may have been unexpected or unplanned.

See the Resources section for the user’s guide he created with Jess Dart, an explanatory video by InsightShare and more tools for using this methods.

Evaluation Indicators