Defining benefits and effectiveness

Introduction

This Information Sheet provides information on Topic 3: Defining benefits and effectiveness.

The following information sheets are part of this series:

  • Introduction to the module
  • Topic 1: Defining adaptation objectives and activities
  • Topic 2: Developing and allocating costs
  • Topic 3: Defining benefits and effectiveness
  • Topic 4: Calculating and comparing costs and benefits
  • Topic 5: Using project implementation tools
  • Topic 6: Using project closure tools

By the end of this topic you will understand the following:

  • How to identify potential benefits associated with an intervention
  • How to categorise benefits for an intervention

Identifying benefits

Every choice, even the choice of inaction, has risks and potential benefits.

One way to assess these risks and benefits is to undertake a PESTLE analysis. This is a way to develop a broad idea of the external factors which exist around the climate change project and which impact upon its functionality, either to enhance and support it, or to hinder and constrain the activities of the project. See below for an explanation of PESTLE, and examples that may help you as you think about your project’s external context.

PESTLE Analysis
Aspect Explanation Example
P Political Ask: What political factors are likely to be impacted on by the climate change project? Elections; degree of government stability; appointment of the Minister; budget decisions; union agreements / action, etc.
E Economic Ask: What economic factors may be impacted on by the climate change project? Joblessness and unemployment; levels of indebtedness; employment opportunities available, etc.
S Social Ask: What social factors may be impacted on by the climate change project? Education level of community; lifestyle; beliefs; crime; HIV/AIDS; village/ community leadership and landholdings, etc.
T Technological Ask: What technological factors may be impacted on on by the project? Infrastructure – access to electricity, water, telephones, sanitation, internet, and reliability thereof.
L Legislative Ask: What legislative factors may be impacted on by the climate change project? The legislative framework on all issues; regulations and policy, which set requirements on any number of issues from foreign donors to compliance regulations.
E Environmental Ask: What environmental factors may be impacted on the climate change project? Catastrophic floods or drought; impact of pollution; waste management; landfills / tips; availability of running water and electricity, etc.

The PESTLE analysis can assist you in analysing the significance of both potential benefits and problems, and is effective in  identifying ‘where’ the benefits and problems occur.

Assessing benefits

To determine the benefits from a particular intervention, it will be necessary to develop a structure assessment plan. The steps below provide some guidelines on what can be included as part of this assessment process.

1. Decide what benefits need to be measured

In this step, you need to determine what must be measured. Do this by:

  • Reviewing the RVCC Integrated Project Monitoring and Evaluation System as this will provide guidelines on the broader programme expectations.
  • Listing what needs to be measured or what evidence will help to demonstrate that you have addressed the identified need/solved the problem or completed the activities at a programme/project level. You can brainstorm with others, including the project team and stakeholders (such as beneficiaries). This helps to establish buy-in into the process and the interventions.

Here are some examples of data that may need to be collected:

  • Degree of soil degradation (dongas, erosion, soil depth, water runoff, etc.);
  • Green cover – quality and quantity;
  • Grazing area – unconfined versus confined, rotational or not;
  • Crop – yields, methods, diversity, household participation in;
  • Surface temperature – quantify, patterns, trends and cycles;
  • Precipitation (rainfall) – volume, patterns, trends and cycles;
  • Employment levels in agriculture, forestry, beekeeping, etc. – quantity and gender distribution, disaggregated by Council;
  • Stated needs of stakeholders in the project area;
  • Poverty indexes for areas / household income;
  • Social capital;
  • Economic activity – type, level of participation etc.;
  • Knowledge and skill – type, level, etc.;
  • Migration; and
  • Equity.
  • Considering the quantitative (numerical) and qualitative (descriptive) measurements that can be used.

You may select both quantitative and qualitative measurements. It is not always possible to measure certain aspects quantitatively. For example, measuring social capital lends itself to more qualitative measurements. Whereas measuring training and the number of men and women trained is a quantitative measurement.

  • Deciding on the key areas/conditions from the list you developed, i.e. prioritise to ensure that you gather evidence that demonstrates achievement of objectives (and impact).
  • Developing indicators for the key areas/activities/conditions.
  • Confirming indicators.

You can make use of SMART, SPICED or MACIB to confirm that the indicators[4] are aligned and meet requirements (see below).

SMART SPICED MACIB
Specific Subjective – use of informants who provide unique insights and offer a good return on the investigators time. Measurable – Can it be measured? Is the indicator sufficiently sensitive to an improvement/deterioration in the conditions?
Measurable Participatory – developed with project beneficiaries, local staff and other stakeholders. Accessible and affordable – How easy is it to collect the information needed? What is the cost of collecting? Is the relevant data already available/collected?
Achievable Interpreted and communicable – locally defined and can be defined (if needs be). Credible and valid – Do the indicators measure something that is important to the community, the implementing organisation and/or funders? Do they measure what they are supposed to measure, or will people argue over their meaning?
Realistic (and Relevant) Cross-checked and compared – to ensure the validity, i.e. measuring what they are supposed to measure (and nothing else). Influential – Will the evidence be useful to communities, implementers and decision-makers?
Time bound Empowering – the process of setting and assessing should empower others. Balanced – Do the indicators provide a comprehensive view of the key issues?
Diverse and aggregated – different indicators should be obtained from a range of groups (especially men and women). This must be recorded so changes can be assessed over time.

This should be done with stakeholders to ensure ongoing engagement and buy-in into the process and investment in the outcomes.

2. Decide on the methodology

There are a number of factors that must be determined in setting up the methodology for your assessment, and you will look at each of these in turn:

  1. Sampling

You must clearly define the target population.

In the RVCC programme this is the Lithipeng, Khoelenya and Thaba Mokehele Community Council. However, you may need to drill down further.

Then decide on the sampling method. There are various methods[5]:

  • Simple random sampling – individuals are chosen by chance, with everyone having an equal probability of being included.
  • Systematic sampling – individual are chosen at selected intervals from a defined population, e.g. every 50th person that enters.
  • Stratified sampling – the population is divided into sub-groups and individuals from each are included.
  • Clustered sampling – the population is divided sub-groups (clusters) and clusters are randomly selected (not individuals within the sub-group).
  • Convenience sampling – participants are selected based on availability and willingness to participate.
  • Quota sampling – a quota is set for each category and this forms the sample population. For example, 20 men, 20 women, 10 boys, 10 girls.
  • Purposive sampling – a more subjective and selective sampling method where the target group is selected based on certain characteristics.
  • Snowball sampling – is based on a referral system, where ‘hard to reach’ participants are referred by someone else within the target population.

The size of the sample also depends on a number of factors:

  • Level of confidence required in the findings. Confidence in data refers to ability to extrapolate from the sample to the population more broadly. This is considered more important when collecting quantitative data. The assumption is that the larger the sample size, the higher the level of confidence that the findings reflect the target population.
  • The instrument/tool used. For example, a survey typically has a much larger sample than a structured interview or a focus group.
  • Time available.
  • Resourcing. This includes both financial resources to conduct the baseline, as well as the availability of skilled data collectors/analysers.

2. Types of data

This should be aligned to what is being measured. Decide on what will provide you with the most useful information.

  • Quantitative – This is numerical data.
  • Qualitative – This is descriptive information.
  • Mixed – includes both numerical data and descriptive information.

3. Sources of data

You will need to decide whether you will need to collect primary data or use secondary data, or a combination of the two.

Primary data Secondary data
What is it This is data that is collected first hand by the researcher. This is data that is collected by someone else and has been reported on.
Types Case studies

Experiments

Interviews

Group discussions

Observation

Rapid assessments

Simulations

Surveys

Archived records

Census conducted by government

Climate change reports

Documentary evidence

Newspapers

Project reports

Reports, e.g. Vulnerability Assessment

Weather reports

Advantages Designed to meet specific requirements so data should be more accurate.

Reliable as have control over the methodology.

Cost-effective.

Can be used to confirm/triangulate primary data.

Quicker to access.

Disadvantages Can be expensive to collect.

Can be time consuming to collect (and may have long lead times).

Requires high levels of effort.

May not meet the specific requirements / purpose.

Data may be unreliable as do not have control over the methodology.

4. Method and instruments

Once you have determined the type of data you will need to collect, you can then finalise the method and the tools/instruments you will use. Below are some of the methods commonly used to gather data:

Method Explanation Example/Tool
Survey Wide scale questioning that typically involves a large percentage of the sample population.

Surveys include both closed-ended and open-ended questions.

Can be conducted face-to-face or electronically.

Census

Online questionnaire completed as part of the Capacity Needs Assessment for this training

Training evaluation

Interviews Face-to-face way of gathering data from target population.

Typically, do not interview all members of the target population but rather a selected sample that is ‘representative’ of the target population. This allows for findings to be extrapolated

Can be structured or semi-structured.

Make use of an interview schedule to guide the information gathering process.

Telephonic interviews

Face-to-face interviews

Group discussion Face to-face way of gathering data from target population.

Typically, do not conduct group discussions with the entire sample population.

Make use of a question schedule to guide the information gathering process.

Focus group conducted as part of the Capacity Needs Assessment for this training
Rapid assessments Short survey that answers a very specific question.

The question is usually closed-ended, although could require some explanation.

Typically includes a larger percentage of the sample population.

Likes on Facebook posts

Twitter polls

Radio call ins

Observation Direct observation.

Typically, do not observe all members of the target population, but rather a selected sample that is ‘representative’ of the target population. This allows for findings to be extrapolated to the broader population.

Observation may be guided by a checklist (structured) or may simply record what is observed (open-ended).

Existing agricultural practices
Case studies An in-depth investigation of a single person, group, event or community.

Data is gathered from a variety of sources using a range of methods (e.g. observation and interviews).

Agricultural practices in the Khoelenya Community Council, 2015-2020
Simulations Mathematical modelling of an actual or probable real life condition / event / situation to project/forecast an outcome or find the underlying cause.

Requires the selection of a specific model/approach.

Vulnerability Assessment

Climate Change scenario modelling

Documentary evidence Review of existing evidence (secondary data).

Can be used to validate data collected in the field.

Weather data – rainfall, surface temperature, etc.

Crop production/yields

Economic indicators

Census (2016)

The data collection method is how the data will be collected. Will it be collected face-to-face? Electronically?

There are a number of factors that influence the method used. These could range from costs of data collection, timeframes, level of confidence needed and ability to extrapolate to the broader population, etc.

3. Prepare for the assessment

In this step it is important that you:

  • Document the methodology: You, the project team, and other stakeholders have worked through everything and it must now be written down/documented. This guiding document should include the purpose, approach, sampling, the data collection tools/instruments, collection method, and how the data will be analysed and reported on.
  • Training: All those involved in data collection and analysis should be trained to ensure that they understand the methodology. This will help to ensure consistency in data collection, recording and reporting.

4. Conduct the assessment

  • Gather data in line with your plan.
  • Record data using the tools/instruments provided. Part of this process involves capturing the data and ‘cleaning’ the data.
  • Record issues/problems in the collection process.
  • Analyse the data. This involves evaluating the data using specific analytical tools.

5. Report on findings

Finally, you will report on the findings. Data could be presented in a report that should cover both methodology and findings/results.

The findings can be presented using tables, graphs and charts as well as narrative/descriptive paragraphs.

6. Storage of assessment

Since baseline data and findings will be used not only as input in the CBA/CEA process, but also in project monitoring and evaluation, it is important that this is securely stored and can be accessed by required parties through the project.

Quantifying effectiveness

You may not always be able to measure benefits in financial terms, you still need to be able to measure the anticipated gains of project implementation or how effective it was. Indicators are quantitative or qualitative factors that provide a measurement of achievement or reflect performance or changes from the baseline (current state). As such they are good measures of effectiveness.

A range of indicators must be set in order assess effectiveness. Good indicators are valid, reliable and simple.

The following steps will assist you in developing effectiveness indicators in your project:

  1. Review the RVCC Integrated Project Monitoring and Evaluation System Final Report. This will give insight into the Programme output indicators and should inform the effectiveness indicators you develop.
  2. Develop a list of what could be measured in projects (that demonstrate effectiveness). This could include measurements around social capital, equity, etc.
  3. Categorise the listed measurements (from 2) under one of the following – inputs, outputs, outcomes.
  4. Prioritise – decide on key areas/conditions that will be used to measure effectiveness.
  5. Develop indicators that are SMART/SPICED/MACIB.
  6. Confirm that indicators are SMART/SPICED/MACIB.

Below are examples of indicators taken from Reducing vulnerability from climate change in Foothills, low Lowlands and Senqu River Basin in Mohale’s Hoek District in Lesotho, 2015-2020. They show what indicators look like at different levels in the RVCC M&E system:

Indicator
Project Objective: To mainstream climate change risk considerations in the Land Rehabilitation Programme of Lesotho for improved ecosystem resilience and reduced vulnerability of livelihoods to climate shock. % of community members who say climate-driven vulnerabilities information is used in planning and implementation of Land Rehabilitation Programmes.
Outcome: Increased technical capacity of the Ministry of Forestry, Range and Soil Conservation (MFRSC) and relevant departments to apply up-to-date climate science for the management of evolving risk and uncertainty linked to climate change. % of MFRSC and relevant departments’ technical staff competent in skills for management of evolving risks and uncertainty linked to climate change (i.e. meet set standard when applying skill learned in training [data disaggregated by gender]).
Output Training of technical staff of engineering, planning and monitoring sections of the Ministry of Forestry, Range and Soil Conservation on climate science conducted. Number of staff trained in climate science from engineering. planning and monitoring sections (data disaggregated by gender & unit).

 


References

Burke, R. 2017. Fundamentals of Project Management: Tools and Techniques. Everbest, HK/China: Burke Publishing.

Burke, R. (2010) Project management techniques (college edition).Everbest, HK/China: Burke Publishing

Developing your project baseline. 2017. [Online] Available at: https://www.jisc.ac.uk/guides/transforming-assessment-and-feedback/project-baseline.

Orotin, P. (2017, December). Reducing vulnerability from climate change in Foothills, lower Lowlands and Senqu River Basin in Mohale’s Hoek District in Lesotho, 2015-2020.

Salkind, N. J. 2010. Encyclopedia of research design [Online] Thousand Oaks, CA: SAGE Publications, Inc. Available at: http://methods.sagepub.com/reference/encyc-of-research-design/n333.xml

Shantikumar, S. 2018.  Methods of sampling from a population. [Online] Available at: https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1a-epidemiology/methods-of-sampling-population

Sindhu, A. 2011. Sales promotion strategy of selected companies of FMCG Sector in Gujarat region. Thesis (PhD). Available at: http://shodhganga.inflibnet.ac.in:8080/jspui/bitstream/10603/3704/1/01_title.pdf

UNAIDS, n.d. Glossary: Monitoring and Evaluation Terms [Online] Available at: http://www.unaids.org/sites/default/files/sub_landing/files/11_ME_Glossary_FinalWorkingDraft.pdf

United Nations Framework Convention on Climate Change, 2011. Assessing the costs and benefits of adaptation options: An overview of approaches [Online] At: https://unfccc.int/resource/docs/publications/pub_nwp_costs_benefits_adaptation.pdf

https://evaluateblog.wordpress.com/2013/05/28/baseline-studies-surveys/

http://www.endvawnow.org/en/articles/959-baseline-studies.html

https://sswm.info/planning-and-programming/exploring-tools/preliminary-assessment-current-status/baseline-data-collection


Key terms  

 

Adaptation benefits This refers to the costs that are avoided or the benefits obtained as a result of the implementation of a specific climate change adaptation intervention/option.
Assumptions A factor in the planning process that is considered true/real/certain, without proof or evidence necessarily being provided.
Costs The costs of planning, preparing for, facilitating, and implementing adaptation interventions / projects / measures, including transition costs.
Effectiveness This is the extent to which the intervention (at a programme or project level) has achieved the objectives in real-life.
Equity In the M & E framework, this refers to gender equity or the availability of opportunities.
Indicator A quantitative or qualitative variable (factor) that provides a reliable and valid measurement of achievement or reflects performance or changes from the current state.
Risk An uncertain event or condition in a project, that if it comes about, will affect one or more project objective.
Social capital This refers to the ‘the links, shared values and understandings in society that enable individuals and groups to trust each other and so work together.’