Module 3: Monitoring and evaluation of climate change interventions

Introduction

This Information Sheet provides an introduction to monitoring and evaluation for Module 3: Monitoring and evaluation of climate change interventions.

This information provides an overview of:

  • Understanding the difference between monitoring and evaluation
  • Understanding the Theory of Change (ToC) and the logframe
  • Understanding key concepts in M&E

Key themes

Understanding monitoring and evaluation

You will have come across both these terms – monitoring and evaluation – in Module 2 when looking at the project life cycle – preparation, implementation and completion. These two terms are often used simultaneously, but refer to different processes and occur at different times in the lifespan of a project/programme.

Understanding monitoring

What is monitoring?

This is a systematic and ongoing process of collecting, analysing and reporting on achievements in a specific project. Typically monitoring focuses on performance and how this, over the long-term, contributes to achieving the project deliverables.

Monitoring looks at:

  • What activities have been performed?
  • What progress has been made in terms of time, cost and quality?
  • Whether indicators have been achieved?

Why is monitoring important?

Monitoring is an important control tool:

  • it provides feedback on project progress;
  • provides the organisation with the opportunity to improve design and implementation (if project progress is not going according to plan);
  • and is the basis for evaluation as data is collected at intervals and can be rolled-up; and
  • provides opportunities for learning. This can be used to establish best practice, inform future planning and allocation of resources, and build knowledge that facilitates sound decision making etc.

When do we monitor?

In the planning phase, the project manager and their team, will decide on the frequency (and format) of monitoring. Typically this means that monitoring happens at specified intervals. This could be on a monthly basis or a quarterly basis. There are several factors that could inform the frequency of monitoring:

  • size of the project – in terms of budget and complexity;
  • geographical location/accessibility of the project;
  • donor/funder requirements
  • level of risk – with high risk projects requiring more frequent monitoring

Understanding evaluation

What is evaluation?

Evaluation is a summative process that compares whether the planned outcomes, outputs, indicators and targets have been met through the implementation of the project. Evaluation looks at the following:

  • What objectives, outcomes, indicators and targets have been met?
  • What has been achieved in relation to these?
  • How were these achieved?
  • What is the impact?

Why is evaluation important?

Like monitoring, evaluation:

  • demonstrates accountability to funders/donors and other key stakeholders;
  • measures impact;
  • identifies successes and challenges

When do we evaluate?

Evaluation can take place at different points in the project – this could be conducted at the beginning the project (i.e. baseline), then mid-way through, and finally upon completion.

Understanding frameworks

What is a framework?

A framework is ‘a system of rules, ideas or beliefs that is used to plan or decide something’.

Within the Monitoring and Evaluation context, a framework is therefore a system used to guide the monitoring and evaluation process. It is based on what constitutes monitoring and evaluation, what is important in this process and key constructs/concepts around monitoring and evaluation (ideas and beliefs) and how (rules) it should be conducted.

The M&E framework adopted by an organisation, thus guides practice.

An M&E framework and a Theory of Change (ToC) or Logframe, as described below, are two ends of the same process. Whereas the ToC or Logframe is the result of a consultative and planning process with stakeholders, and is prepared at the beginning of the project, its elements should be used to monitor progress and evaluate effectiveness/ impact. So the goals, indicators, assumptions and so on, will be examined and the evidence of achievement scrutinised to determine correlation.

Understanding Theory of Change and logframe

Theory of Change

The Theory of Change (ToC) is a tool used to describe how and why a desired change is expected by ‘mapping out’ the activities/interventions and how these will lead to the achievement of the final intended impacts for the programme. A ToC is always linked to a specific context.

Having a ToC provides a strong rationale for why we do what we do. It is often described as “filling in the ‘missing middle’ between  the activities or tasks in a change project and how these activities lead to the achievement of goals.”

This is used when evaluating impact and allows evaluators to extrapolate learnings from one site/intervention to another site/intervention.

The components of a Theory of Change

The following are key components of the Theory of Change:

pasted image 0 5

Preparing a Theory of Change

The components of a ToC are prepared in a particular order. See below for an explanation:

Identifying long-term goals and outcomes

These are the high-level statements that will need to be met or fulfilled in order for the project to be deemed to be successful. They should be clearly and succinctly crafted.

Backmapping and connecting outcomes

Backmapping means moving downwards towards actual actions. It is about becoming clearer about what is required and also making connections between different ideas.

Identifying pre-conditions

Pre-conditions need to be met before the next step can be taken.  For example, training may be required before certain equipment should be handled.  Or stakeholders may need to have acquired a certain level of literacy to be able to read gauges, for example.

Identifying assumptions

An assumption is often challenging to articulate as it may be ‘assumed’ to be natural, or given, or fixed. But surfacing these ideas enables then to be interrogated – especially as different stakeholders may have different, unspoken assumptions. The idea is to reach agreement on assumptions operating in the project..

Developing indicators

This is the item being measured. It is usually necessary to describe the population being measured – such as, in the logframe below, the women in a village who are involved in agriculture.

You may be asked to state the minimum, or threshold, for success – the lowest number or percentage required to achieve success.

Identifying interventions

These are the various activities/ tasks that need to be completed in order for the outcomes of the project to be met. This is where the change process really happens.

What is the relationship between a Theory of Change and a Logframe?

A ToC and a Logframe both attempt to describe a change project. But they do so in very different ways. One COULD say that a ToC is more complex and ‘messy’, describing the preconditions that need to be in place before a change initiative can be implemented properly, taking care to think about assumptions and clarify these, and so on. A ToC can also indicate relationships and interrelationships between concepts.

A Logframe is simpler, clearer, but also more rigid. It presents a summary of the proposed project in 16 ‘boxes’. The interrelationships depicted in a ToC are not evident, except for the hierarchy of the structure from Goals to Activities from top to bottom of the table on the left hand side, as opposed to a Project Summary on the left hand top row to Risks/assumptions on the right hand top row. It is certainly simpler and easier to see at a glance what a Logframe presents.

The structure of a logframe

A standard logframe is divided into four rows, which are the long- to short-term objectives ranging from top to bottom:

  • Goal (overall aim).
  • Outcome/Purpose (what will be achieved, who will benefit, and by when).
  • Outputs (specific results the project will generate).
  • Activities (what tasks need to be done in order for the output to be achieved).

These are achieved and measured by the headings from left to right:

  • Project summary (explaining the objectives).
  • Objectively verifiable indicators (how you will measure the achievements).
  • Means of verification (how you will collect the information for the indicators).
  • Risks and assumptions (external conditions needed to get results).

See below for a simple example of a logframe:

Example of a logframe

  Project summary Indicators Means of verification Risks/ assumptions
Goal 10% increase in women employed in agriculture % of women employed in agriculture Employment records N/a
Outcome Skills in beekeeping methods Employment Demonstrated proficiency in beekeeping Lack of uptake in employment after training
Output Number of women trained in beekeeping Number of women employed in agriculture Attendance register No equipment available to allow uptake
Activity Training in beekeeping Number of training events Records of the training Diminished attendance due to other responsibilities

Key points to remember about a logframe:

  • Keep the statements positive
  • Involve project stakeholders in its preparation
  • Be flexible and adjust if necessary as the project proceeds

Understanding key M&E concepts

As indicated previously, a framework is a system, that has key ideas/concepts associated with the system. In order to feel comfortable with any system, it is important that you understand the terms (jargon) used by the system. The following are common terms found within a M&E system:

pasted image 0 4

 

Additional support material

RVCC Monitoring and Evaluation Framework – https://drive.google.com/file/d/1u-zFMcovJqG1sxftlXInfSp5b06-43hi/view?usp=sharing

Odiri, P. 2017. Reducing vulnerability from climate change in Foothills, lower Lowlands and Senqu River Basin in Mohale’s Hoek District in Lesotho, 2015-2020. Integrated Project Monitoring and Evaluation System. Final Report.

References

Cambridge Dictionary [Online] At:  https://dictionary.cambridge.org/dictionary/english/framework

Center for Theory of Change. n.d. What is a Theory of Change [Online] At: https://www.theoryofchange.org/what-is-theory-of-change/

Learning for Action. n.d. What is a theory of change. [Online] At: http://learningforaction.com/what-is-a-theory-of-change/

Logframe. n.d. https://www.theguardian.com/global-development-professionals-network/2015/aug/17/how-to-write-a-logframe-a-beginners-guide

NHS. n.d. What is an outcomes framework? [Online] At: http://www.healthscotland.com/ofhi/what_is_it.html

Odiri, P. 2017. Reducing vulnerability from climate change in Foothills, lower Lowlands and Senqu River Basin in Mohale’s Hoek District in Lesotho, 2015-2020. Integrated Project Monitoring and Evaluation System. Final Report.

Rogers, P. 2014. Theory of Change [Online] At: http://devinfolive.info/impact_evaluation/img/downloads/Theory_of_Change_ENG.pdf

Theory of Change vs Logical Framework n.d. At: http://www.tools4dev.org/resources/theory-of-change-vs-logical-framework-whats-the-difference-in-practice/

UNAIDS. Glossary – Monitoring and Evaluation Terms. [Online] At: http://www.unaids.org/sites/default/files/sub_landing/files/11_ME_Glossary_FinalWorkingDraft.pdf

 

Key terms

Assumptions Hypotheses about factors or risks that could influence the progress or success of a programme/intervention
Baseline The status (in numerical terms) against which progress can be assessed or comparison made
Data Distinct pieces of collected information and expressed in a specific manner. Quantitative data are pieces of information expressed as numbers, whereas qualitative data are pieces of information expressed in a descriptive and narrative format
Evaluation The collection of information about programme/intervention activities, characteristics and outcomes that determine the worth of the programme/intervention that informs decisions about future resource allocation
Formative evaluation Takes place during the implementation phase and helps guide the overall process of improvement for the programme.
Impact This is the high level and long term cumulative effect of the programme/intervention
Indicator A qualitative or quantitative variable that measures the level of performance for a project and/or activity
Logical framework A management tool used to improve the design of interventions
Monitoring Routine tracking and reporting about programme/project inputs, outputs, outcomes and impact
Objective A statement of the programme/project results
Outcome The effects (short and medium term) of the project outputs, e.g. change in knowledge, skill, attitudes, behaviours and practices
Outcomes framework A tool that assists intervention planners in linking the activities with what they want to achieve (outcomes)
Output These are the project specific deliverables that provide the necessary conditions to achieve the outcomes, e.g. number of people trained, hectares of land protected, better managed and rehabilitated
Programme The overarching response to an identified need, problem or issue. A programme usually includes a number of inventions
Project An intervention that is designed to meet specific objectives, using specified resources, within a specified timeframe. Usually a number of projects contribute to the achievement of programme goals and objectives
Qualitative data This is descriptive data that is collected through interviews, focus groups, and observation. It is usually expressed in narrative form (not numerical)
Quantitative data This is numerical data that is analysed using statistical methods. It is usually expressed using tables and graphs/charts
Result The intended or measurable change in condition that directly or indirectly affects the established target
Risk Uncertain events or conditions, that if they happen, will influence the project objective
Summative evaluation Takes place at the end of the programme/intervention and is used to assess whether the outcomes and impact of the programme/intervention have been achieved
Target Is a measurable value/desired value for an indicator that the programme/intervention is working towards, e.g. 7000 households adopting climate-smart livelihood strategies by 2020
Theory of change Provides a detailed description of how and why a desired change is expected in a specified context.
Verification The evidence collected and used to confirm that outcomes, indicators, and targets have been met for a programme/intervention