Capturing Experience
How Do You Set Up a System
for Monitoring and Evaluation?

Monitoring and Evaluating Program Implementation

Excerpt from:
Ivo Imparato, Diagonal Urbana, and Jeff Ruster, World Bank. Participation in Upgrading and Services for the Urban Poor: Lessons From Latin America. The World Bank, 1999.
Contact: diagonal@elogica.com.br

Objectives

• To monitor the development of the program as a whole, and of its component projects, in relation to changes in the context and circumstances of their implementation;
• To monitor the development of the program as a whole, and of its component projects, in regard to goals, timelines, and any unforeseen circumstances that may occur;
• To implement a rapid problem identification system, as well as a system for internal communications to the various stakeholders;
• To facilitate evaluation procedures during and after activities, through the definition of specific indicators.

Key Process Issues

In addition to daily control over operations, deadlines, and any problems that may arise, the institution promoting a development program, or an administration responsible for policy implementation, should also perform periodic evaluations of the program or policy as a whole. This is needed to gauge the results obtained by each stage of implementation of the program or policy, within its economic, social, cultural, institutional, and environmental context.

A system of appropriate indicators must be developed for the purposes of such evaluations - which relates directly to project monitoring. The system of indicators should be divided into two sub-systems. The first of the two is composed of those indicators that allow a sort of ‘reality monitoring’.

‘Reality Monitoring’

All development programs or policies (and particularly those that aim to improve informal settlements) can be seen as initiatives for fighting poverty and social exclusion, as well as for strengthening social capital1 . Indicators relative to ‘reality monitoring’ should therefore be focused on these three orders of macro-phenomena: poverty, social exclusion and social capital.

Poverty Indicators

If we define poverty as a continued deprivation of the means of well-being over time, we should consider the following six major points:

• percentage of homes built with non-permanent materials;
• percentage of households which do not have access to safe water supplies;
• percentage of households which do not have access to sanitation facilities;
• percentage of economically active individuals with annual incomes lower than a certain sum (based on the poverty line of the country in question);
• percentage of children between the ages of 5 and 14 who do not attend school;
• percentage of children who are malnourished.

Obviously, not all changes recorded over time with regard to the conditions of poverty, social exclusion and social capital can be considered solely as effects of the project being monitored. But if a project implements environmental sanitation activities, and a clear improvement of health indicators is recorded, there is probably a connection between the two. In this case, the variation of health indicators would measure, even if indirectly, the effects of the project. This first sub-system of indicators should permit the monitoring of the evolution of the economic, social, cultural, institutional, and environmental contexts, and therefore the adoption, in a timely manner, of corrective measures in order to maintain the project’s pertinence and relevance.

Effectiveness, Efficiency, Impact and Pertinence Indicators

The second sub-system should be formed by specific indicators pertaining to the activities undertaken and their expected results. Obviously, these indicators can be conceived only in conjunction with the formulation of their corresponding activities. They should include:

effectiveness indicators, which:
- measure the level of execution of the various activities within a program (for example, in the case of constructing a sewer system, one can measure the length of pipes installed in relation to the total to be installed; in a training activity, one can analyze the number of people who participated in specific activities in relation to the anticipated beneficiaries; etc.).
- measure the satisfaction of participants (or beneficiaries) regarding specific activities.

efficiency indicators, which can be determined in the following ways:
- comparing actual costs with those budgeted for specific activities;
- comparing actual costs with those incurred by analogous projects.

impact indicators, which can be determined by linking the specific activities of the project to the variations in the related indicators in the first sub-system (for example, changes in the “youth unemployment rate” would be related to the number of stable jobs created for youths by specific project activities).

pertinence indicators, which, similarly to impact indicators, would link the various activities undertaken by the program with social exclusion factors on the one hand and social capital factors on the other.

Sequence of Activities

The development of the appropriate indicators needs to be an integral part of the program formulation exercise. The establishment of procedures for the regular collection of the information that will be needed to measure the pre-defined indicators needs to be envisaged and spelt out during the preparation of the Operating Manual, and taken into account by the program’s management system.

During the implementation period of the project, the system of indicators described above could be utilized in the following manner:

the first sub-system of indicators would be measured once a year. Secondary information sources would be tapped most of the time, although rapid on-site research would be conducted on an ad-hoc basis. As mentioned above, the variations would be evaluated. In this manner, indirect information on the development of the project can be obtained;

the second set of indicators would be applied on a more frequent basis. These indicators would be measured on the basis of the internal documentation of the project and through rapid on-site appraisal techniques.

Special attention should be paid to the timely execution of monitoring activities. Delays can lead to a loss of control of the implementation of the program, with negative economic consequences and loss of impact.

If indicators are well designed and consistently monitored, stakeholders can be kept abreast of the development of the program, and participatory evaluation techniques can be utilized to gauge the results obtained during each stage of implementation, to correct shortcomings in its operating strategy and to adapt the program to changes in the circumstances in which it operates.

Considerations Before Moving On

Before moving on to the project implementation cycle, it is essential to have a reliable monitoring and evaluation setup firmly in place. The definition of responsibilities for data collection, processing and storage is as important as the definition of good indicators; and care should be taken not to waste resources by generating an exaggerated amount of data that will not be used.

1 CERFE, Document prepared for a seminar held in conjunction with the World Bank on poverty, social exclusion, and social initiatives. Washington, November 16th, 1998.