Capturing Experience: Evaluation
Evaluation and Impact Assessment Methods

Table of Contents for:

Monitoring and Evaluating Urban Development Programs: A Handbook for Program Managers and Researchers
Bamberger, Michael, and Hewitt, Eleanor. 1986. Washington DC: The World Bank.
Introduction: A User’s Guide to the Handbook

Chapter 1: The Monitoring and Evaluation Framework

    A. Monitoring and Evaluation: Essential Management Tools
    B. Framework for Monitoring and Evaluation of Urban Development Projects
    1. A model of the project implementation process
    2. The functions of monitoring and evaluation
    C. Planning the Monitoring and Evaluation System
    1. Defining the issues to be studied
    2. Organizing the monitoring and evaluation
    3. The scope and intensity of the studies
    4. Ensuring that the evaluation is “user oriented” and not “technique oriented”
    5. Defining resource requirements
    D. Managing the Monitoring and Evaluation
    1. The role of the project manager in monitoring and evaluation
    2. Defining the main users of monitoring and evaluation
    3. The outputs of monitoring and evaluation and their practical utility
    4. The importance of regular reviews of monitoring and
    evaluation of outputs
    5. Potential problems in the design and implementation of a monitoring and evaluation system and some possible solutions

Chapter 2: Performance Monitoring

    A. Performance Monitoring
    B. Designing the Performance Monitoring System
    C. Monitoring Software Components
    D. Methods for Collecting and Analyzing Data for Performance
    E. Reports
    F. Potential Problems and Possible Solutions in the Design and
    Implementation of a Monitoring System

Chapter 3: Process Monitoring – Monitoring the Project Delivery System

    A. Key Issues
    B. When and Why to Monitor the Implementation Process
    C. Data Collection Methods
    1. The importance of a multi-method approach
    2. Modelling the process of project implementation
    3. Quantitative surveys
    4. Direct observation
    5. Secondary data
    6. Ethnographic methods
    D. Methods for Monitoring the Efficiency of the Implementation Process
    1. Monitoring the overall efficiency of project implementation
    2. Developing summary indicators
    3. Indicators of the efficiency of individual project components
    4. Studying community level organizations
    E. The Design of Continuous, Periodic and One-Time Process
    1. Continuous panel studies
    2. Periodic studies
    3. One-time studies
    F. Defining the Strategy for Process Monitoring

Chapter 4: Impact Evaluation

    A. Approaches to Impact Evaluation
    B. Some Key Issues in the Design of Urban Impact Evaluations
    1. Is it necessary to evaluate project impacts?
    2. Is it possible to measure impacts and assess causation?
    3. Do cost-effective methods exist for impact evaluation?
    4. Quantitative versus qualitative methods
    C. Examples of Impact Evaluations
    1. A large-scale randomized experimental design: the experimental housing allowance program
    2. A quasi-experimental design: evaluating the impact of a sites and services housing project on employment and income in El Salvador
    3. Rapid impact evaluation employing a multi-method approach: evaluating the social and economic impacts of a cooperative program in El Salvador
    D. Alternatives to Large Scale Quantitative Evaluation Designs
    1. The selection of the most appropriate research techniques and the use of triangulation
    2. Rapid impact studies
    4. Simple quantitative methods
    5. Using secondary data
    E. Quantitative Estimates of Net Project Impacts
    1. Modelling the implementation process
    2. Operational definitions of expected impacts
    3. Experimental and quasi-experimental designs: the adaptation of textbook evaluation designs to the realities of the urban context
    4. The use of panel and independent samples
    5. Sample design
    6. Analysis and interpretation of the survey data
    F. Comparing the Effectiveness of Different Projects
    1. Issues in the definition and measurement of costs
    2. Issues in the definition and measurement of benefits and effectiveness
    3. Cost benefit analysis
    4. Cost-effective analysis
    5. Cost-utility analysis
    G. Choosing the Appropriate Strategy for Impact Evaluation
    1. What are the future options currently being considered and what are the key factors which will affect the decisions
    2. Time scale
    3. Available resources
    4. Scale and complexity of the project
    5. The nature of the project and the complexity of its objectives
    6. The need for a multi-method approach

Chapter 5: Managing the Evaluation

    A. The Monitoring and Evaluation Functions and Needs of Different Organizations
    B. Organizing Monitoring and Evaluation at the Project Implementation Level
    1. The use of consultants
    2. Locating the evaluation unit in the organization
    C. Organization of Monitoring and Evaluation for the Project Coordinating Agency (PCA)
    D. Monitoring and Evaluation by Sectoral Agencies
    E. Organization of Monitoring and Evaluation at the National Level
    1. National urban development agencies
    2. Finance ministries and legislative watchdogs
    F. Resource Requirements for Monitoring and Evaluation
    1. Resource requirements for local implementing agencies
    2. Resource requirements for the local project coordinating agency (PCA)
    3. Evaluation resources for the national development agencies
    4. Fitting staff into civil service categories
    5. Staff training
    6. Temporary assignment of other project staff to the evaluation
    7. Using consultants to overcome salary constraints
    G. The Role of Donor Agencies in the Evaluation
    H. Potential Problems and Possible Solutions

Chapter 6: Issues and Approaches in Evaluating Non-Shelter Related Urban Projects

    A. New Directions in Urban Development Projects
    B. Income and Employment Generation Project Components
    1. Key policy issues
    2. Evaluation issues
    3. Applicability of the monitoring and evaluation framework discussed in previous chapters
    C. Family and Public Health Programs
    1. Key policy issues
    2. Evaluation issues
    3. Applicability of the monitoring and evaluation framework
    D. Urban Transport
    1. Key policy issues
    2. Evaluation issues
    3. Applicability of the monitoring and evaluation framework
    E. Municipal and institutional development
    1. Key policy issues
    2. Evaluation issues
    3. Applicability of the monitoring and evaluation framework
    F. Summary and Conclusions

Annex A: Defining and Using a Model of the Project Implementation Process

    A. Defining the Project Model and its Assumptions
    B. Defining measurable objectives

Annex B: Methods of Data Collection for Monitoring and Evaluation

    A. The Importance of the Choice of Research Methods
    B. The Use of Triangulation to Build in Consistency Checks
    C. Participant Observation and Related Ethnographic Techniques
    D. Direct Observation
    E. Informal Group Discussions
    F. Unstructured (“Open”) Interviews
    G. Structured Questionnaires
    H. Secondary Sources
    I. Further Reading

Annex C: Managers Guide to the Design and Implementation of a Monitoring and Evaluation System

    A. Initial Decisions on the Scope, Organization and Objectives of the Evaluation
    B. Definition of the Organizational Structure
    C. Defining and Mobilizing Financial and Human Resources
    D. Research Design
    E. Integrating the Evaluation into the Project Development Cycle
    F. Defining the Main Users of the Studies and their Information Requirements
    G. Defining the Planning and Review Cycles

Annex D: The Basic Concepts of Sample Design

    A. Why are Samples Used?
    B. Sample Precision and Confidence Intervals
    C. Methods of Sample Selection
    D. Related (Panel) and Independent Samples
    E. Sample Designs for the Evaluation of Sites and Services Projects
    F. Sample Design for the Evaluation of Upgrading Projects
    G. Further Reading

Annex E: Experimental and Quasi-Experimental Designs in Urban Impact Evaluation

    A. The Logic of the True Experimental Design
    B. Threats to Validity in the Interpretation of the Analysis
    C. Problems in Applying the Experimental Design to the Evaluation of Urban Projects
    D. Alternative Quasi-Experimental Designs
    E. Application of Quasi-Experimental Designs in Upgrading and Sites and Services Projects
    F. Further Reading

Annex F: Methods of Data Analysis

    A. Statistical Analysis of the True Experimental Design
    B. Tests of Association and Differences for Use With Some of the Simpler Evaluation Designs
    C. The Use of Multiple Regression in the Analysis of Quasi-Experimental Designs
    D. The Use of Multiple Regression When not all Participants Receive the Same Package of Services or When There is No Control Group
    E. Path Analysis
    F. Cost Benefit Analysis
    G. Hedonic Price Analysis
    H. Further Reading

Annex G: Sample Outline of a Quarterly Progress Report

    A. Introduction
    B. Contents of the Report
    C. Financial Performance
    D. Services and Programs
    E. Attitudes of Project Beneficiaries

Annex H: Designing a Network Based System to Monitor Project Implementation

    A. Introduction and Objectives
    B. Designing the System
    C. Designing the Monitoring System

Annex I: Designing a System for Network-Based Financial Monitoring

    A. The Elements of Network-Based Budgeting
    B. Definition of Cost-Bearing Activities
    C. Estimation of Expenditure Schedules for Each Component
    D. Monitoring Cost and Expenditure Schedules
    E. Cost Variation Analysis
    F. Revision of Cost and Implementation Schedules

Annex J: A Basic Evaluation Library

    A. General Textbooks
    B. More Specialized Reference Books
    C. Basic Statistical Textbooks

Annex K: Glossary of Terms Used in the Handbook

    References Cited in the Text

Bibliography Links

| Bibliography Index | Upgrading Strategies | Social Issues | Physical Issues |
| Financial Issues | Institutional Issues | Implementation Issues |
| Evaluation and Impact Assessment Methods | Overview | Handbooks, Manuals |

| What is Urban Upgrading? | Doing Urban Upgrading | Case Examples |
Issues and Tools | Resources | About This Site |
Search Web Site | Site Map | Home | Ask Grady | Feedback |

Copyright © 1999-2001, The World Bank Group. All Rights Reserved.