Capturing Experience
Indicators to See If You Got It Right

 
Typical indicators and alternatives in measuring impact are highlighted in Measuring Upgrading Impact. A structure for establishing indicators oriented around poverty reduction is provided in Determining Indicators.

Shlomo Angel. UPGRADING INDICATORS: MEASURING THE PERFORMANCE OF NATIONAL URBAN UPGRADING PROGRAMS. Presentation prepared for the Scaling-Up Urban Upgrading in Latin American Workshop, World Bank, Washington, D.C. September 13-17, 1999.

There are four objectives of measurement of indicators:

• Monitoring of the progress
• Establishing global and regional norms
• Measuring impact
• Correcting the course of upgrading programs by measuring and comparing the impacts of alternative strategies.

There are ten typical indicators for monitoring the progress of upgrading programs:

1. Program coverage structure: the proportion of households living in unauthorized settlements where the upgrading program has been fully implemented, partially implemented, planned, surveyed, and untouched.

2. Program coverage potential: the proportion of housing units in unauthorized settlements where the program can be legally implemented.

3. Tenure registration structure: the proportion of housing units in unauthorized settlements with fully registered ownership documents, partial registration in some approved form, and no registration.

4. Program cost-to-income ratio: the ratio of the median cost per household of implementing the upgrading program and the median household income in the city during the past year.

5. Program cost structure: the proportion of total program cost-to-date spent on overhead, surveying and planning, infrastructure, titling, social services, compensation to landowners and resettlement.

6. Program time horizon: the time required to fully implement the program to serve all the population living in existing and anticipated unauthorized housing, given the program implementation rate during the latest year.

7. Housing subsidy structure: the proportion of the total government subsidies to the housing sector (both on-the-books and off-the-books) allocated to urban upgrading, housing allowances, public and public-assisted housing, interest-rate subsidies, and tax credits.

8. Cost recovery rate: the percentage of the total cost of the program to-date which has been recovered during the latest year.

9. Program decentralization structure: the proportion of total program cost to-date spent by central government, transferred and spent by local government, transferred and spent by voluntary organizations, and transferred and spent by communities.

10. Program privatization structure: the proportion of total program cost to-date spent by government agencies, non-government agencies, the private sector, and self-help work by members of the community.

Measuring the Impact of Upgrading Programs

Impact must be measured by linking policies and programs to housing conditions. One form of linkage is a before-and-after comparison of housing conditions in a group of communities which underwent an upgrading process. A second form is a control-group comparison of conditions between communities that underwent an upgrading program and a control group of communities in similar circumstances which did not undergo such a program. A third, more analytical form of linkage is econometric modeling where variations in housing conditions are explained statistically through variations in a number of other variables. Measurement is only possible if indicators used to measure housing conditions and upgrading programs are comparable and consistent.

A measurement system has to fulfill the following five requirements:

1 - It should be comprehensive and address main concerns of the key actors.

2 - It should contain both cause and effect, indicators and composite indices that characterize the economic, social and political context, the housing policy environment and the upgrading program, housing market conditions; and housing market outcomes.

3 - It should be both temporal and cross-sectional, monitoring communities over time and comparing communities, cities and countries.

4 - It should be balanced between efficiency and distributive measures, so that judgments can be made.

5 - The set of chosen indicators and composite indices should be as small as possible and with little overlap.

Each individual indicator or composite index should fulfill the following four criteria:

1 - It should be unambiguous, clearly indicating 'which way is up?'.

2 - It should be sensitive to change, simple and transparent, consistent, precise and cost-effective.

3 - They should be insightful inventions.

4 - Where a simple ratio is insufficient, composite indices should be constructed in a consistent and transparent manner, assigning self-evident weights to each element of the index.

Ten typical indicators that measure housing conditions:

1 - Land price-to-income ratio: ratio of median price per sq.meter of built-up land and the median annual household income in past year.

2 - House value-to-income ratio: ratio of value of median-priced house and median annual household income in the past year.

3 - Rent-to-income ratio: ratio of annual median rent for a dwelling unit and median annual rental-household income in the past year.

4 - Owner occupancy: percentage of households which own the dwelling units which they occupy.

5 - Credit-to-value ratio: ratio of total amount of housing credit and total value of housing stock in past year.

6 - Residential economic activity: percentage of household income which is earned in economic activities in or in front of houses.

7 - Floor area per person: median usable floor area per person in past year.

8 - Construction quality index: composite measure of durability and fire-resistance, and presence of basic amenities in a median-priced house (piped water, electricity, toilet, heating and cooling equipment).

9 - Infrastructure quality index: composite measure of water supply, legality, reliability and rates: electricity supply, legality, reliability and rates; garbage collection, drainage and sewerage; all weather road and road repair; open space; fire damage and truck access in the community.

10 - Residential mobility: percentage of households that moved into their current unit in the past year.

Six sources of indicator data:

1 - Air photographs and ground truth metropolitan survey

2 - Sample household survey

3 - Real estate appraisal

4 - Program accounts auditing

5 - Secondary data sources: census data, national accounts, etc.

6 - Interviews with knowledgeable informants

Before and after comparisons in single communities and control-group comparisons between communities can measure the impacts of the upgrading program in a single city. They cannot be used to effectively compare program impacts among different cities and countries with different economic, social and political contexts and different housing policy environments. To compare different cities, we can use indicators for econometric modeling, where variations in housing conditions are explained statistically through variations in a number of other variables.

Excerpt from:
Indicators for Monitoring Poverty Reduction. Carvalho, Soniya, and White, Howard. 1994. World Bank discussion Paper No. 254.
 

Performance indicators can facilitate corrections in project design and implementation. They are not a substitute for more in-depth evaluative work - they are indicative only and constitute only one part of the project management and monitoring process. Keep in mind that data should not be collected for their own seek, but clearly must be related to need.

Three classes of indicators

Input: measurement of the ‘means’ by which projects are implemented.

Process: measurement of the extend to which the project is delivering what it is intended to deliver.

Impact: measurement of the project’s impact upon the living standards of the poor. This is a broad indicator and not just for income. It is the most difficult to collect partly because of the time lag between implementation and impact.

Good monitoring requires a balanced use of all three. For each, one should consider the different levels of objectives, for example, immediate, intermediate, and overall.

If indicators are designed at the same time as the project is designed, it is more likely that beneficiaries, borrowers, project managers and funders will feel ownership of the project’s outcome.

Involvement of Beneficiaries

• They can help to identify indicators to use.
• They can be respondents in beneficiary contact monitoring.
• They can participate in beneficiary assessment.

Desirable Properties of Indicators

Unambiguity: Indicators must be precisely defined so that their measurement and interpretation is unambiguous.

Consistency: Indicators should give objective not subjective data, that is, they should be independent of the person who is collecting the data.

Specificity: The indicators should reflect those things that the project intends to change, avoiding measures that are largely subject to external influences.

Sensitivity: Indicators should be sensitive to project-induced changes.

Ease of Collection: It must be feasible and relatively inexpensive to collect chosen indicators within a reasonable time frame.

Sample Indicators

Input	Process	Impact
For Water supply and Sanitation:
Number of connections and standpipes; number of meters and number of meters working; number of facilities constructed or rehabilitated; training undertaken (including partner organizations).	Percentage of population cover by water and sanitation facilities; latrines built; water sources sanitized; water production; volume billed; consumption by type of user (standpipe, private, commercial, government and internal); availability of maintenance staff; female involvement in water committees; awareness created by health/sanitation message; daily water supply period.	Percentage of the population served by water supply; beneficiary satisfaction with services provided; time spent in water collection; reduction in water related diseases; degree of satisfaction with services provided.
For urban projects:
Execution of subprojects (requested and implemented); number of roads constructed or rehabilitated; staff training; supply of services to needy areas.	Person days of work created; jobs created for unskilled workers.	Employment generation; property/rental values.