Back to Table of Contents

Go to: Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 | Part 7

Part IV. The Impact Measurement Process

---

 9. The Impact Measurement Process

Once expenditures are incurred, governments may wish to use an impact measurement process to evaluate whether they have achieved the desired results. This can be an annual, end-of-year activity or part of an ongoing process throughout the budget year. Frequently the executive measures the impact of expenditures, but more recently, countries’ supreme audit institutions have begun measuring budget impact through value-for-money audits (see Part Five).

Unfortunately, most developing countries have not instituted systems to comprehensively measure the impacts of their budgets. In these countries, in-year and year-end government reports show whether money was spent as appropriated in the enacted budgets, and audit reports examine the government’s adherence to financial laws and regulations, but these documents generally do not examine the results of government expenditures.

Presented below are four techniques that a government can use to measure and evaluate the impacts of its budgets.

a. Performance Indicators and Performance Targets

Budgets that include non-financial performance data, including specific performance targets for expenditure programs, are often called “performance budgets” or “outcome budgets.” Countries that identify performance targets for their budgets may include them in budget documents or in associated strategic or performance plans.

Unfortunately, some governments attempt to adopt a performance budget before setting up a support system that would enable officials responsible for implementing the budget to report on performance measures. This can undermine the usefulness of the performance targets.

Table 3 presents examples of performance targets contained in the 2005/06 annual plan of India’s Department of Elementary Education.

Sometimes it is difficult for government to measure its performance accurately or to set targets that will help it achieve its policy goals. For example, agencies that primarily provide support services for other agencies – such as the Department of Finance – may not be able to set the same kind of performance measures that the Department of Education would set.

Box 3: Understanding Performance Indicators Performance indicators can focus on inputs (such as “number of staff required in a hospital”) or outputs (such as “number of patients treated in a hospital”). Though it is more difficult to measure actual outcomes, which can be influenced by multiple factors and can take years to assess, the best indicators do address outcomes (such as “decrease in incidence of HIV/AIDS”) or program efficiency (such as “cost expended per patient served”).

One of the most common errors governments make as they develop performance indicators is to focus on the processes by which they deliver services rather than the results these services are intended to accomplish. For example, an agency may list as its performance indicator the number of meetings it plans to hold to formulate a policy rather than establishing indicators that assess the policy’s intended impact (such as a reduction in maternal mortality).

Agencies may also formulate performance measures that are not easily measurable. For example, an agency may set as a performance measure that it will “provide the best urban transportation service in the country” rather than that it will “operate 50 new bus routes and purchase 200 new buses for these routes.”

Performance indicators included in budget documents should clearly indicate what government will try to achieve – and enable observers to monitor what has actually been achieved. A year-end report or related documents should compare performance targets with actual results.

One method often used to promote the utility of performance measurement is the SMART system. According to this system, performance indicators should be:

Specific: The indicator should clearly define what should be done, by whom, and where.
Measurable: The indicator should contain an answer to the question, “How much or how many?”
Attainable: Goals should be attainable during the period for which they have been set.
Realistic: Goals should be attainable through steps that are practically achievable in light of government’s capacity and the external environment.
Timely: A goal should have a specific time frame. Given the annual nature of budgets, performance goals submitted with the budget should generally cover a one-year period.

b. Efficiency Measures

Some agencies attempt to monitor their own efficiency by conducting internal performance reviews using efficiency measures, such as the share of total expenditures devoted to administration (refer to Table 4) or to personnel. Such indicators become useful measures of efficiency when results are tracked over time or compared to results obtained by other, similar agencies.

Table 4: Administrative Expenditures as a Share of Total Budget Allocation per Department
All figures are stated in percentages

c. Use of Customer Satisfaction Surveys

Performance monitoring can provide information on whether services have been delivered and what they achieved, while efficiency reviews can provide information on how services were delivered. But neither tool can tell us whether the people who received the services were satisfied with them.

In order to collect this type of information, some governments conduct “customer” (user) satisfaction surveys. Such surveys generally pose two kinds of questions: whether users felt that the service was appropriate to their needs (i.e., would they prefer government to provide something else instead?) and whether they were satisfied with the service delivered.

One example is the 2006 survey by the government of the Philippines on how many customers the civil registry service of the National Statistic Office served within 30 minutes of their arrival in the office (National Statistical Office, GoP, 2006). It states in part, “The number of satisfied clients who reported to have been served within 30 minutes increased by 17 percentage points to 61 percent from 44 percent in September 2006. It was also 14 percentage points higher than a year ago in December 2005 when 47 percent of the clients were satisfactorily served within 30 minutes.”

d. Value-for-Money Audits

Performance measurement systems generally indicate whether services were delivered – but they often do not tell us much about how these services were delivered. As a result, they often do not provide information that can help the government or the public measure waste in government spending or assess the quality of service delivery.

In some countries, public audit institutions have initiated value-for-money (or performance) audits that try to assess the economy, efficiency, and effectiveness of government service delivery. When done properly, these audits provide a good measure of government performance.

Canada’s national audit institution, the Office of the Auditor General, uses four tools to conduct value-for-money audits (OAG, Canada, 2004):

(i) Case studies highlight good practices by comparing instances of good performance with instances of poor performance and explaining why the one provided better value than the other.

(ii) Performance benchmarking utilizes a range of measures – such as resource utilization, unit costs, and efficiency – to compare performance among several agencies.

(iii) Focus groups of 8 to 12 people can be selected as samples of a larger population (such as senior officials, service users, etc.) and then enlisted to provide feedback on the quality of services delivered by the government. The evidence gathered from focus groups is qualitative in nature and gives real-life perspectives on a given topic. Focus groups are particularly useful in helping to identify issues, understand why particular actions have been taken, test emerging findings to generate survey questions, and develop practical approaches to policy issues.

(iv) High-level comparisons with other organizations that perform similar functions can also provide useful information on an entity’s performance. A number of criteria can be used, such as organizational structure, operational policies, and problem solving processes. The emphasis is on learning from others’ experiences and putting findings regarding the organization in a broader context.

This issue is discussed in greater detail in Part Five.

---

10. Case Studies of Successful Civil Society Initiatives to Measure Budget Impact

This chapter presents two case studies profiling successful civil society projects to measure the impact of budget execution. The first, from India, describes “citizen report cards” that measure public satisfaction with the delivery of public services. The second, from Tanzania, describes a community scorecard used to track government expenditures in local communities and gauge public satisfaction with government services.

1. Public Affairs Centre Develops Citizen Report Cards in India

Organizational Profile The Public Affairs Centre (PAC), a non-profit organization formed in India in 1996, is dedicated to improving the quality of governance in the country. Beginning as a small citizens’ initiative to help citizens make informed choices during the Bangalore municipal elections, PAC has since striven to enhance transparency, accountability, and citizen participation in electoral and governance processes. PAC undertakes and supports research on public policy and services, disseminates research findings, facilitates collective action, and provides advisory services to state and non-state agencies.

a. Introduction

Bangalore is one of India’s largest cities and a major software and industrial center.¹³ However, the city suffers from poor public services. Agencies in charge of municipal services (water, garbage collection, road and park maintenance, etc.) are unreliable, and corruption is rampant among all service providers. Inspired by the private sector practice in India of conducting client satisfaction surveys, a group of residents undertook a citizen report card exercise in 1993 to measure citizen satisfaction with public service providers. Subsequently, the group formed PAC to undertake additional surveys. The report card exercise raises awareness of service providers’ poor performance and compels them to take corrective action.

The initiative asks users: How satisfactory are the public services you receive? Which aspects of the services are satisfactory and which are not? What are the direct and indirect costs (including bribes) of acquiring these services?

The first report card exercise, in 1993, surveyed 480 middle-income and 330 low-income households that had interacted with one of eight selected public service providers in the preceding six months. Subsequent surveys in 1999 and 2003 focused on seven of these agencies, which were chosen because they served the largest number of people. Agencies assessed in all three report card exercises to date include the water and sewer board, electricity board, public hospitals, development authority, and regional transport office. The questionnaire used for the survey measured user satisfaction in such areas as staff behavior, the number of visits required to complete a task, the ease with which problems were resolved, and the quality of information provided.

The 1993 exercise revealed low levels of public satisfaction with all service providers. Just one percent of the people surveyed expressed satisfaction with the Bangalore Development Authority, for example. Corruption was widespread in almost all agencies; one-third of the poor households surveyed had paid a bribe to public officials in the previous six months. Middle-income customers generally gave low marks in regard to staff behavior, problem resolution, and the number of visits required. The situation was even worse for the poor, many of whom reported ill-treatment by public officials.

These results were shared with senior agency officials and were widely publicized. Many newspapers and magazines highlighted the findings of corruption in public services. The findings were also disseminated through seminars and meetings across the city. A series of “open house” meetings brought together citizen groups that had not been involved in the survey.

The report card exercise was repeated in 1999 and 2003. The 1999 report card results showed a partial improvement for two agencies, but overall citizen satisfaction remained low and respondents seemed even less satisfied with staff in all of the agencies than in 1993. The 2003 report card, in contrast, revealed substantial improvement in almost all agencies; average user satisfaction increased by more than 40 percent between 1999 and 2003.

b. Methodology

The process of developing a citizen report card (CRC) can be divided into six phases, described below. (For more detail, see Wagle et al., “Citizen Report Card Surveys – A Note on the Concept and Methodology,” 2004.)

Phase One: Identification of Scope, Actors, and Purpose
First Step: Clarify Scope of CRC Evaluation

The first step is defining the scope of the survey to be conducted, namely, what type(s) of public services will be assessed and how the findings will be used. CRC surveys are often repeated on a single subject or in a single community; the first CRC is frequently used to identify poor service providers, while later CRCs are used to determine whether providers have improved in response to earlier survey findings.

Second Step: Coalition Building for Credibility

The credibility of survey findings depends to some degree on the initial legitimacy of the group conducting the survey. In some cases, the involvement of international organizations can heighten the survey’s credibility, but in other cases this type of external involvement may be ineffectual or even counterproductive. The World Bank, for instance, has technically proficient staff who can assist or fund an organization conducting a survey, but they are not always looked upon favorably.

Phase Two: Questionnaire Design
First Step: Arrange Focus Group Sessions With Service Providers and Service Users

Prior to soliciting views from service users, the group conducting the survey must decide which users will be surveyed – for example, those who have used the services within the past three months, six months, 18 months, or during any time period? Service providers can be asked for information on the services they provide and on needed improvements that have already been identified. The information gathered from these focus groups should inform the questionnaire content.

Second Step: Define the Structure and Size of the Questionnaire

As with any survey, time and quality constraints limit the number of questions that can be asked. One useful practice is to break the questionnaire into sections that different members of the household can answer. Another is to conduct “rotating interviews,” in which the first household is asked questions related to one set of public services, the next household is asked about a different set of services, and so on.

Guidelines for CRC Questions

Where standards for a particular public service exist, the questionnaire should check whether they are being met. For example, if the provider of drinking water has committed to provide water once a day, include a specific question to assess whether service is provided as promised:

Sample question: How often do you get drinking water?
1 – More than once a day 2 – Once a day 3 – Less than once a day

When standards do not exist (which is the case in many countries), another way must be found to evaluate the quality of service delivery. The following are examples of questions that can be asked to obtain feedback on the accessibility of health centers.

Option 1: How long does it take to travel to the health center that you most regularly visit? Followup question: And what is your mode of transportation?
Option 2: What is the distance from your house to the health center that you most regularly visit?

Both options provide useful information, so try to identify the type of feedback that would be most useful given the purpose of the CRC and the local setting.

To avoid collecting outdated information, include a time frame when necessary. For example, one might want feedback only from households that have used a service in the past year.

Sample question: Have you or anyone in your household used public hospital services in the past year?

If you are asking a distance-related question or any other question for which the unit of information (distance in kilometers, time in minutes, etc.) affects the respondent’s answer, make sure the unit is identified in the question. For open-ended questions, ask investigators to note the unit of measurement used in the answer, even when the unit is specified in the question. This provides a double-check and allows for conversions when a respondent’s answer is given in a unit different from the one in the question.

Sample question: On average, how many minutes do you wait to see a member of the hospital staff?

Several types of answer scales are commonly used to collect feedback. The type of scale used affects the interpretation of the survey findings: extensive scales allow for more nuanced conclusions, while simple scales provide findings that are easy to convey.

Sample question: How do you rate the reliability of public transport service in your area/village? 1 – Good, 2 – Average, 3 – Poor, 4 – Not applicable

Finally, questions should be written as neutrally as possible. A question such as “What do you dislike the most about the services you receive?” presupposes that the user dislikes something about the services unless balanced by another question that asks: “What do you like the most about the services you receive?”

Third Step: Pre-Test the Questionnaire

Questionnaires should be tested on focus groups similar to those organized in the first step, and any necessary modifications made prior to the survey’s launch. The persons who will conduct the survey should also test it before going into the field, and time should be allowed to modify questions or questioning methods based on their feedback. If a question confuses a significant number of people or requires a great deal of clarification, it should probably be changed.

If the survey has been translated into multiple languages, all surveys should be re-translated back into the primary language (by someone other than the original translator) before the survey is conducted, to ensure consistency across all instruments.

Phase 3: Sampling
First Step: Identify the Geographic Regions from Which Respondents Will Be Drawn

Prior to determining the survey sample size, attention should be paid to the geographic region(s) in which the survey will be launched. It may be useful to divide the regions into areas based on factors such as type of housing, age of locality, or median income. Within these divisions, select a number of localities from which households in different income brackets will be chosen once the survey sample size has been decided.

Second Step: Determine Survey Sample Size

Budget, time, and organizational capacity will limit the survey size; enumerators (the people who conduct the survey) must be paid and perhaps transported to the survey areas, leaflets must be printed, and so on. Keep in mind that the goal is to achieve good representation of different parts of the population rather than simply to generate a large number of completed questionnaires. Try to capture as many social strata as possible in the locality being surveyed. PAC has found that a sample size of 300-350 households is ideal for each public service that is the subject of a survey.

Third Step: Choose Sample Respondents

In a CRC survey, the most likely unit of analysis is the household. However, even within a household, sample respondents need to be selected. Typically, the head of the household will be chosen to respond, though if the survey is broken into different sets of questions, other household members may be approached for answers. Other members of the household may also be better informed about some questions than the head of the household; for example, someone who does not work outside the home may know more about daytime power outages than a household head who works outside the home.

Proper sampling is no easy task. One useful technique is random sampling, in which each household is assigned a number and then numbers are drawn at random; households with numbers corresponding to the drawn numbers are surveyed. Another useful technique is stratified random sampling, in which researchers establish categories such as men/women or slum/non-slum households and then choose random samples from within each category. Many other ways of sampling exist as well.

Phase 4: Execution of Survey
First Step: Select and Train a Team of Enumerators

Survey personnel should have a good understanding of the purpose of the project and how the survey contributes to this purpose. They should receive training before being sent to conduct the survey. In some cases, it may be useful to employ female interviewers to speak to female respondents and male interviewers to speak to male respondents.

Second Step: Perform Random Checks of Interviews

To ensure the survey’s credibility is not compromised by inaccurate recording of household responses, it is useful to perform random spot monitoring of interviews. If survey questions are misinterpreted or some answers are found to be inconsistent, re-interviewing is required. Enumerators should ask the questions exactly as they are written. This prevents an interviewer’s own biases from affecting the answers and ensures better comparability across survey responses.

After completing each interview, survey personnel should review the collected information and identify any inconsistencies. Only after the collected information has been deemed accurate and satisfactory should it be entered into data spreadsheets.

Phase 5: Analyze Data

Once all the data have been consolidated, analysis can begin. Respondents likely will have rated government services on a scale, for example, -5 to +5, or 1 to 7. One way to generate aggregate scores is to tally the responses for a common set of questions, calculate the average response, and then express this average as a percentage. For example, if the average user rating on a particular public agency’s overall effectiveness was 3.7 on a scale of 1 to 7, the percentage would be 53 percent (3.7 divided by 7). This percentage can be read as the “grade” for the report card.

Statistical tests should be run on the data to determine whether the survey results can be applied to the greater population and whether differences between sub-groups are statistically significant. It is helpful – if not essential – to find a person or group (for example, a research center at a university, a graduate student, or an experienced survey company) with the statistical analysis skills to assist with this phase of the process, if these skills are not available internally.

Phase 6: Dissemination
First Step: Engage Officials

It may not be helpful to use report card results to publicly embarrass particular service providers. Instead, one can first share the preliminary findings with them so they can respond; any genuine explanations should then be noted in the final report and factored into the recommendations.

Second Step: Engage the Media

The findings from the report card initiative could be presented at a press conference or similar event. To increase coverage of the event, the group that conducted the survey can prepare press kits that include brief printable stories, media-friendly press releases, and translations of the report into local languages. The group should seek to attract multiple media formats – including print, television, radio, and new media such as websites, discussion boards, and blogs.

Third Step: Foster Communication between Service Providers and Users

Bringing together service providers and users after the report cards have been published gives both parties a chance to discuss their reactions. These discussions can put added pressure on service providers to improve their performance; at the very least, they allow users to voice their opinions. If possible, schedule an event like a public hearing during which the public can ask questions of service providers.

Fourth Step: Present Your Message

The final report should present the survey results, draw conclusions from them, and recommend steps to fix any problems the survey identified. It should include both the positive and the negative results, and apart from exceptional cases, it should be a catalyst for change rather than a condemnation of service providers.

For a more in-depth look at CRCs, consult Improving Local Governance and Service Delivery: Citizen Report Card Learning Toolkit, created by PAC, the Asian Development Bank, and the Asian Development Bank Institute, at http://www.citizenreportcard.com/index.html. The appendices of the PDF version include additional resources such as sample press releases, sample final reports, and budget checklists.

c. Results Achieved

Successes

Public satisfaction with service delivery fluctuated among the three surveys conducted over the ten-year period in Bangalore – though the 2003 results showed remarkable improvements in public satisfaction over the 1993 results. Even though the 1999 survey did not compare very favorably to the 1993 survey, its findings showed that some agencies had attempted to respond to the public dissatisfaction revealed in the first report card.

Three agencies – Bangalore Telecom, the Electricity Board, and the Water and Sewerage Board – streamlined their bill collection systems after the 1999 survey. With PAC’s assistance, the Bangalore Development Authority developed its own report card, which it used to obtain feedback from customers on corruption and to identify weaknesses in service delivery. The Bangalore City Corporation and the Bangalore Development Authority also initiated a joint forum of representatives from non-governmental organizations and public officials to identify solutions to high-priority problems. Two large public hospitals in the city that had received very poor rankings agreed to support an initiative designed by a non-governmental organization to set up “help desks” to assist patients and to train their staff to be more responsive to patients’ needs.

PAC acknowledges that a number of factors caused the dramatic improvement in agency performance between 1999 and 2003. These include pressure from international donor organizations and a responsive chief minister in the state, who encouraged civic participation in governance. However, ten years of sustained advocacy and media publicity regarding the report card results also played a major role in pressuring city agencies to improve their service delivery.

Challenges

Completing a CRC project can take six months to a year, or even longer. PAC has assisted groups in many other countries seeking to implement its methodology and has developed a list of issues that any group interested in conducting a report card survey should consider.

i. Requirement of a Strong Lead Institution: The ultimate success of a CRC project depends in large part on the institution that leads it. This institution should be legitimate in the eyes of those who will be surveyed and familiar with the locality where the CRC is to take place. It should also be experienced in conducting surveys and willing to work with multiple stakeholders drawn from throughout society. It is also helpful if the institution can draw from an established network of organizations and individuals to supplement its own skills and personnel.

ii. Evaluation of the Socio-Political Context: Governments must be able to respond to feedback in order for a CRC to produce meaningful changes. Furthermore, the relationships among different sectors of society (government, media, civil society, businesses, and citizens) must be conducive to the use of a CRC. Citizens must not be too intimidated to respond to survey questions, and the safety of enumerators and respondents should not be in question.

iii. Development of an Advocacy Strategy: Advocacy efforts should always be directed to the level of government (local, state, or national) responsible for the service being assessed. Further, the group conducting the CRC survey should cultivate strong relationships with local media, which can help disseminate the survey results and thereby build the pressure needed to effect change in a service provider’s practices. The greater the amount of media censorship in a country, the less likely it is that a group’s advocacy efforts will succeed.

Advocacy efforts will be even less likely to succeed if service providers are not willing to change their practices. Including some survey results that reflect favorably upon the service provider will help the provider feel more comfortable with the process. Repeating the report card at regular intervals also gives providers input on whether their efforts to improve services are working, which could even create an incentive for them to become involved in the survey process.

iv. Requirement of Technical Skills: The group conducting the CRC survey may need technical assistance from outside groups on such issues as survey techniques, details of local service provision, and survey fieldwork. Universities and private companies can be good places to find people with skills in these technical areas.

v. Consideration of Cost: The cost of a CRC survey will vary depending on factors such as the sample size, the number of personnel needed to conduct the survey and the level of training they will need, communication and information equipment needed (computers, phones, etc.), the cost of printing questionnaires, wages to be paid to interviewers and supervisors, any fees due to outside agencies to which certain tasks have been outsourced, and travel and dissemination costs.

Information on PAC can be obtained from the organization’s website, http://www.pacindia.org.

2. Hakikazi Catalyst Uses PIMA Cards in Tanzania

Organizational Profile Formed in 2000, Hakikazi Catalyst is a Tanzania-based economic and social justice advocacy organization that empowers marginalized people both to influence government decisions affecting their lives and to achieve their civil and political rights at the local, national, and international levels. The organization has 14 full-time staff members and is governed by a board of directors, an advisory council, and a committee of members. Hakikazi produces a number of publications every year that disseminate information on public policy to local communities in accessible and popular formats. Its overarching goals are to support the initiatives of poor and marginalized communities, influence change in national and local policies, and eradicate poverty.

a. Introduction

In 2003, the International Budget Project organized a conference in Mexico that brought together budget advocacy organizations from 40 countries. One of the presentations was made by the Public Affairs Centre from India, which has successfully implemented citizen report cards to evaluate public satisfaction with government service delivery (see previous case study). After attending this conference, the director of a Tanzanian economic and social justice advocacy organization, Hakikazi Catalyst, decided to adapt the citizen report card methodology to his country context and developed the PIMA card.

Hakikazi’s PIMA cards (pima means “measure” in Swahili) provide a simple, flexible evaluation tool that enables communities to gather qualitative and quantitative information on inputs (what funds did the community receive?), outputs (how were the funds used?), and outcomes (how did the projects affect the community?) of government expenditures on poverty-reduction strategies.

Under this methodology, both local communities and district governments complete a card assessing the quality of goods and services provided by the district government to local communities. Based on the results, the district government and local communities decide on the next steps to be taken to address communities’ priorities and to continue information-sharing in a systematic way.

Between 2003 and 2007, Hakikazi completed two budget monitoring exercises using the PIMA card process. The main objective was to determine how the government’s poverty reduction strategies are being funded and implemented at the local level and whether they are actually improving the welfare of poor communities.

The government’s monitoring system for Tanzania’s poverty reduction strategy, popularly known by its Swahili acronym MKUKUTA, is set forth in MKUKUTA Monitoring Master Plan and Indicator Information (2006), which describes a framework to enhance the participation of all stakeholders – particularly civil society – in monitoring the strategy’s implementation. The document also outlines specific indicators for each of the goals in MKUKUTA, which are useful tools for civil society organizations and other groups conducting monitoring activities.

b. Methodology

The PIMA card process involves eight steps:¹⁴

Step 1: District-Level Groundwork

Hakikazi selects villages to participate in the exercise based on the relationships the group has developed with them over time. It organizes district workshops to mobilize stakeholders, explain budget monitoring systems and the PIMA card process, and generate support for the process. Participants include village and ward-level government leaders; district officials from the planning, agriculture, natural resources, education, health, and community development departments; district council members; civil society organizations; and community representatives. During these meetings, local officials are approached about providing information on budgeting and planning on a regular (quarterly) basis in an agreed-upon format. If an agreement is reached, this information is disseminated to participating communities.

Step 2: Skills Building

Hakikazi organizes workshops to train individuals within a community (drawn from existing community-based organizations) who will lead the PIMA card process. These “training of trainers” workshops are a key component of the MKUKUTA monitoring process. They provide participants with the skills to gather quantitative and qualitative budget information in communities, analyze government budgets, and present their results to decision-makers and communities.

Civil society representatives, local government staff (e.g., community development workers), and research facilitators from Hakikazi participate in the training workshops and are involved in every subsequent step of the PIMA card process, including designing the PIMA cards, analyzing budgets, facilitating community meetings, facilitating data collection in communities, and analyzing and disseminating findings.

Step 3: Community-Level Groundwork

Next, Hakikazi convenes public debates on MKUKUTA in the participating communities. Their objective is to enable communities to understand MKUKUTA and its monitoring system and to provide feedback on poverty reduction strategies.

At the public debates, communities discuss the causes of poverty; MKUKUTA strategies, targets, and indicators; and the purpose and benefits of budget monitoring in general and PIMA cards in particular. Following discussions in small groups, community members select two of the MKUKUTA sectors they want to monitor, such as education, health, roads, agriculture, or water. Each community also selects seven to 15 people to serve as a village monitoring committee, which will collect information on the selected priority areas using the PIMA cards. Members are chosen from a broad cross-section of the community, including youth, women, elderly people, and disabled people.

Step 4: Design of Village PIMA Cards

Monitors use the PIMA cards to collect information in their communities on the quality and quantity of expenditures in the sectors under investigation, including:

• which MKUKUTA activities were allocated funds in the district budget in the selected sectors;
• how much funding was received in the community for sector activities in the last 12 months (inputs);
• how many activities were completed in the community in the last 12 months (outputs);
• what were the actual expenditures on those activities;
• what was the level of community satisfaction with the results of activities;
• which MKUKUTA activities carried out in the community in the last 12 months were not allocated funds in the district budget (and how they were funded); and
• what are the community’s other priority issues, ranked by importance.

Table 5 illustrates the information collection process for one common budget expenditure: agriculture and markets. Hakikazi creates similar PIMA cards for each of the other budget expenditures that a community prioritizes for monitoring.

Table 5: Sample Village PIMA Card A – The Village Development Budget

Hakikazi provides the following instructions for completing this sample village PIMA card:

• The PIMA card should be completed using the village development budget.
• The source of the funds should be stated. For example, was the money provided by the government, community contributions, or an organization?
• The actual amount spent on the development work should be stated. This should be available in the village bank statement(s) and accounting records maintained by the village government.
• Finally, it should be stated whether the work was completed and whether the village is benefiting from the work in the way intended. This information should be based on a physical verification of the project site(s).

Table 6 is a sample village PIMA card that collects information on the quality of expenditures in the agriculture and markets sector.

Table 6: Abstract of Village PIMA Card B – Production Factors

Hakikazi provides the following instructions for completing this card:

• Consider and record all the training or information services your community has received from agricultural experts in the last 12 months.
• If training or information services were not received in the last 12 months, please mark the box indicating that the service was “Not Received.”
• If a service was received, discuss and agree on whether the service was Poor, Satisfactory, or Good and mark the corresponding box.

Similar instructions are provided on the PIMA card to collect information on many other service delivery and infrastructure issues, such as access to agricultural credit, access to seeds and fertilizers, and the condition of irrigation infrastructure.

Step 5: Design of District-Level PIMA Cards

The district-level PIMA Card is called a district self-evaluation and is completed by a district-level government official. Hakikazi designs a self-evaluation for each sector selected by communities for the monitoring exercise. For example, the district agriculture officer should complete the self-evaluation for the agriculture sector (refer to Table 7). The questions in the self-evaluation card should mirror the questions asked by the village monitors at the community level. For example, the card should ask about MKUKUTA activities funded in the budget, the sources of funds, actual expenditures, and the impact of funded activities.

Table 7: Sample Abstract of District Self-Evaluation Card

Step 6: Information Collection with PIMA Cards

Information collection at the village level generally takes about two weeks. Hakikazi and its facilitators start by training one village monitoring committee on how to collect data using the PIMA card, which is then tested in that committee’s community. If needed, the PIMA card is revised after the testing and then used to train other village monitoring committees.

With support from Hakikazi’s trained facilitators, the village monitoring committees collect information on allocations received from the district and on expenditures at the community level through analysis of the village government’s bank statements, accounting records, and receipts. Hakikazi reviews this information for accuracy and consistency and obtains more information from communities, if necessary. When the PIMA cards are completed, Hakikazi facilitators work with village monitoring committees to help them summarize their findings, which are then presented and discussed at village meetings.

Step 7: Analysis of Local Government Budgets

Hakikazi analyzes district budgets to identify budget allocations for the sectors selected for monitoring. District budgets, obtained from the district planning office, show approved expenditure estimates – the funds that are meant to be disbursed to village governments and service providers. The district budget analysis is conducted to identify:

• recurrent (operating budget) expenditures and development (capital and operating budgets for new programs) expenditures for the selected sectors;
• what MKUKUTA activities for the selected sectors were included in the budgets;
• what MKUKUTA activities were not included in the budgets; and
• expenditures in “Other Charges” and development (capital) budgets for MKUKUTA activities.

Hakikazi compares the results from its budget analysis with the results reported in the PIMA cards completed by the district officials (through the self-evaluation) and by the communities.

Step 8: Analysis and Feedback

A report combining information from both the district and communities is drafted, peer-reviewed, and then shared with the communities and local government. Community representatives and local government officials then meet to discuss the monitoring results, decide on next steps to address communities’ priorities, and determine how to continue information-sharing.

The results of the PIMA card studies are shared at the community level (where the village government can act upon them), the district and regional levels (where key practical decisions that favor poor people can be influenced), and the national level (where policymaking bodies can respond to them).

Hakikazi also collaborates with other civil society organizations to use its findings to influence pro-poor decision-making at all levels of government.

c. Results Achieved

Successes

As stated earlier, Hakikazi has undertaken PIMA card studies only twice. However, it has already achieved some success in identifying problems in village development expenditures. For example, one of the communities that participated in Hakikazi’s 2006 project is Mkonoo, a village near Arusha in northern Tanzania. Mkonoo’s village assembly chose to monitor the use of primary education funds and made the following discoveries:

• Of the Tshs. 9,300,000 (approximately US$7,500) withdrawn from the village’s bank account, no records were available to account for expenditures totaling Tshs. 1,100,000 (approximately US$880).
• A physical verification of the schools constructed during the year under investigation revealed that the iron sheets used for roofing on the classrooms were thinner than those listed in the budget (and required by government standards). The thinner sheets that were used could easily blow off during a strong storm.
• The village government also had no explanation or accounts for Tshs. 1,124,700 (approximately US$900) allocated for the construction of a teacher’s house, even though this amount had been withdrawn from the village bank account.

The misuse of funds was brought to the attention of the village, ward, and district governments by the village monitoring committee and Hakikazi. The district government subsequently formed a team to investigate. At the time of this writing, the government’s Prevention of Corruption Bureau was also investigating the matter, since district officials may have been involved. This is one small example of how simple, community-level monitoring can lead to improvements in local governance and accountability and, ultimately, in the lives of poor people. Hakikazi has achieved similar successes in other communities too.

Challenges

Hakikazi relies on a government guideline that requires local governments to provide information to the public upon request. However, in the absence of a national right to information law, access to information remains a major obstacle for Hakikazi and the local communities that implement the PIMA card process.

Hakikazi has also found that variations in the standard of facilitation during the information collection process lead to variations in the completed PIMA cards. In the future, it will be necessary to ensure that all facilitators are committed and competent.

Finally, analysis of district budgets has often been difficult and time-consuming due to their opaque and inconsistent presentation. It is also frequently questionable whether budget documents provide honest representations of what development activities are realistically possible, given the extreme financial constraints.

In spite of these challenges, Hakikazi plans to continue improving the PIMA card process and to build on its expenditure tracking experience by working in additional communities and districts.

Information on Hakikazi can be obtained from the organization’s website, http://www.hakikazi.org.

¹³ This case study draws on Bhatnagar et al., 2006 and Thampi, 2006.
¹⁴ This section draws on Hakikazi Catalyst, REPOA, TGNP, 2007.

---

11. Other Successful Initiatives to Measure Budget Impact

This chapter presents two additional case studies profiling successful civil society projects to measure the impact of a government’s budget execution. The first, from Nicaragua, describes an effort to track the impact of hurricane relief aid and gauge the level of public satisfaction with the government’s response to the hurricane. The second, from Guatemala, describes an effort to monitor the impact of government education programs through local surveys.

1. Coordinadora Civil Compiled Customer Satisfaction Surveys to Measure the Impact of Hurricane Relief Aid in Nicaragua

Following the devastation wrought in Nicaragua by Hurricane Mitch in 1998, civil society was concerned with the government’s management of the resulting international aid flows. In addition, there were rumors about discrimination in the delivery of aid based on party or religious preferences and about the use of aid funds for unofficial purposes. A local civil society network, Coordinadora Civil para la Emergencia y la Reconstrucción, decided to investigate these claims.

Because the large number of affected communities made it difficult to measure the distribution of aid, Coordinadora Civil undertook an audit of communities’ perceptions of aid delivery. The first phase of the audit was carried out through surveys and interviews with respondents from roughly 10,000 households in 61 municipalities, along with 179 community leaders, 48 mayors, and 82 directors of non-governmental organizations and associations. Interviewees were asked when aid had been delivered and by whom, how much had been delivered, and whether the aid had been appropriate for each situation.

In the second phase, Coordinadora Civil sought to gauge households’ perceptions about whether their conditions had improved and how they evaluated the government’s (and other organizations’) service delivery. Of the 6,000 households interviewed, 29 percent said their situation had improved, 34 percent said it had remained the same, and 37 percent said it had worsened. When asked the most important action the government had taken in response to the hurricane, 60 percent said “nothing,” 29 percent identified some action, and 10 percent said they did not know. When asked to identify civil society, religious, national, and international organizations that had responded to the disaster, 49 percent mentioned an international organization, 27 percent a non-governmental organization, 18 percent the Red Cross, and 6 percent a religious organization. The audit also obtained data on the percentages of households that currently receive agricultural, housing, and food assistance.

Coordinadora Civil also sought to highlight the most pressing needs by asking interviewees to identify problems they felt should be reconstruction priorities. Responses included emotional despair, mistreatment of women (which many feel has increased since the hurricane), and inadequate preparation for future disasters. Coordinadora Civil then designed a survey of community leaders and households heads to measure the levels of these problems and the hurricane’s impact on them. The survey results, along with the final audit report, were delivered to all survey participants.

Coordinadora Civil compared these responses to official reports from donor governments and institutions and the Nicaraguan government. The audit confirmed the charge that aid distribution had been mismanaged. The group shared its findings with local organizations and authorities and then informed government officials, national organizations, the international community, and the media. Reporters subsequently found additional information supporting the audit’s results. The government first denied the findings, then began harassing those who presented them. However, the audit subsequently helped form the basis for investigations that led to the 2003 imprisonment of former president Arnoldo Alemán for money laundering and embezzlement (Quiros, 2005).

Information on Coordinadora Civil can be obtained from the organization’s website, http://www.ccer.org.ni/.

2. Asociación de Investigación y Estudios Sociales Tracks Education Expenditures in Guatemala

Guatemala has one of the lowest educational standards in Central America. In 2002, more than 31 percent of the country’s population was illiterate, student dropout rates averaged approximately 50 percent annually (reaching as high as 75 percent in rural areas), and investment in education as a share of GDP was less than 7 percent.¹⁵ By comparison, Guatemala’s neighbors invest approximately 10 percent of their GDP in education.

The Ministry of Education initiated a number of programs in 1996 to improve school retention rates, including free breakfasts and lunches for all students and the distribution of free textbooks for all students. However, almost seven years later, there was still no tangible evidence that education standards in the country were improving.

In 2002, ASIES – a research and advocacy organization – launched a Grand National Campaign for Education (Gran Campaña por la Educación or GCNE), working with a coalition of 77 non-governmental organizations from around the country. Since then, GCNE has organized a number of community-focused surveys (which it calls social audits) to monitor the implementation of public primary school programs, particularly programs that provide free meals and textbooks.

Since the government does not maintain detailed information on education programs, GCNE has come to rely on its own surveys, instituted in individual public schools, for information on the delivery of education programs. GCNE uses a scientific stratified sampling methodology to select schools from across the country for its surveys. The surveys target principals, teachers, parents, and students regarding issues such as their awareness of the availability of free meals and textbooks, the adequacy of the budgets for these programs, and their level of satisfaction with these programs.

The surveys have uncovered startling findings that help explain the country’s lack of improvement in education. For example, one survey found that approximately 80 percent of principals were unaware of the free meal program and that approximately 75 percent of schools did not receive textbooks for all students.

GCNE prepares a formal report following each survey (see http://www.asies.org.gt/grancampaña/CGAUDITORIASOCIAL2005.pdf) and has made a series of recommendations to the government, such as that it drastically increase the number of free textbooks it distributes. GCNE also sends an annual report to senior government officials, the media, and other stakeholders, including the teachers’ union. In 2006, GCNE’s persistent efforts won a commitment from the Ministry of Education that it will follow up on the results of the audit.

Information on ASIES can be obtained from the organization’s website, http://www.asies.org.gt/.

¹⁵ The case study draws on ASIES, 2003 and Rubio, 2002

Back to Table of Contents

Go to: Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 | Part 7