OBS Methodology

Open Budget Survey 2015: Methodology

PDFs of the methodology for the 2015 Open Budget Survey and for prior years are available for download in multiple languages at the bottom of this page.

Implementing the Open Budget Survey 2015 and calculating the Open Budget Index and other scores

The Open Budget Survey assesses the three components of a budget accountability system: public availability of budget information; opportunities for the public to participate in the budget process; and the strength of formal oversight institutions, including the legislature and the national audit office (referred to here as the supreme audit institution). The majority of the Survey questions assess what occurs in practice, rather than what is required by law.

The Survey assesses the public availability of budget information by considering the timely release and contents of eight key budget documents that all countries should issue at different points in the budget process, according to generally accepted good practice criteria for public financial management.[1] Many of these criteria are drawn from those developed by multilateral organizations, such as the International Monetary Fund’s (IMF) Code of Good Practices on Fiscal Transparency, the Public Expenditure and Finance Accountability initiative (whose secretariat is hosted by the World Bank), the Organisation for Economic Co-operation and Development’s (OECD) Best Practices for Fiscal Transparency, and the International Organsation of Supreme Audit Institutions’ Lima Declaration of Guidelines of Supreme Audit Precepts. The strength of such guidelines lies in their universal applicability to different budget systems around the world, including countries with different income levels.

The Open Budget Survey 2015 is a collaborative research process in which IBP worked with civil society partners in 102 countries over the past 18 months. The 102 countries cover all regions of the world and all income levels. The Open Budget Survey 2015 is the fifth round; earlier rounds were completed in 2006, 2008, 2010, and 2012. The Survey is typically conducted biennially. However, IBP delayed the most recent round by one year in order to under- take a series of improvements to the Survey questions and to deploy a new online platform to collect Survey results. The changes to the Survey and their implications are discussed in the Technical Note.

The Open Budget Questionnaire

The results for each country in the 2015 Survey are based on a questionnaire, comprising 140 questions, that is completed by researchers typically based in the country surveyed. Almost all of the researchers responsible for completing the questionnaire are from academic institutions or civil society organizations. Although the mandates and areas of interest of the research groups vary widely, all have a common interest in promoting transparent and responsive budgeting practices in their countries. Most of the researchers belong to organizations with a significant focus on budget issues.

Most of the Survey questions require researchers to choose from five responses. Responses “a” or “b” describe best or good practice, with “a” indicating that the full standard is met or exceeded, and “b” indicating the basic elements of the standard have been met. Response “c” corresponds to minimal efforts to attain the relevant standard, while “d” indicates that the standard is not met at all. An “e” response indicates that the standard is not applicable, for example, when an OECD country is asked about the foreign aid it receives. Certain other questions, however, have only three possible responses: “a” (standard met), “b” (standard not met), or “c” (not applicable).

Once completed, the questionnaire responses are quantified. For the questions with five response options: “a” receives a score of 100, “b” receives 67, “c” receives 33, and “d” receives 0. Questions receiving “e” are not included in the country’s aggregated scores. For the questions with three response options: “a” receives 100, “b” receives 0, and “c” responses are not included in the aggregated score.

The Research Process

For the 2015 Survey, researchers began collecting data for the Survey in May 2014 and completed the questionnaire for their country by the end of June 2014. The Open Budget Survey 2015 thus assesses only events, activities, or developments that occurred up to 30 June 2014; any actions occurring after this date are not accounted for in the 2015 Survey results.

All responses to the Survey questions are supported by evidence. This includes citations from budget documents; the country’s laws; or interviews with government officials, legislators, or experts on the country’s budget process. Throughout the research process, IBP staff assisted the researchers in following the Survey methodology, particularly the guidelines for answering Survey questions.[2]

Upon completion, IBP staff members analyzed and discussed each questionnaire with the individual researchers over a three- to six-month period. IBP sought to ensure that all questions were answered in a manner that was internally consistent within each country, and consistent across all survey countries. The answers were also cross checked against published budget documents and reports on fiscal transparency issued by international institutions, such as the IMF, World Bank, and the OECD.

Each questionnaire was then reviewed by an anonymous peer reviewer who has substantial working knowledge of the budget systems in the relevant country. The peer reviewers, who were not associated with the government of the country they reviewed, were identified through professional contacts and variety of other channels.

IBP also invited the governments of nearly all survey countries to comment on the draft Survey results. The decision to invite a government to comment on the draft results was made after consulting with the relevant research organization responsible for the Survey. IBP made a major effort to encourage governments to comment on the draft results; many governments that did not initially respond to IBP letters were contacted on five or six separate occasions. Of the 98 governments that IBP contacted, 53 commented on the Survey results for their country. These comments can be seen in their entirety in the relevant questionnaires at openbudgetsurvey.org.

IBP reviewed peer reviewer comments to ensure that they were consistent with the study’s methodology. Any peer reviewer comments that were inconsistent were removed, and the remaining comments then were shared with researchers. Researchers responded to comments from peer reviewers and their government, if applicable, and IBP refer- eed any conflicting answers in order to ensure consistency across countries in selecting answers.

The Open Budget Index

The Open Budget Index (OBI) assigns each country a score from 0 to 100 based on the simple average of the numerical value of each of the responses to the 109 questions in the questionnaire that assess the public availability of budget information. A country’s OBI score reflects the timeliness and comprehensiveness of publicly available budget information in the eight key budget documents.

MEASURING  THE TIMELY RELEASE OF INFORMATION  TO THE PUBLIC THROUGHOUT THE BUDGET PROCESS
Budget Document Release Deadlines for “Publicly Available” Documents 2015 Survey Question Numbers* Number of questions per document 
Pre-Budget Statement Must be released at least one month before the Executive’s Budget Proposal is submitted to the legislature for consideration. 54-58,
Table 3-pbs
6
Executive’s Budget Proposal and supporting documents Must be released while the legislature is still considering it and before it is approved. In no case would a proposal released after the legislature has approved it be considered “publicly available.” 1-53
Table 3-ebp
54
Enacted Budget Must be released no later than three months after the budget is approved by the legislature. 59-63,
Table 3-eb
6
Citizens Budget Must be published within the same timeframe as the underlying document. For example, a Citizens Budget for the Executive’s Budget Proposal must be released while the legislature is still considering the Executive’s Budget Proposal and before it is approved. 64-67 4
In-Year Reports Must be released no later than three months after the reporting period ends. 68-75,
Table 3-iyr
9
Mid-Year Review Must be released no later than three months after the reporting period ends. 76-83,
Table 3-myr
9
 Year-End Report Must be released no later than 12 months after the end of the fiscal year (the reporting period). 84-96,
Table 3-yer
14
 Audit Report Must be released no later than 18 months after the end of the fiscal year (the reporting period). 97-102,
Table 3-ar
7

* Along with the 102 numbered questions used to assess the eight key budget documents, Table 3 of the Survey questionnaire also contains questions on seven of the eight documents. These are denoted by the acronym for the document. For example, Table 3-pbs refers to the question from Table 3 pertaining to the Pre-Budget Statement.

Measures for Oversight Institutions and Public Participation

The 31 Survey questions that are not used to calculate the OBI assess the opportunities for public engagement during the budget process and the oversight capacity of legislatures and supreme audit institutions. To measure the extent to which governments provide opportunities for public participation in budget decision making and monitoring, as well as the strength of the legislature and the supreme audit institution to conduct oversight, the responses to the questions pertaining to each area are averaged. Thus each area – public participation, legislatures, and supreme audit institutions – is given a separate score.

Compared to the OBI, these measures are less comprehensive (with many fewer questions in each area) and thus provide more of an indication of the state of public participation, legislatures, and supreme audit institutions than a detailed assessment.

Evaluating Oversight Actors and Practices
Indicator Measured 2015 Survey Question Numbers Number of Questions per Indicator
Public engagement in the budget process 114, 119-133 16
Strength of the legislature 103-113 11
Strength of the supreme audit institution 115-118 4

Weighting the relative importance of key budget documents and implications on scores

As mentioned above, each country’s OBI 2015 score is calculated from a subset of 109 Survey questions. Though each of the eight key budget documents assessed may have a different number of questions related to it, the OBI score is a simple average of all 109 questions which had responses a, b, c, or d. In calculating the OBI scores, no method of explicit weighting was used.

Though using a simple average is clear, it implicitly gives more weight to certain budget documents than others. In particular, 54 of the 109 OBI questions assess the public availability and comprehensiveness of the Executive’s Budget Proposal, and thus are key determinants of a country’s overall OBI score. In contrast, the Citizens Budget and the Enacted Budget are the focus of only four and six questions, respectively.

This implicit weighing is justified. From a civil society perspective, the Executive’s Budget Proposal is the most important budget document, as it lays out the government’s budget policy objectives and plans for the upcoming year. It typically provides details on government fiscal policies not available in any other document.  Access to this information is critical for civil society to understand and influence the budget prior its approval, and to have as a resource through- out the year. That said, as discussed in Annex B, one of the changes IBP introduced to the questionnaire was to increase the emphasis on the other seven budget documents, reflect- ing their role in ensuring sufficient information is provided throughout the entire budget cycle.

Do You Have Questions?

This page presents a basic description of the methodology used in producing the Open Budget Survey 2015. For further information about the Open Budget Survey and its methodology, contact IBP at info@internationalbudget.org.


[1] The Open Budget Survey considers a document to be “publicly available” if it is: 1) published by the institution or agency responsible for producing it within a given timeframe; and 2) available at minimal cost to any person who wants the document (i.e., the government must not make documents available selectively).
[2] See the OBS 2015 Questionnaire Guidelines