Policing the Policemen - the case for an independent evaluation mechanism for the IMF
Briefing||14 June 2000|
Angela Wood , Bretton Woods Project
Carol Welch, Friends of the Earth US
Thank you to Marijke Torfs, Friends of the Earth US, and Matthew Lockwood, Christian Aid, for their helpful comments on earlier drafts of this briefing.
We would also like to thank Yassine Fall for her analysis of the impacts of the ESAF programme in Senegal, and Ross Hammond and the Evangelical Lutheran Church in Tanzania for the assessment of Tanzania's ESAF programme.
This briefing paper was produced with support from the C. S. Mott Foundation.
Non-government organizations (NGOs) have been calling for reform of the International Monetary Fund (IMF) since the 1980s, and systematic campaigns on the institution have been ongoing for about a decade. One critical component of these campaigns has been the push for an independent process of evaluation to assess the impacts of IMF programs and policies.
As a public institution, the IMF is accountable to the people who Fund it - taxpayers from around the world. However, accountability by the IMF has been severely hampered by the lack of information disclosure and public participation, and by the IMF's failure to develop an adequate evaluation process.
Establishing an effective evaluation unit at the IMF is an imperative if reformers wish to make the IMF an institution that makes the best use of its money by benefiting the people most in need- the poorest sections of society- and by promoting long term development that incorporates social and environmental sustainability into the economic equation.
In late 1996, the IMF established a trial external evaluation process. So far it has only conducted one review which examined the effectiveness of the Enhanced Structural Adjustment Facility. The findings of this review were published in March this year. This mechanism was established in response to calls for an independent review mechanism. While this is an important step, and NGOs commend the IMF for taking it, it is only the first step towards a more autonomous evaluation procedure.
The precise nature of an independent evaluation body at the IMF is a negotiable matter. NGOs would like to work with the IMF to develop a process that is in the interest of both the Fund and civil society. However, there are certain principles, such as transparency and public participation, that are essential for achieving a valid process. These principles and possible structures for an evaluation unit are discussed in section three. Section two examines why an evaluation unit is necessary and why current evaluation mechanisms employed by the IMF are inadequate.
The IMF asserts that the quality of the development process corresponds in part to the quality of governance and it is increasingly including good governance criteria in its reform programmes. But whilst it advocates good governance for governments it has not applied the philosophy to its own activities.
One of the central requirements of good governance is accountability to the public. The principle of accountability asserts that decision makers should be held responsible for the consequences of their decisions. Therefore, accountability encompasses the issues of transparency and evaluation. The principle of transparency asserts that interested parties outside an institution should have access to information pertaining to what decisions were made, who made them, what factors and options were considered, and the information on which decisions were taken. Evaluation allows the public to ascertain whether or not the decisions taken were the right ones, what impacts there have been, and whether or not the institution is meeting its mandate and how efficiently it is achieving that mandate. Evaluation also encourages the development of more effective approaches and operational instruments, through learning and feedback about the impact and effectiveness of ongoing and completed operations and programs.
The IMF argues that it is a government owned institution and therefore the individual governments, rather than the Fund itself, should be accountable to the public. The IMF contends that it only needs to be accountable to its member governments, and that its current internal process of evaluation is sufficient for achieving that degree of accountability. The reality, however, is that the IMF has considerable influence (sometimes dominance) regarding policy matters in many developing countries and its advice directly impacts on all strata of society for better or worse. Therefore, the IMF should be directly accountable to civil society in these countries and the public should have the right to petition the IMF directly.
There are many other reasons for external evaluation. The IMF is supported with tax payers' money, and the public has a right to know how effectively their money is being spent. Furthermore, the IMF itself, when it suits its purposes, advocates principles of evaluation: one of the IMF's main activities is its surveillance of governments' handling of their economies which is formalised through its annual Article IV consultations with member governments. It also sporadically supports transparency: the IMF insists that good and timely information is essential for improving the efficiency of markets and resource allocation. These arguments can and should equally be applied to global institutions like the IMF.
Despite the fact that the Fund advocates the benefits of external evaluation it prefers to keep its own operations secret. Indeed, in the market for a limited supply of aid resources it may suit the IMF not to evaluate itself publicly if it fears that evaluation conclusions will prove to be critical. Yet, such evaluation would benefit the public and the market for aid finance by helping to better direct aid resources to effective programs and effective institutions.
The IMF has taken few steps to openly evaluate its operations. Unlike the World Bank, it has no external evaluation department and it has no systematic procedure for evaluating the impacts of its surveillance operations, lending programmes and policy advice. This lack of systematic, objective evaluation has led to a stalemate between those who support the IMF's economic strategy and those who do not. With both those for and against able to present evidence showing success and failure of IMF programmes, but with no independent body with full access to the Fund's information and policy documents able to mediate between the two, there is little hope that IMF supporters and critics will find common ground to work together to improve the institution's functioning.
As the UK's Treasury Select Committee noted in its fourth report,
"The Committee was presented with such conflicting information [on the impact of structural adjustment programmes] that it has been impossible to disentangle the evidence sufficiently to arrive at any, even tentative conclusions."
The Committee concluded that,
"We do not believe that ad hoc external evaluations are sufficient to help improve the transparency of Fund operations and recommend that the Chancellor should instruct the UK Executive Director to continue to press the Fund's Board to establish an independent evaluation unit with a wide remit to cover in particular the impact of IMF structural adjustment programmes on the Sub-Saharan African economies."
Not only would independent evaluation serve to inform and therefore improve the work of the IMF, it would also inform the work of critics, academics and policy analysts outside the IMF, potentially leading to more public support for the Fund.
2a. Current evaluation structure
The IMF's Policy Development and Review Department (PDR) reviews programmes and Article Four surveillance operations to ensure that IMF policy advice is consistent throughout its operations. These are current and forward looking reviews. The PDR also carries out a Biannual Review of Surveillance. This is a stock taking, backward looking exercise which examines the major issues that have arisen out of the Article 4 consultation process. In particular, how advice has evolved, how country perspectives have changed or are changing, and whether or not there are weaknesses in its policy advice.
Periodically PDR has retrospectively reviewed the impact of the IMF's programmes and its operations. These reviews have included two reviews of ESAF (including the latest review which was published in September 1997); a review of the IMF's use of conditionality; an examination of the impact of adjustment on growth; and an evaluation of the Fund's technical assistance programme. The Board decides what should be reviewed and when, although the staff has the opportunity to put suggestions to the Board. All these reviews, except the technical assistance study, have been published as Occasional Papers, and although they are not automatically published (publication is a matter for the Board to decide), there is a presumption that such reviews will be published.
Internal reviews are useful but they are limited to the extent that they can be impartial. Staff are implicitly imbued with the IMF's culture and theoretical background, and have incentives not to be overly critical. Therefore they tend to be biased. Another major problem is that the scope of internal reviews is quite narrow and there is a danger that the evaluation process itself becomes censored by failing to address more fundamental issues. While understandably staff must focus on specific issues for the sake of time and depth, it is clear that the motivations for conducting reviews are different for those working within and those working outside the IMF. Thus issues of importance to civil society, such as social impacts of programs and the degree of local ownership of a program, are unlikely to be chosen as topics. This is partly because staff are more concerned with addressing problems within the framework they work within while critics tend to be more concerned with the actual theoretical underpinnings of the framework.
The recent internal evaluation is a case in point. Firstly it is questionable why a review of ESAF was undertaken so soon after the 1993 review. According to IMF staff, if the decision had been left to the PDR department it would not have conducted another review of ESAF for another couple of years. But it is clear that from the IMF management's perspective, a review could be useful for persuading donors that they should replenish ESAF once more. In fact the 1993 ESAF study was thought to have been very influential in encouraging donors to replenish the ESAF in 1994. Secondly, and more importantly, the review failed to address some of the important questions that critics were raising, such as ESAF's impact on the poor and small businesses.
Although most of the PDR department's reviews have been published, they tend to be regarded as the reflections of individual researchers within the Fund rather than necessarily expressing the views of the IMF's management or the Executive Directors, and it is unclear to what extent the IMF management follow through on the reviews' recommendations and conclusions.
The Audit department
The IMF's audit department, the Office of Internal Audit and Inspection, was expanded in 1996 and it now reviews the IMF's organisational and operational effectiveness as well as audits the Fund's accounts. The audit department identifies issues for review in consultation with the management and the Board. The reports are intended for the Board's use only to identify potential problems and areas for improvement in the Fund's operations. A recent study examined the function and work of the IMF's resident representatives and a forthcoming review will examine the Fund's experience with technical assistance.
Ad hoc evaluations
Very occasionally the Fund will commission evaluations carried out by external consultants, usually in response to some issue of apparent failure in the operations of the Fund. The most recent ad hoc evaluation examined the cause of the Mexico crisis, the IMF's failure to detect the crisis earlier, and what lessons were learned for averting widespread crisis. This report remained unpublished, and even the staff did not see it.
In response to NGO pressure, the Executive Board established a task force in 1992 to examine the IMF's evaluation procedures. Their report, which is not available to the public, recommended that the Fund should establish an evaluation unit, but this was not carried out. In 1995, once again after NGO pressure, the major shareholders of the IMF- the G7 governments- called for the Fund to establish an independent evaluation unit. The Fund's Board and management resisted this pressure but as a compromise the Board recently established an external evaluation procedure on a trial basis. The significance of this experiment is that it relies on external consultants to review the IMF's work which increases the opportunities for greater objectivity in assessing IMF policies and programs. The first external evaluation in this trial, a review of the implementation of ESAF programmes, was carried out in 1997 and was published in March 1998. The Board agreed to commission two or three evaluations a year, although in the first year only one review has been conducted.
A sub-committee of Executive Board members headed by the German Executive Director (ED), with the assistance of the Office of Internal Audit and Review, selects the issues, frames the terms of reference and appoints an independent panel of experts to review various aspects of the IMF's operations. The IMF staff are consulted about possible issues for review and there are opportunities for others outside of the Fund, such as civil society groups and academics, to propose possible areas for review through their ED's office. The review of ESAF is likely to be made publicly available but there is no automatic requirement that future reviews will be. This decision is the responsibility of the Executive Board. No formal process has been established for reviewing this procedure, but it is expected that it will be tested over a period of about two years. Topics being consider for future reviews include the impact of IMF surveillance advice on the policies of countries not using the IMF's financial assistance, the IMF's involvement in financial sector reform, and the status of reform in transition economies.
Whilst the establishment of this review procedure should be regarded as a welcome step in the right direction, the failure of the EDs to commission more reviews reflects the problem that there is no systematic review mechanism with a dedicated staff team. It is clear that the directors value the need for external evaluation but it must be questioned whether they have the time to ensure that this ad hoc mechanism works effectively. Given that the two year trial period is nearly over and only one review has been carried out the answer would seem to be no.
2b. Failure to systematically monitor programmes
The policies advocated in the Fund's programmes impact on all groups in society for better or for worse, but the Fund does not have any systematic way of examining this impact at the micro level. Nor does it have procedures to examine retrospectively whether its programmes have actually achieved their aims on a case by case basis once they have been completed or why programmes may have broken down. It is essential that this is rectified. The IMF claims that as an institution concerned with macroeconomic policy, it does not have the capacity to analyse effects at the micro level. However, the reality is that IMF macro policies have major implications for micro issues, and for the medium and long term economic viability of a country, and thus it should be aware of the full impacts of its programmes. As well as monitoring economic fundamentals, programmes should be monitored for their impacts on poverty and the distribution of assets, their social and economic impacts on different socioeconomic groups, and their impacts on the environment. Programme failures also need to be carefully investigated before a new programme is designed.
According to the staff, establishing such a procedure is not necessary because there is an implicit revue of the previous programme when a new programme is designed and during Article IV consultations. But this has not been the case. New programmes have tended to include policies that were not implemented in previous programmes as a matter of course, without examining why the policy was not implemented in the first instance. In the light of the latest ESAF review which found that many programme conditions were not implemented, PDR staff are now conducting more analysis of past programmes. These more explicit and candid assessments are to be included in country strategy papers and subsequently should feed into staff reports. While these efforts are welcome, it cannot be assured that they will be sustained without a formal procedure for doing so.
2c. Internal versus external evaluation
The proposal for an independent evaluation unit is not new, even within the IMF, but earlier proposals have been defeated by a deeply divided Board of Directors. There has also been a great deal of dissension within the IMF's management as to how useful an independent or quasi-independent evaluation unit would be. The IMF has three objections. Firstly, its argued that internal evaluations are more beneficial than commissioned evaluations from external experts, because staff learn by evaluating their work and are more likely to absorb recommendations and conclusions if they themselves have carried out the evaluations and have gone through the "learning process" and the "discipline of publishing". These are very real benefits, but given the limitations of the internal studies the ability to learn and apply useful recommendations based on them is diminished. It cannot be assumed that self-evaluation is sufficient, nor should self-evaluation be used to exclude outsiders from having effective input and assessment of a public institution.
The IMF also suggests that the Executive Board fears that creating an evaluation unit would grant it a monopoly on evaluation that would deter the staff from conducting their own evaluations. However, this need not be the case. The World Bank still carries out its own internal evaluations despite the existence of its semi-autonomous Operations Evaluation Department. For example it recently established the Quality Assurance Group, a staff-based evaluation team which monitors the quality of Bank projects, as part of its Portfolio Improvement Programme.
Finally, the IMF is also concerned about the relationship between evaluation activities and the Fund's Board of Directors. The Board is integrally involved in virtually all the Fund's decision making. Systematic evaluation would therefore implicitly assess the Board's performance and it is unlikely that the Board would invite scrutiny of its own operations. However, if the Board is properly scrutinising programmes then it should have nothing to fear from external evaluation. If on the other hand the Board simply tends to rubber stamp programmes then it is fair that they should be criticised and evaluation may help to ensure that programmes are properly scrutinised before they are agreed.
An external evaluation unit has many advantages over an internal evaluation procedure including:
The IMF's sister organisation, the World Bank, has responded to external criticisms by introducing a number of independent and semi-independent review mechanisms such as the Operations Evaluation Department (a semi-autonomous department which rates the development impact and performance of loans), and an Inspection Panel (a forum of independent experts to which people affected by Bank-funded projects can appeal if they believe they have been harmed due to Bank staff's failure to follow Bank practices). These mechanisms have helped improve the accountability of Bank staff and have boosted the image of the Bank amongst non-governmental organisations and their critics. The IMF would gain similarly from establishing such mechanisms.
An independent evaluation unit would be charged with:
The exact structure of an external evaluation unit could take one of many forms but it is important that it should embody the following principles:
3a. Options for new evaluation structures
The following options are outlined as possible structures for improved evaluation procedures at the IMF. They are not listed in order of desirability, nor are these the only feasible options. The intention is to offer suggestions, outline the advantages and disadvantages of each, and encourage debate.
3b. Financial Accountability for programme failure
Whilst the IMF plays an enormous role in formulating a country's economic strategy, it does not share the responsibility for programme failures, and governments must always repay the IMF in full. The establishment of an evaluation unit could pave the way for an inspection type mechanism similar to that developed by the World Bank, whereby a panel of experts could be called upon to determine whether or not a programme's failure is due to the government's shortcomings, external factors, or poor policy advice. Where the latter two factors have led to problems with programmes there should be some compensation process, whereby the government is not required to repay the full loan costs or some part of the loan is made repayable at more concessional terms.
3c. The IMF's disclosure policy for evaluation results
Currently, internal reviews are published at the discretion of the Board. The record on publication has been patchy, giving the impression that the Fund is trying to shield itself from public criticism. The IMF should establish a formal information disclosure policy with regard to its internal evaluations. All internal reviews should be made publicly available in full in all countries and in French, Spanish and English. In addition, Board and management should make public an official response to each review, and release Board plans to act on each review's recommendations.
Establishing more independent and effective evaluation procedures at the IMF would benefit both the IMF and civil society. The Fund would benefit from more effective operations, fewer programme failures, and greater public support for its role in today's global economy. This public support is no small matter, as the IMF has run into serious problems in attempting to secure more funding for its operations. The Fund's high degree of secrecy and its reluctance to open itself up to the scrutiny of independent evaluation, have resonated among parliamentarians as they contemplate whether to allocate scarce aid funds to the IMF. Civil society would clearly benefit as lessons are learned and new lending programmes are improved.
Despite the virtues of evaluation, it is likely that the process of improving the IMF's evaluation procedures will be painful. There are major questions of trust and motives that will surface between the Fund and civil society. Staff may resist losing control. Furthermore, no mechanism is without is shortcomings, and unrealistically high expectations may plague the initial products of the evaluation unit.
However, the risk of worthwhile. Too much stands to be gained not to improve the independence and depth of evaluation at the IMF. IMF member governments need to be fully supportive of genuine efforts to move in the direction of greater accountability by the IMF. Civil society and IMF Board, management, and staff need to shed their biases and come to the table to work together to establish a process of evaluation that is in the interests of civil society and the IMF.
This text may be freely used providing the source is credited.
Published: 14 June 2000 , last edited: 27 May 2010
Viewings since posted: 4415
Climate Investment Funds Monitor 7: April 2013 25 April 2013
Working paper: The private sector and climate change adaptation: International Finance Corporation investments under the Pilot Program for Climate Resilience 24 April 2013
The UK's role in the World Bank and IMF: Department for International Development and HM Treasury 13 March 2013
The World Bank and industrial policy: Hands off or hands on? 6 December 2012
Climate Investment Funds Monitor 6: October 2012 26 October 2012