Policing the Policemen – the case for an independent evaluation mechanism for the IMF

briefing by:

Angela Wood , Bretton Woods Project

Carol Welch, Friends of the Earth US

April 1998

Acknowledgements

Thank you to Marijke Torfs, Friends of the Earth US, and Matthew Lockwood, Christian Aid, for their helpful comments on earlier drafts of this briefing.

We would also like to thank Yassine Fall for her analysis of the impacts of the ESAF programme in Senegal, and Ross Hammond and the Evangelical Lutheran Church in Tanzania for the assessment of Tanzania’s ESAF programme.

This briefing paper was produced with support from the C. S. Mott Foundation.

Contents

1. Introduction
2. The case for external evaluation
   • 2a.Current evaluation structure
   • 2b.Failure to systematically monitor programmes
   • 2c.Internal versus external evaluation
3. An external evaluation unit: principles and possibilities
   • 3a.Options for new evaluation structures
   • 3b.Financial accountability for programme failure
   • 3c.The IMF‘s disclosure policy for evaluation results
4. Conclusion
5. Appendix 1: The internal ESAF review – a critical discussion of its findings
6. Appendix 2: The need for comprehensive assessment of IMF programmes

Introduction

Non-government organizations (NGOs) have been calling for reform of the International Monetary Fund (IMF) since the 1980s, and systematic campaigns on the institution have been ongoing for about a decade. One critical component of these campaigns has been the push for an independent process of evaluation to assess the impacts of IMF programs and policies.

As a public institution, the IMF is accountable to the people who Fund it – taxpayers from around the world. However, accountability by the IMF has been severely hampered by the lack of information disclosure and public participation, and by the IMF‘s failure to develop an adequate evaluation process.

Establishing an effective evaluation unit at the IMF is an imperative if reformers wish to make the IMF an institution that makes the best use of its money by benefiting the people most in need- the poorest sections of society- and by promoting long term development that incorporates social and environmental sustainability into the economic equation.

In late 1996, the IMF established a trial external evaluation process. So far it has only conducted one review which examined the effectiveness of the Enhanced Structural Adjustment Facility. The findings of this review were published in March this year. This mechanism was established in response to calls for an independent review mechanism. While this is an important step, and NGOs commend the IMF for taking it, it is only the first step towards a more autonomous evaluation procedure.

The precise nature of an independent evaluation body at the IMF is a negotiable matter. NGOs would like to work with the IMF to develop a process that is in the interest of both the Fund and civil society. However, there are certain principles, such as transparency and public participation, that are essential for achieving a valid process. These principles and possible structures for an evaluation unit are discussed in section three. Section two examines why an evaluation unit is necessary and why current evaluation mechanisms employed by the IMF are inadequate.

The case for external evaluation

The IMF asserts that the quality of the development process corresponds in part to the quality of governance and it is increasingly including good governance criteria in its reform programmes. But whilst it advocates good governance for governments it has not applied the philosophy to its own activities.

One of the central requirements of good governance is accountability to the public. The principle of accountability asserts that decision makers should be held responsible for the consequences of their decisions. Therefore, accountability encompasses the issues of transparency and evaluation. The principle of transparency asserts that interested parties outside an institution should have access to information pertaining to what decisions were made, who made them, what factors and options were considered, and the information on which decisions were taken. Evaluation allows the public to ascertain whether or not the decisions taken were the right ones, what impacts there have been, and whether or not the institution is meeting its mandate and how efficiently it is achieving that mandate. Evaluation also encourages the development of more effective approaches and operational instruments, through learning and feedback about the impact and effectiveness of ongoing and completed operations and programs.

The IMF argues that it is a government owned institution and therefore the individual governments, rather than the Fund itself, should be accountable to the public. The IMF contends that it only needs to be accountable to its member governments, and that its current internal process of evaluation is sufficient for achieving that degree of accountability. The reality, however, is that the IMF has considerable influence (sometimes dominance) regarding policy matters in many developing countries and its advice directly impacts on all strata of society for better or worse. Therefore, the IMF should be directly accountable to civil society in these countries and the public should have the right to petition the IMF directly.

There are many other reasons for external evaluation. The IMF is supported with tax payers’ money, and the public has a right to know how effectively their money is being spent. Furthermore, the IMF itself, when it suits its purposes, advocates principles of evaluation: one of the IMF‘s main activities is its surveillance of governments’ handling of their economies which is formalised through its annual Article IV consultations with member governments. It also sporadically supports transparency: the IMF insists that good and timely information is essential for improving the efficiency of markets and resource allocation. These arguments can and should equally be applied to global institutions like the IMF.

Despite the fact that the Fund advocates the benefits of external evaluation it prefers to keep its own operations secret. Indeed, in the market for a limited supply of aid resources it may suit the IMF not to evaluate itself publicly if it fears that evaluation conclusions will prove to be critical. Yet, such evaluation would benefit the public and the market for aid finance by helping to better direct aid resources to effective programs and effective institutions.

The IMF has taken few steps to openly evaluate its operations. Unlike the World Bank, it has no external evaluation department and it has no systematic procedure for evaluating the impacts of its surveillance operations, lending programmes and policy advice. This lack of systematic, objective evaluation has led to a stalemate between those who support the IMF‘s economic strategy and those who do not. With both those for and against able to present evidence showing success and failure of IMF programmes, but with no independent body with full access to the Fund’s information and policy documents able to mediate between the two, there is little hope that IMF supporters and critics will find common ground to work together to improve the institution’s functioning.

As the UK‘s Treasury Select Committee noted in its fourth report,

“The Committee was presented with such conflicting information [on the impact of structural adjustment programmes] that it has been impossible to disentangle the evidence sufficiently to arrive at any, even tentative conclusions.”

The Committee concluded that,

“We do not believe that ad hoc external evaluations are sufficient to help improve the transparency of Fund operations and recommend that the Chancellor should instruct the UK Executive Director to continue to press the Fund’s Board to establish an independent evaluation unit with a wide remit to cover in particular the impact of IMF structural adjustment programmes on the Sub-Saharan African economies.”

Not only would independent evaluation serve to inform and therefore improve the work of the IMF, it would also inform the work of critics, academics and policy analysts outside the IMF, potentially leading to more public support for the Fund.

2a. Current evaluation structure

Staff Reviews

The IMF‘s Policy Development and Review Department (PDR) reviews programmes and Article Four surveillance operations to ensure that IMF policy advice is consistent throughout its operations. These are current and forward looking reviews. The PDR also carries out a Biannual Review of Surveillance. This is a stock taking, backward looking exercise which examines the major issues that have arisen out of the Article 4 consultation process. In particular, how advice has evolved, how country perspectives have changed or are changing, and whether or not there are weaknesses in its policy advice.

Periodically PDR has retrospectively reviewed the impact of the IMF‘s programmes and its operations. These reviews have included two reviews of ESAF (including the latest review which was published in September 1997); a review of the IMF‘s use of conditionality; an examination of the impact of adjustment on growth; and an evaluation of the Fund’s technical assistance programme. The Board decides what should be reviewed and when, although the staff has the opportunity to put suggestions to the Board. All these reviews, except the technical assistance study, have been published as Occasional Papers, and although they are not automatically published (publication is a matter for the Board to decide), there is a presumption that such reviews will be published.

Internal reviews are useful but they are limited to the extent that they can be impartial. Staff are implicitly imbued with the IMF‘s culture and theoretical background, and have incentives not to be overly critical. Therefore they tend to be biased. Another major problem is that the scope of internal reviews is quite narrow and there is a danger that the evaluation process itself becomes censored by failing to address more fundamental issues. While understandably staff must focus on specific issues for the sake of time and depth, it is clear that the motivations for conducting reviews are different for those working within and those working outside the IMF. Thus issues of importance to civil society, such as social impacts of programs and the degree of local ownership of a program, are unlikely to be chosen as topics. This is partly because staff are more concerned with addressing problems within the framework they work within while critics tend to be more concerned with the actual theoretical underpinnings of the framework.

The recent internal evaluation is a case in point. Firstly it is questionable why a review of ESAF was undertaken so soon after the 1993 review. According to IMF staff, if the decision had been left to the PDR department it would not have conducted another review of ESAF for another couple of years. But it is clear that from the IMF management’s perspective, a review could be useful for persuading donors that they should replenish ESAF once more. In fact the 1993 ESAF study was thought to have been very influential in encouraging donors to replenish the ESAF in 1994. Secondly, and more importantly, the review failed to address some of the important questions that critics were raising, such as ESAF‘s impact on the poor and small businesses.

Although most of the PDR department’s reviews have been published, they tend to be regarded as the reflections of individual researchers within the Fund rather than necessarily expressing the views of the IMF‘s management or the Executive Directors, and it is unclear to what extent the IMF management follow through on the reviews’ recommendations and conclusions.

The Audit department

The IMF‘s audit department, the Office of Internal Audit and Inspection, was expanded in 1996 and it now reviews the IMF‘s organisational and operational effectiveness as well as audits the Fund’s accounts. The audit department identifies issues for review in consultation with the management and the Board. The reports are intended for the Board’s use only to identify potential problems and areas for improvement in the Fund’s operations. A recent study examined the function and work of the IMF‘s resident representatives and a forthcoming review will examine the Fund’s experience with technical assistance.

Ad hoc evaluations

Very occasionally the Fund will commission evaluations carried out by external consultants, usually in response to some issue of apparent failure in the operations of the Fund. The most recent ad hoc evaluation examined the cause of the Mexico crisis, the IMF‘s failure to detect the crisis earlier, and what lessons were learned for averting widespread crisis. This report remained unpublished, and even the staff did not see it.

External evaluations

In response to NGO pressure, the Executive Board established a task force in 1992 to examine the IMF‘s evaluation procedures. Their report, which is not available to the public, recommended that the Fund should establish an evaluation unit, but this was not carried out. In 1995, once again after NGO pressure, the major shareholders of the IMF– the G7 governments- called for the Fund to establish an independent evaluation unit. The Fund’s Board and management resisted this pressure but as a compromise the Board recently established an external evaluation procedure on a trial basis. The significance of this experiment is that it relies on external consultants to review the IMF‘s work which increases the opportunities for greater objectivity in assessing IMF policies and programs. The first external evaluation in this trial, a review of the implementation of ESAF programmes, was carried out in 1997 and was published in March 1998. The Board agreed to commission two or three evaluations a year, although in the first year only one review has been conducted.

A sub-committee of Executive Board members headed by the German Executive Director (ED), with the assistance of the Office of Internal Audit and Review, selects the issues, frames the terms of reference and appoints an independent panel of experts to review various aspects of the IMF‘s operations. The IMF staff are consulted about possible issues for review and there are opportunities for others outside of the Fund, such as civil society groups and academics, to propose possible areas for review through their ED‘s office. The review of ESAF is likely to be made publicly available but there is no automatic requirement that future reviews will be. This decision is the responsibility of the Executive Board. No formal process has been established for reviewing this procedure, but it is expected that it will be tested over a period of about two years. Topics being consider for future reviews include the impact of IMF surveillance advice on the policies of countries not using the IMF‘s financial assistance, the IMF‘s involvement in financial sector reform, and the status of reform in transition economies.

Whilst the establishment of this review procedure should be regarded as a welcome step in the right direction, the failure of the EDs to commission more reviews reflects the problem that there is no systematic review mechanism with a dedicated staff team. It is clear that the directors value the need for external evaluation but it must be questioned whether they have the time to ensure that this ad hoc mechanism works effectively. Given that the two year trial period is nearly over and only one review has been carried out the answer would seem to be no.

2b. Failure to systematically monitor programmes

The policies advocated in the Fund’s programmes impact on all groups in society for better or for worse, but the Fund does not have any systematic way of examining this impact at the micro level. Nor does it have procedures to examine retrospectively whether its programmes have actually achieved their aims on a case by case basis once they have been completed or why programmes may have broken down. It is essential that this is rectified. The IMF claims that as an institution concerned with macroeconomic policy, it does not have the capacity to analyse effects at the micro level. However, the reality is that IMF macro policies have major implications for micro issues, and for the medium and long term economic viability of a country, and thus it should be aware of the full impacts of its programmes. As well as monitoring economic fundamentals, programmes should be monitored for their impacts on poverty and the distribution of assets, their social and economic impacts on different socioeconomic groups, and their impacts on the environment. Programme failures also need to be carefully investigated before a new programme is designed.

According to the staff, establishing such a procedure is not necessary because there is an implicit revue of the previous programme when a new programme is designed and during Article IV consultations. But this has not been the case. New programmes have tended to include policies that were not implemented in previous programmes as a matter of course, without examining why the policy was not implemented in the first instance. In the light of the latest ESAF review which found that many programme conditions were not implemented, PDR staff are now conducting more analysis of past programmes. These more explicit and candid assessments are to be included in country strategy papers and subsequently should feed into staff reports. While these efforts are welcome, it cannot be assured that they will be sustained without a formal procedure for doing so.

2c. Internal versus external evaluation

The proposal for an independent evaluation unit is not new, even within the IMF, but earlier proposals have been defeated by a deeply divided Board of Directors. There has also been a great deal of dissension within the IMF‘s management as to how useful an independent or quasi-independent evaluation unit would be. The IMF has three objections. Firstly, its argued that internal evaluations are more beneficial than commissioned evaluations from external experts, because staff learn by evaluating their work and are more likely to absorb recommendations and conclusions if they themselves have carried out the evaluations and have gone through the “learning process” and the “discipline of publishing”. These are very real benefits, but given the limitations of the internal studies the ability to learn and apply useful recommendations based on them is diminished. It cannot be assumed that self-evaluation is sufficient, nor should self-evaluation be used to exclude outsiders from having effective input and assessment of a public institution.

The IMF also suggests that the Executive Board fears that creating an evaluation unit would grant it a monopoly on evaluation that would deter the staff from conducting their own evaluations. However, this need not be the case. The World Bank still carries out its own internal evaluations despite the existence of its semi-autonomous Operations Evaluation Department. For example it recently established the Quality Assurance Group, a staff-based evaluation team which monitors the quality of Bank projects, as part of its Portfolio Improvement Programme.

Finally, the IMF is also concerned about the relationship between evaluation activities and the Fund’s Board of Directors. The Board is integrally involved in virtually all the Fund’s decision making. Systematic evaluation would therefore implicitly assess the Board’s performance and it is unlikely that the Board would invite scrutiny of its own operations. However, if the Board is properly scrutinising programmes then it should have nothing to fear from external evaluation. If on the other hand the Board simply tends to rubber stamp programmes then it is fair that they should be criticised and evaluation may help to ensure that programmes are properly scrutinised before they are agreed.

An external evaluation unit has many advantages over an internal evaluation procedure including:

• less bias – an autonomous evaluation unit staffed with experts drawn from outside the IMF would operate with different incentives which are not tied into the IMF as an institution;
• ability to address the right questions – because external reviewers are not embroiled in the day to day workings of the institution they have a clearer picture of its strengths and weaknesses and of the view points of those outside the Fund. Thus external evaluators are more likely to address the most pressing questions;
• skilled evaluators – an external evaluation unit would be able to employ professional evaluators from many different disciplines, whereas the Fund’s operations staff are economists;
• correct techniques – at present the techniques used by the IMF‘s staff to evaluate their programmes have proven to be inadequate or poorly applied and mis-specified. Professional evaluators would be armed with the most current evaluation techniques which would allow better, sharper analysis;
• consistent approach – a dedicated staff unit is more likely to maintain a high standard of review and to apply appropriate evaluation techniques more consistently;
• continuous surveillance function – currently the IMF staff do not have sufficient capacity to monitor and evaluate individual programmes on a continuous basis. Staff are confined to conducting large backward looking studies every few years in order to ascertain the impact of programmes generally. A dedicated evaluation unit would be able to monitor programmes on a more systematic basis which would allow problems to be addressed more quickly;
• monitoring micro-level impacts – although the Fund’s programmes are intended to alleviate poverty, its staff do not have the expertise to evaluate what impact its programmes actually have at the micro level.

The IMF‘s sister organisation, the World Bank, has responded to external criticisms by introducing a number of independent and semi-independent review mechanisms such as the Operations Evaluation Department (a semi-autonomous department which rates the development impact and performance of loans), and an Inspection Panel (a forum of independent experts to which people affected by Bank-funded projects can appeal if they believe they have been harmed due to Bank staff’s failure to follow Bank practices). These mechanisms have helped improve the accountability of Bank staff and have boosted the image of the Bank amongst non-governmental organisations and their critics. The IMF would gain similarly from establishing such mechanisms.

3.An External Evaluation Unit: Principles And Possibilities

An independent evaluation unit would be charged with:

• Providing the Executive Directors, IMF staff, and the public with sound analysis and unbiased assessments of how programmes have performed, identifying problems and recommending more appropriate or finely tuned policies where necessary. Principally, the Fund is concerned with achieving economic stabilisation goals – low inflation, stable balance of payments and low government budget deficit – and, in the case of ESAF borrowers, poverty reduction. Therefore, an evaluation procedure must examine how well programmes have achieved these aims. Monitoring the impacts on poverty reduction is a particularly pressing need and one that the IMF staff is unable to carry out under the present system of review. Other social and environmental impacts also need to be examined.
• Undertaking assessments of the efficiency and effectiveness of the overall institution and its mechanisms for achieving its aims.
• In the case of a dispute between a borrower and the Fund, over loan conditions for example, the evaluation unit could also be used as an arbitration service, where it would conduct its own investigation into the issues surrounding the disagreement and offer an independent source of analysis for settling disputes.

The exact structure of an external evaluation unit could take one of many forms but it is important that it should embody the following principles:

• Transparency:

  People should be told what aspects of policy or practice are being reviewed, what are the terms of reference, and what the process is for conducting the evaluation (for example which groups will be consulted).

• Broad inclusion:

  An effective evaluation process must survey and reflect the experiences of a broad cross section of people who are affected in different ways by the policies and programs being examined, including representatives of small and large farmers, peasant associations, women’s networks, social service organizations, labour unions, business associations, and environmental groups. Focus group discussion techniques and other participatory tools can be used to obtain the views of stakeholders. Evaluation results should also be discussed with stakeholders with the opportunity for stakeholders to respond to the evaluation’s findings.

• Participation:

  Participation goes beyond consultation and the analysis of a wide range of experiences. Participation also means that affected people have a role in determining the terms of reference and recommending expert evaluators. Mechanisms for soliciting this participation could include notices in local newspapers, town meetings in regional capitals, and notices disseminated from the Resident Representatives’ offices. These solicitations could invite submissions on names of evaluators, and suggestions and/or comments on terms of reference. This process should be initiated with sufficient time to allow for an adequate response.

• Dissemination and access to reports and information:

  The evaluation unit’s reports and the process for operationalising its recommendations should be made public. A process should be established to allow civil society representatives, comprised of a diverse set of leading groups, to comment on draft reports and these comments should be properly attributed and included in the final document. The civil society representatives could be country specific where appropriate. The IMF‘s staff, where relevant, should also be allowed to see and comment on the report in advance of publication, and their comments should be included in the final report.

  Reports should be published in full and should not be edited by IMF staff or Board members prior to publication. All evaluation reports should be made available free of charge in all Fund member countries and should be published in French and Spanish as well as English.

• Skills mix:

  The unit should be staffed by evaluation specialists from both economic and non-economic disciplines, including poverty specialists, anthropologists, geographers, poverty experts, gender analysts, and environmental professionals. A major problem with the IMF‘s current processes of evaluation is that the relatively narrow expertise of the macro economists involved prevents a more thorough, encompassing approach. Evaluation staff must be equally able to analyse data drawn from both micro and macro sources.

• Staffing:

  Current staffing procedures in the Multilateral Development Banks’ evaluation units vary, but generally evaluation unit staff are assigned from within the MDB itself and are likely to be reassigned to other units within it. While this may be beneficial for staff development, it brings into question the ability of staff to look critically at the work of the institution if they fear that this will affect where and when they are relocated to other bank divisions. The unit’s staff must feel free to challenge the roles, the accepted objectives and theoretical foundations of the IMF‘s work where appropriate.

  To ensure that staff feel free to be critical, senior evaluation unit personnel should not be allowed to work for the IMF after completing their post in the evaluation unit. Any junior staff in the unit should respect an appropriate time period, two years for example, before accepting a position within the IMF. Appointments should be made by a steering group comprising members of the Executive Board, academics, NGOs, trade unions, business associations and other civil society groups.

• Autonomy:

  The work of the independent unit should not be influenced by the IMF‘s management or staff. To ensure this, issues for discussion should be identified by a steering committee. The director of the unit should report directly to the committee and the Fund’s Board.

• Mechanisms to ensure recommendations are followed up:

  While autonomy is essential, it is also critical that the Unit’s results and recommendations are taken seriously by the IMF‘s staff so that they are respected and acted upon. Channels must be established to communicate evaluation results and recommendations to the appropriate staff members. This needs to be done in a manner that emphasises the need to build positively on the results of evaluations rather than to reprimand staff. Ultimately, the Executive Board must be responsible for ensuring that recommendations are followed up, and where there is a dispute over specific recommendations the final decision should rest with the Board.

• Financial resources:

  Clearly there are major budget implications associated with this type of proposal. The IMF has previously used a lack of staff and financial resources as an excuse for avoiding establishing an evaluation unit. The IMF should provide financial resources to help establish and maintain an evaluation unit but it should not necessarily be expected to contribute all the resources. Indeed, it may even be preferable to use bilateral donations to co-fund the unit to allow more autonomy from the IMF. It is likely that civil society organizations would mobilize behind providing such targeted funding.

3a. Options for new evaluation structures

The following options are outlined as possible structures for improved evaluation procedures at the IMF. They are not listed in order of desirability, nor are these the only feasible options. The intention is to offer suggestions, outline the advantages and disadvantages of each, and encourage debate.

• Option 1: self-evaluation supplemented by external reviews.

  One possibility is to continue with the present system: internal evaluations, supplemented by occasional external reviews. The Executive Board would thus remain largely in control of the evaluation process. However, the current external evaluation process would need to be redesigned and formalised to allow public participation in decisions regarding the choice of issues for review, the selection of experts, and in framing the terms of reference.

  To assist this process it might be useful to establish a civil society committee to work with and advise the Board (or the Board’s own evaluation committee). The committee could be comprised of relevant experts drawn from NGOs, academic institutions, and the private sector with representation from both the North and the South, and it could be country specific where appropriate. It could also comment on the final report and solicit comments from other relevant individuals and organisations. It might be desirable to formulate a new civil society committee for each review. This would help to ensure that the committee is comprised of the most suitable people. However, it could make the whole process rather drawn out and complicated, especially if there is little unanimity as to who should be appointed to each committee.

  As is currently the case, once chosen, the external evaluators should be free to establish their own work methodologies but should adhere to the principles of broad inclusion. The Board would also need to establish a formal process for publishing reports based on the principles of full and timely publication.

  Under the current experimental external review procedure, it was envisaged that two or three reviews a year would be carried out, but in the first year only one has been conducted. If this system is to work effectively, then staff may have to be redeployed from other activities to deal with the necessary administrative matters. However, it is questionable whether the IMF‘s directors are willing and able to commit sufficient time to the process to ensure that several reviews a year are carried out effectively.

  Another problem with individually commissioned reviews is that systematic monitoring of programmes would fall out of the reviewers’ purview and capabilities. Systematic monitoring of programmes, and monitoring whether review recommendations and results are implemented, would still need to be conducted by Fund staff. Not only is this time consuming to administer, particularly for staff who also have other programme and policy responsibilities, but internal monitoring by the IMF‘s staff is likely to be influenced by promotional and other incentives which would bring bias into the review process. These biases may be diminished if it there is a formal policy to publish both internal and external evaluations. But there remains the problem that the IMF‘s staff is too narrowly focused on macroeconomics. This could be rectified by employing new staff with more appropriate skills or through extensive retraining schemes.

• Option 2: in-house evaluation unit.

  The IMF could establish, using its own resources, a semi-autonomous evaluation unit similar to the World Bank’s Operations Evaluation Department (OED). The unit could report to the IMF‘s Board but it would be independent from the IMF management. As in option one, civil society involvement could be achieved by forming a committee of civil society representatives to oversee and advise the unit when informing and soliciting views from the public.

  To be consistent with the principles outlined above, it would probably be necessary for the Board to hire a new team of staff rather than redeploy staff internally, particularly if an appropriate skills mix is to be achieved. Whether or not this action would be enough to ensure that the unit is regarded as sufficiently autonomous from the IMF is difficult to judge. Critics might argue that the unit would be easily influenced by the IMF‘s management and staff. Such fears could be allayed by a formal policy of full disclosure of evaluation results (the OED does not publish all its evaluation results). Where sensitive information is involved, there could be grounds for releasing summary documents rather than the full text but there should be a presumption in favour of publishing information.

  It is possible that animosity could arise between the staffs of the IMF and the evaluation unit. Care must be taken to ensure that IMF staff do not feel that evaluation exercises are biassed against them. Whether or not this is in fact a problem will largely depend on how well the management and the Board deal with the communications issues, and what mechanisms are introduced to follow up on recommendations. The reverse may equally be true: staff may be willing to heed the advice of an evaluation department if there is sufficient managerial support for it, and the unit’s work is seen to be fair and authoritative.

  The expense of establishing an in-house unit is an obvious constraint, particularly the cost of hiring more personnel. However, objections on financial grounds are not strong: the unit may lead to longer-term savings for the Fund by ensuring programmes are more effective and it may lead to greater public support for the IMF.

• Option 3: independent evaluation unit.

  A third option is to establish a dedicated unit which is independent from the IMF staff and management. Given that the IMF and the World Bank work more and more closely together it might be a useful option to extend and delink the OED from the World Bank so that it also has responsibility for evaluating the IMF‘s activities.

  While this greater autonomy would have obvious benefits, it could make follow through of recommendations harder to achieve. This difficulty could be overcome by establishing a formal mechanism for ensuring that recommendations are followed through and by setting up a process by which management and/or staff are accountable to a disciplinary body should they repeatedly fail to act upon the advice of the unit.

  This approach has an added benefit. The World Bank and IMF are supposed to work together in many of their lending operations, particularly in the poorest countries where greater World Bank expertise in environmental and social matters is critical in creating an effective programme. Yet their collaboration is not readily apparent. This approach would increase the likelihood that the two institutions would work more closely together and therefore it would enhance their effectiveness.

• Option 4: evaluation committee.

  A fourth option is to have an evaluation committee comprised of external experts, and possibly IMF staff, who would each be appointed for a specific term. The committee would oversee each review, being responsible for determining the topics for review, selecting terms of reference, and choosing experts. The committee would be responsible for ensuring that this process is transparent and participatory.

  To ensure impartiality, the committee could be chaired by an outside expert who has not worked for the Fund. And to ensure that there are good links with the Fund’s management and staff it may also be desirable to include, for example, a retired senior level Fund staff member to serve as an IMF representative. This would help to lend greater legitimacy to the findings of the evaluation process, both within and outside the IMF. Moreover, a retired staff member would not be governed by the same incentives as current staff but should have credibility within the institution. This would be helpful when disseminating results and recommendations within the IMF. The committee could be responsible for appointing new members or there could be some voting procedure for electing new members with governments, the IMF board and civil society selecting representatives. To ensure continuity it may be necessary to appoint members for overlapping terms.

  Whilst the committee would be able to oversee occasional evaluations it would not necessarily be able to address the need for systematic, on-going evaluation of completed programmes. Either it would be necessary to dedicate a team of Fund staff to the job which would lead to the same incentive and skill problems outlined in option 1, or a dedicated unit would need to be established within the committee. The latter may be more favourable but this would add greatly to the work of the committee. Another problem might lie in maintaining continuity and lessons learned as committee members leave and new ones are installed.

3b. Financial Accountability for programme failure

Whilst the IMF plays an enormous role in formulating a country’s economic strategy, it does not share the responsibility for programme failures, and governments must always repay the IMF in full. The establishment of an evaluation unit could pave the way for an inspection type mechanism similar to that developed by the World Bank, whereby a panel of experts could be called upon to determine whether or not a programme’s failure is due to the government’s shortcomings, external factors, or poor policy advice. Where the latter two factors have led to problems with programmes there should be some compensation process, whereby the government is not required to repay the full loan costs or some part of the loan is made repayable at more concessional terms.

3c. The IMF‘s disclosure policy for evaluation results

Currently, internal reviews are published at the discretion of the Board. The record on publication has been patchy, giving the impression that the Fund is trying to shield itself from public criticism. The IMF should establish a formal information disclosure policy with regard to its internal evaluations. All internal reviews should be made publicly available in full in all countries and in French, Spanish and English. In addition, Board and management should make public an official response to each review, and release Board plans to act on each review’s recommendations.

4.Conclusion

Establishing more independent and effective evaluation procedures at the IMF would benefit both the IMF and civil society. The Fund would benefit from more effective operations, fewer programme failures, and greater public support for its role in today’s global economy. This public support is no small matter, as the IMF has run into serious problems in attempting to secure more funding for its operations. The Fund’s high degree of secrecy and its reluctance to open itself up to the scrutiny of independent evaluation, have resonated among parliamentarians as they contemplate whether to allocate scarce aid funds to the IMF. Civil society would clearly benefit as lessons are learned and new lending programmes are improved.

Despite the virtues of evaluation, it is likely that the process of improving the IMF‘s evaluation procedures will be painful. There are major questions of trust and motives that will surface between the Fund and civil society. Staff may resist losing control. Furthermore, no mechanism is without is shortcomings, and unrealistically high expectations may plague the initial products of the evaluation unit.

However, the risk of worthwhile. Too much stands to be gained not to improve the independence and depth of evaluation at the IMF. IMF member governments need to be fully supportive of genuine efforts to move in the direction of greater accountability by the IMF. Civil society and IMF Board, management, and staff need to shed their biases and come to the table to work together to establish a process of evaluation that is in the interests of civil society and the IMF.