By Angela Wood
30 August 2000
The Bretton Woods Project welcomes the establishment of an EVO. As the Executive Board recognises, this is an important step for ensuring more effective transparency and governance of the IMF and for enhancing the IMF‘s operations. However, Bretton Woods Project is concerned that current proposals could undermine the independence of the EVO and its effectiveness.
Firstly, Bretton Woods Project would like to highlight two missing but very significant elements of an effective evaluation unit. These are 1) mechanisms for involving external stakeholders in various aspects of the EVO‘s work; and 2) mechanisms for ensuring IMF management and staff implement accepted recommendations.
Involving civil society
It is vital that the EVO is regarded by those outside the IMF as an effective evaluation mechanism. An important aspect of effectiveness is its willingness to respond to the concerns of those outside the institution. Much criticism has been levelled at the IMF because it is perceived to be unable to respond to these concerns because it does not have proper procedures for doing so and because its vision is limited to established ways and means of conducting its business.
In order for the EVO to rectify this problem it will be necessary to establish mechanisms through which civil society and other external stakeholders can raise their concerns about specific operational procedures, policies and programmes and feed into the evaluation process.
What steps are necessary?
1. Requesting an evaluation
- a clear mechanism should be established for external stakeholders to suggest topics suitable for external evaluation;
- on receipt of such a request, the EVO should be required to investigate the case for an evaluation. On the basis of this, the Director of the EVO should have ultimate responsibility to decide whether or not to proceed with the evaluation. If there is not good cause, a public statement should be made explaining why;
- in its annual report, the EVO should detail all requests for evaluation from external stakeholders and what actions were subsequently taken and why.
2. Input into terms of reference and selection of consultants
- the draft terms of reference should be made publicly available (probably through the EVO‘s website) for a limited time period in order for external stakeholders to comment on them;
- these comments should be taken into consideration when the final terms of reference are drawn up. Where there is a significant difference between the final terms of reference and comments made by external stakeholders, the reasons for this should be stated in the terms of reference. The final terms of reference should be published;
- a mechanism should also be established to allow external stakeholders to suggests possible consultants to carry out evaluations;
- where possible evaluation teams comprised of external evaluators should seek to include persons from the NGO sector as well as academic and other private sector, national or international institutions.
Ensuring effective follow up
The EVO will only be effective if there are mechanisms in place to ensure that its recommendations are put into practice. Whilst the Board has ultimate responsibility for ensuring that this is so, it would be helpful to establish mechanisms to assist the Board. For example, experience with the external and internal evaluations of ESAF demonstrated that staff prioritised their own recommendations before those made in the external review and follow-up in some areas, now implicitly being pursued through the PRSP, is still weak.
What steps are necessary?
- on receipt of each evaluation report, the Executive Board (EB) should make a report (which will also take in account comments received from the management and staff) detailing which recommendations it accepts and what steps it will require the management and staff to undertake to fulfill these recommendations and a timetable for doing so. Where the EB rejects certain recommendations, the case for doing so must also be made clear. This report should be published;
- in accordance with the timetable laid out in the EB‘s response, the EVO should be tasked with monitoring implementation and should make regular reports to the EB. These reports should also be made public. A summary of actions taken at the IMF should also be included in the EVO‘s annual report.
Further comments on the background paper
Accountabilities
In order to guarantee the independence of the EVO it is essential that it has total control over its own work programme. Whilst the director should be concerned to consult with the EB, the EB should not be required to approve the work programme. This is for two reasons: 1) it ensures that publicly the EVO is seen to be at “arms length” from EB (and therefore management and staff) influence; and 2) it will allow the EVO to respond flexibly to issues as they arise.
There is a grave danger that giving the Board such control over the EVO‘s work programme will be viewed by those outside the IMF as a means of ensuring that the EVO does not undertake evaluations of sensitive issues, particularly since the PDR department is very influential with the Executive Board. Moreover, the EVO should not be required to consult with staff regarding its work programme. Staff and management should be allowed to submit comments on the programme but these should be made public.
Scope of activities and coordination with other work underway at the Fund
The current proposal sees no basis for the EVO to undertake evaluations of on-going operations, which should remain the task of internal evaluators. Whilst the Bretton Woods Project is sympathetic to the Board’s concern that daily operations should not be disrupted, this fear is over emphasised. Indeed, it should not be the objective of the EVO to harass staff but to assist them to carry out their work more effectively. Moreover, whilst repetition should be avoided, the current proposal to simply add on the EVO to existing mechanisms is inappropriate and runs the risk of being perceived externally as simply a failed public relations exercise.
Whilst the staff implicitly evaluate programmes as they are completed, the quality of this work is hard to determine (largely because such documents are still not made public). Nor is it clear to what extent staff learn from internal reviews. For example there have been instances where unfulfilled conditions have been inserted into new programmes time after time, apparently without any analysis of why they were not implemented in the first place. Also, there appears to be no systematic monitoring of whether programmes achieved their objectives (although PDR has carried out occasional ad hoc studies), and even less analysis of the social and environmental impacts of programmes. Moreover, the few internal reviews which have been published have often be regarded as biased and of poor quality by those outside and fail to address many of the key concerns of external stakeholders.
The paper identifies a key problem which is how to define whether a programme is “on-going” or “ex post”, ie completed. Clearly some aspects of programmes are taken forward into later programmes which makes this distinction almost meaningless. The danger is therefore, especially since many programmes also breakdown before completion, that if the EVO‘s remit was limited to reviewing “completed” programmes that effectively it would be prohibited from evaluating most programmes, ie the management will always have an argument for blocking evaluations that they do not want.
Where problems arise in programmes, for example there currently appears to be disagreement between the Mozambique government, the World Bank and the IMF about sugar tariffs, there is currently no external arbitrator to help resolve disputes. This is a potentially important role for the EVO.
Also, it appears to be rather defensive to argue that the EVO should “avoid pre-empting efforts by management to enhance the efficacy of the Fund’s work and manage the institution” and “Policies and procedures under active discussion in the Fund and current Fund programs would not therefore be appropriate areas for EVO Evaluation”. Firstly, it is precisely in these areas that the EVO could add value to the internal discussions. Secondly, from these statements, it appears that the EB, ie. the management, is worried that the EVO will try to score “victories” over the IMF management. Whilst this is possible, it is unlikely if managers are making the necessary reforms and the EB is positively facilitating the relationship between the two. This is a management issue for the EB and should be treated as such. It should not be used as an excuse to limit the function of the EVO. This negative attitude needs to be addressed early on. The EVO should be seen as a benefit to the IMF not a hindrance. The apparent emphasis on the latter suggests that the IMF‘s managers wants something to remain hidden.
What steps are necessary?
- the EVO should be given the remit to carry out systematic evaluations of on-going programmes, particularly to ensure that there are no unexpected adverse social impacts;
- evaluation reports of these activities should be made available to the public, where these contain market sensitive information this should be withheld until an appropriate time after which it should also be made public;
- all internal reviews should be made publicly available (including reviews of programmes).
Organisational structure
The Bretton Woods Project agrees with much of this section, however, it is totally inappropriate that the Managing Director should have any involvement in the selection of the EVO Director or other staff. To ensure independence, selection of the Director should rest with the EB and all other staffing matters should be the sole responsibility of the Director.
In addition:
- the Director should not have worked for the IMF in the previous 5 years before s/he is appointed;
- the Director should have a proven history of working on social matters;
- candidates for the Director should be made public along with their CVs;
- IMF staff should comprise less than half of the EVO‘s staff (not including the Director);
- there should be an equal balance of skills between social and economic specialists.
Handling of evaluation results and publication of reports
- in addition to staff, management and governments, civil society should also have the opportunity to make written responses to evaluation reports, which should be circulated along side those of the others. All such responses should be published;
- there should be automatic disclosure of all reports, it should not be left to the discretion of the EB. Where there is sensitive market information this should be removed and disclosed at a later date (no later than 18 months after the publication of the report);
- what constitutes “confidential information” should be defined in advance.
