We are very pleased to see the IEO continue its excellent work reaching out to stakeholders, external parties and shareholders, while both considering its work programme and conducting evaluations. This is a vital component of the IEO’s approach that ensures well-informed and well-considered analysis.

In relation to our assessment of the possible topics for evaluation in the medium-term, we wanted to comment on the currently proposed topics and express some priorities. To facilitate clarity, we have utilised the categories set out in the IEO’s publication of the possible topics for evaluation in the medium term, generally commenting on what we think are the most promising topics within each category.

Category I: Crisis management

Topic 3. Euro Area Crisis

The multiple and complex aspects of the IMF’s involvement in the euro area crisis clearly merit IEO attention, as is set out in the document of possible topics for evaluation in the medium term. We urge the IEO to put this topic on its research agenda. However, it is difficult to see the value of conducting part (i) Antecedents and Management of Initial Phase. Given the changed nature and in some respects unprecedented role of the IMF, in addition to the fundamentally changed institutional and regulatory environment of the euro area, few lessons applicable in future are likely to be learned from how the IMF interacted up until May 2010 with a euro area that is now hardly recognisable, given recent and on-going reforms.

Though the IEO terms of reference preclude immediate IEO assessment of part (ii) IMF Strategy and IMF-supported programs, we believe that the stated intentions for this second phase are very important. However we suggest you need to consider a number of additional aspects of the IMF’s involvement in the euro area crisis. Recognition of the sheer scale of the Fund’s commitment (over three-quarters of credit outstanding in Europe) necessitates a full examination of the Fund’s role and implications, not just for its coordination function and programme design and implementation. It also requires evaluation of the Fund’s capacity to provide appropriate and objective advice and on the Fund’s legitimacy, given the complex interaction with EU institutions and European states that continue to have a disproportionate representation in the Fund’s governance structures.

Concerns have been raised by former staff as to the appropriateness of IMF lending to advanced economies in Europe, and doing so via a Troika structure in which it plays an ostensibly junior role. Any evaluation should consider this and whether it has had implications for the Fund’s ability to provide appropriately neutral advice that takes into account impacts of euro area policies on non-euro area states, and whether the Fund is disempowered – due to the Troika’s structure – from taking stances that differ with the views and prerogatives of the other two Troika members or major euro area states such as Germany.

The evaluation should consider whether and to what extent claims of the suppression of staff opinion in regard to the IMF’s endorsement of the Troika’s expansion of conditionality are true. In addition, to what extent does the conditionality associated with this overwhelming portion of current IMF lending contradict the commitments to ensure that conditionality remains narrowly focused on core macroeconomic and fiscal issues, and even contradict the conclusions of the IMF’s own recent conditionality lending review which excluded from consideration euro area lending, despite its overwhelming predominance? Particular concerns have been raised about the Fund returning to conditionality in areas like state-owned enterprise privatisation and labour market reform.

Finally, given the considerable public hostility and politically controversial nature of Troika-initiated reforms, to what extent has the Fund contributed to assessment of distributional and socio-economic impacts and sustainability of policy reform where programme conditions have been applied?

Category II: Surveillance and program design

Topic 4. Bilateral Surveillance

Given the very recent adoption of the integrated surveillance decision (ISD), it is questionable as to the value of immediately conducting an evaluation of bilateral surveillance. Though this is a timely and highly relevant issue, we feel it is currently too early to conduct such an evaluation as there are insufficient cases and evidence that would have occurred under the new ISD framework.

Topic 5. Macro-financial linkages and integration with IMF surveillance

The aspiration to assess the IMF’s progress in integrating financial sector analysis into both bilateral and multilateral surveillance is welcome, in particular as it is explicitly linked to the major gaps in data and understanding of financial vulnerabilities and their systemic repercussions. This aspect needs to be linked to an evaluation of policy advice and conditionality to individual members. Specifically, advice related to financial sector reform should be evaluated for systematic bias in the shape of a commitment to fostering financial sector deepening, and whether this advice is consistent with the surveillance obligations of the Fund. In short, why does the IMF choose to enforce a financial deepening agenda, and how does it assess whether this is beneficial in light of evidence that financial deepening has not contributed to fostering growth and may have increased financial vulnerabilities of member states and exacerbated their systemic repercussions?

Topic 7. Evolution of Programme Design

As mentioned in relation to topic 3, an evaluation of the nature and scope of the macroeconomic and structural conditionality utilised by the Fund should include an independent assessment of whether conditionality has become more narrow and focused, as per Fund commitments, in particular over the last four years and including programs initiated as part of the Troika. Evidence of Troika programmes’ involvement in reforms of questionable value in terms of growth or immediate fiscal benefit, such as labour market flexibilisation and utility sector reform and/or privatisation, should also be considered.

Moreover, the necessity of ensuring that support designed to address short-term problems does not transform (inadvertently or otherwise) into long-term structural development lending should be taken into account, in particular in light of a number of low-income countries (LICs) that have been borrowing from the Fund for over a decade, such as Burkina Faso, Malawi and Sierra Leone despite in some cases participation in debt cancellation processes.

In light of reforms to facilities provided to LICs that have explicit intentions to consider and accommodate pro-poor outcomes, the evaluation of programme design evolution should consider the recent performance of these facilities and the evidence that there is an over-reliance on extended credit facilities, with longer-term horizons and a greater degree of structural intervention and reforms, as opposed to short-term facilities such as the rapid and standby credit facilities. In particular, an evaluation should consider how well the mix of these facilities’ usage has been suited to LIC debt sustainability challenges. Consideration should be given to the extent to which these problems have been externally driven during the period of the global financial crisis, rather than due to internal structural impediments, as indeed indicated by the managing director (in her Global Policy Agenda statement) when describing an economic context for LICs currently characterised by “weakening external demand” and “adverse terms of trade shocks.”

Category III: Forecasting and data management

Topic 8. IMF Forecasts: Process, Quality, and Country Perspectives

Within this category, this topic has the most pressing need for evaluation. Assessment of IMF forecasts needs to consider whether the Fund’s forecasting processes are free of ideological, political or institutional biases.

Several emerging market nations have challenged the Fund as to its ‘systemic bias’ in the generation of forecasts. Argentina and Brazil have sustained that bias enters into the generation of IMF projections by the selection of which types of data are considered relevant and thus subsequently used to influence programme design, policy advice and surveillance decisions. This problem lends itself to rigorous evidence-based examination.

An IEO evaluation would represent an opportunity to systematically examine the performance of IMF forecasts by contrasting ex ante forecasts with ex post outcomes. Key questions should consider whether there is a systematic bias in a positive or negative direction in terms of forecast impacts of IMF-approved policy advice and programme-related design.

This issue has been brought into even greater prominence by the 2012 World Economic Outlook (WEO) in which the Fund accepted that its performance in assessing the negative growth impact of fiscal austerity (specifically estimation and forecasts of fiscal multipliers) had systematically under-estimated the growth-reducing impact of fiscal consolidation policies. This should be included in any evaluation of forecasting and data management. This evaluation should seek to incorporate whether – as occurred with repeated editions of WEO forecasts – there is a persistent pattern of downgrades or upgrades to past forecasts.

The methodology behind forecasts would also need to evaluated, in particular to assess whether the reliance on cross-country statistical methods – extensively questioned by academic literature – contributed to the poor record of recent IMF forecasting and contributed to overly-rosy expectations for Fund-sanctioned reforms and policy. The socio-economic consequences of overly-optimistic forecasts for fiscal consolidation results should also be assessed to consider distributional impacts (including upon vulnerable groups such as children and for the existence of disproportionate impacts upon women via more than proportionate reliance upon social services). Finally, is the IEO’s stated intention to assess the usefulness of IMF forecasts’ value to the private sector necessary and relevant at this stage? In light of potential biases in these forecasts, this seems a second order question.

Category IV: IMF advice on specific issues and circumstances

Topic 10. IMF advice on monetary policy in LICs and emerging market economies

The desire to assess whether IMF advice on monetary policy is appropriately calibrated to take into account institutional and economic structures is welcome and should be the top priority in this category. One aspect that should be explicitly considered is the role the Fund has played and continues to play in advocating a reliance on inflation targeting policy by necessarily independent central banks in developing countries.

The record of inflation targeting in industrial and developing countries alike has been questionable, and this provides a basis for a rigorous data-led assessment of its validity. There is increased scepticism amongst the key intellectual architects of inflation targeting policy’s adoption, including chairman of the Federal Reserve Ben Bernanke and ex-UK monetary policy committee member Adam Posen, who have demonstrated in policy decisions and publications an increasingly limited adherence to inflation targeting’s key strictures. The IEO should assess the extent to which inflation targeting has been advocated by the Fund and examine the performance of countries, such as Thailand where controversy over inflation targeting has recently erupted, that have adopted these approaches with Fund approval but without complete success.

As such the socio-economic dimensions of inflation targeting and other monetary policy advice should be considered by asking how objective levels and targets, such as for inflation levels, are determined and with what policy trade-offs in mind. A particular dimension to assess therefore is the relationship of higher interest rates with higher growth or inflation levels, and whether countries have succeeded in sustaining growth despite persistent levels of inflation that would be deemed unsustainably high in orthodox inflation targeting policy regimes. The social and distributional impacts of monetary policy, in particular hawkish anti-inflationary stances, should also be assessed, in particular on suppressing employment and exacerbation of long-term unemployment levels in IMF programme countries.

Category V: Partnerships and governance

Topic 13. IMF collaboration with external partners

A welcome topic of consideration for the IEO would be to examine cooperation by the Fund with institutions that can offer a greater degree of expertise and specialism in areas where the Fund has an increasing recognition of its role and/or impact but lay outside its core expertise. As such this would entail consideration of how the Fund works with, for example, the ILO (International Labour Organisation) to consider labour market reforms and processes, or indeed with the World Bank to assess socio-economic impacts on developmental outcomes and poverty would be welcome, including through PSIAs (poverty and social impact analyses). The identification of the need to consider IMF-World Bank collaboration should necessarily assess the Malan Report and the extent to which its recommendations were adopted, and whether the reasons for rejecting certain recommendation were made sufficiently explicit and justified.

In addition, a systematic analysis of the Fund’s role as a junior partner in the Troika should be undertaken in such an evaluation. As mentioned above, an evaluation could assess whether the reduced capacity to speak independently and necessarily having to retrospectively endorse all Troika decisions has been consistent with the Fund’s mandate and obligations.

Topic 14. Internal governance: structures, processes, and incentives

An examination of internal governance as set out by the IEO would be welcome and is rightly recognised as overdue; we particularly endorse the concern about the tendency to work in silos and the impact this can have on reducing Fund effectiveness.

In terms of the factors diminishing the Fund’s capacity to enact effective collaboration and cross-fertilise ideas, we would invite the IEO to assess the extent to which the Fund invites and is able to meaningfully listen to external stakeholder voices, including that of civil society organisations. Even in comparison with the World Bank, IMF engagement processes are limited and opportunities to impact policy prior to decisions being taken are scarce. Thus the IEO should examine the mechanisms and opportunities for civil society consultation and engagement, both at the Fund level and in member nations where policy consultation is often skewed towards certain external stakeholders, such as the finance sector, despite policies’ impact often encompassing all of society and policies’ effectiveness being enhanced by broad engagement and social buy-in.

An additional dimension of this topic is the extent to which the Fund genuinely has extensive internal expertise on all of the areas of policy intervention in which it operates. As such, a highly pertinent case study of whether internal expertise is broadly-based, aware of contradictory evidence and the broader literature, and whether this knowledge is sufficiently brought to bear upon decision-making, is the recent process  to develop an institutional view on capital account management. The subject of capital controls has long been one where some members, particularly from developing countries, have accused the Fund of operating under a ‘liberalisation bias’, pressuring states to liberalise capital account management in a hasty manner that is insensitive to economic context. Work from the research department has been significantly different from that emanating from MCMD and the area departments. The difficulty in reconciling the different views has led to perceptions of a lack of even-handedness and rigour in Fund advice. This contributes to undermining the legitimacy of the organisation, inhibiting the Fund’s capacity to conduct its multilateral surveillance role and provide timely warnings that will be listened to and acted upon.

Overarching priorities

Overall clear priority should be given to evaluating the IMF’s role in the euro area crisis, with due consideration of course to the timing of conducting an evaluation so that it falls within the mandate of the IEO and does not interfere with ongoing programmes. Second priority should go to evaluations on forecasting (Topic 8) and macro-financial linkages (Topic 5). Our third priority would be one of the evaluations on governance, with a slight preference for working on external collaboration (Topic 13).