An annual Independent Evaluation Group (IEG) report on the Results and performance of the World Bank Group 2012 showed declining effectiveness, with the Bank’s worst ratings in areas with the fastest increases in lending or extra priority, such as infrastructure and public-private partnerships (PPPs).
The mid December report from the IEG, the Bank’s arms-length evaluation body, covered projects closed before end June 2012 and looked across the entire World Bank Group. The report’s positive overview and conclusions, filled with encouraging words for the Bank, veil its findings that in many areas the Bank’s projects are performing worse than before with no improvement in sight given low global economic growth. The IEG counsels that “closer attention to the quality of preparation and supervision of investment operations” would help, noting that those are both “factor[s] under the institutions’ control”.
According to the IEG, development policy operations (DPOs), loans for policy reforms rather than specific projects, saw improved results: “DPOs were moderately satisfactory or better for 83 per cent of operations completed in FY09-11 that were evaluated by the end of FY12.” On the other hand, good ratings on investment loans fell from 78 per cent for 2006-08 to 70 per cent in 2009-11, though the IEG cautioned about incomplete data and the use of a projection to come up with the lower figure. The ratings are based on IEG reviews of the Bank’s internally created implementation completion reports.
Particularly problematic were investment loans for infrastructure
Particularly problematic were investment loans for infrastructure and to projects in the East Asia and Pacific region, both of which saw statistically significant drops in good ratings for 2009-11. The IEG found that: “Human development was the only area in which the share of projects rated moderately satisfactory or better in development outcome ratings improved.”
Mediocre results, inadequate consultation
While the IEG results show declines nearly across the board, there is also nuance within those figures: “Among the projects that exited the active portfolio in FY09-11, 26 per cent were rated either satisfactory or highly satisfactory, and 44 per cent were rated moderately satisfactory”, showing that even the positive ratings are driven predominantly by mediocre results.
In its analysis of why things were going badly in infrastructure, agriculture, and beyond, the IEG found “overambitious project design, inadequate consultation with stakeholders, insufficient candour during supervision, and failure to follow up on problems identified during supervision missions as reasons for less-than-satisfactory achievements.” World Bank president Jim Yong Kim is aiming to streamline project preparation and lending decisions (see Update 83).
In a more detailed look at the problems in infrastructure, which the IEG notes has seen a massive boost in lending volume, the report says: “A review of the evaluations of infrastructure operations reveals that challenges associated with projects in urban areas have adversely affected project performance. Seventeen of 32 infrastructure projects rated moderately unsatisfactory or worse were implemented in an urban setting. Among urban development projects, smaller projects focusing on issues such as policy development and capacity building tended to do less well. Moreover, most of the urban projects rated moderately unsatisfactory or worse are in middle-income countries. This may be because Bank programmes in these countries tend to be more comprehensive, encompassing sector policy and institutional capacity issues beyond providing access to basic urban infrastructure and services.”
Moreover, the IEG report also provides evidence of an increasing disconnect between the IEG’s assessment of development results and the self-reporting by Bank staff. For the seventh year in a row, the gap between these ratings has widened. In the most recent year Bank staff rated 83 per cent of their projects moderately satisfactory or better, while the IEG only rated 70 per cent of projects as such. Bank staff incentives are linked to project success outcomes.
Since mid 2012 the IEG has published a database of all its project ratings, for the first time allowing external stakeholders to check in which projects the IEG is finding gaps in Bank performance. For projects finishing after mid-2011 the full text of the IEG’s assessments are now available.
IFC’s “steep decline”
The IEG report also covered the International Finance Corporation (IFC, the Bank’s private sector lending arm), finding that investments with good development outcome ratings dropped from 73 per cent in 2006-08 to 68 per cent in 2009-11, but noted that this was not a statistically significant change. However, IFC investments in the poorest countries “have seen a steep decline in performance”, dropping from 73 per cent to 52 per cent. The IEG partly blamed the drop on “the higher risk of operations in those countries, issues related to IFC’s work quality, effects of the global financial crisis, and specific regional and industry factors”.
The assessment of the IFC’s work quality considered three indicators: (i) quality of screening, appraisal, and structuring; (ii) quality of supervision; and (iii) the IFC’s role and contribution. The IEG “found that IFC’s work quality, a factor within IFC’s control, is the most important variable in determining the likelihood of achieving positive development outcome ratings” in countries eligible for borrowing from the Bank’s low-income country arm, the International Development Association (IDA). It went on, “positive ratings for IFC’s work quality fell from 78 per cent to 59 per cent. This was driven by low ratings for IFC’s appraisal quality (59 per cent) and IFC’s role and contribution, where positive ratings fell from 83 per cent to 60 per cent between these two periods.”
The IEG identified some common shortcomings in the IFC’s work quality: “environmental and social categorisation and requirements; overestimation of management capacity, financial viability, or growth prospects of the company; support to projects with unsuitable sponsors; and lack of leverage over the client. Characteristics associated with projects that have a low rating for IFC role and contribution include weak justification for IFC financing or limited financing role (39 per cent), a passive or nonspecific role for IFC in the project (36 per cent), and weak commitment of clients in raising standards/limited leverage of IFC in influencing clients’ decisions (21 per cent).”
The IFC’s advisory services (see Update 62), which have been a concern for civil society groups working on privatisation and land grabs (see Update 81, 68), were rated especially poorly. Between 2008 and 2010, only 79 per cent of these technical assistance projects “were rated satisfactory or higher in achieving their immediate outputs”, implying that work was not completed or reported. Furthermore, only “59 per cent were rated satisfactory or higher in achieving their outcomes. … “, meaning reaching their objectives. “The share is lower still at 39 per cent in the achievement of longer-term impacts.” The IEG relegates a further recommendation to a footnote: “the IFC’s ultimate objective is to achieve impact – focusing its advisory services on poverty – the corporation should assess the implications of this issue.”
The IFC’s controversial promotion of PPPs through advisory services (see Update 78, 66, 64) received “the lowest development effectiveness ratings” of 46 per cent of the projects being rated at least mostly successful. This “calls for management attention, given that IFC expects to expand this line of business.”
The IFC’s massive increase in short-term finance at the expense of longer term lending was also brought out by the IEG. Short-term finance includes things like trade finance (see Update 76), but the IEG cautioned “the short-term finance operations have not yet undergone IEG evaluation (or systematic self-evaluation), leaving a gap in results measurement and evaluation for a large portion of current IFC operations.” The IEG plans to release an evaluation of short-term finance in the first half of 2013.
The IEG also warned about the increasing use of financial intermediaries (see Update 84, 79, 73, briefing), saying it is essential that the IFC “seek a deeper understanding of the development impact of financial sector projects and use such knowledge to calibrate their intervention strategies. This, in turn, will call for indicators that can measure effects beyond the financial intermediary itself and gauge development progress among its clientele.”
