An evaluation by a panel of self-described ‘academic superstars’ has cast doubt over the independence and reliability of World Bank research. However, in failing to address fundamental problems in the Bank’s role in development research, the report misses an opportunity to provide a clear signal for donors to shift support to developing country research institutions.
The evaluation covers Bank research – conducted in the development economics vice-presidency (DEC), regional and thematic departments, the World Bank Institute, and that by consultants – in the period 1998 to 2005. Chaired by Angus Deaton, professor of economics at Princeton University, the evaluation says that, overall, the Bank’s researchers “have done a creditable job of delivering on the many, potentially inconsistent, demands made of them”. “Much of what we read was of very high quality”, say the report’s authors.
Bank proselytized selected new work without appropriate caveats on its reliability
But this praise is scant recompense for what follows. The report finds that “there is a great deal of research that is undistinguished and not well-directed either to academic or policy concerns”. Amongst this ‘undistinguished’ work, the evaluators found research which is “technically flawed and in some cases strong policy positions have been supported by such (non) evidence”. In some cases, “the Bank proselytised selected new work in major policy speeches and publications, without appropriate caveats on its reliability”. The ensuing litany of critiques is withering:
- New research methods have been promoted “without adequate evaluation”;
- Many Bank researchers try to prove causality by tinkering with economic models; in the words of the evaluators, they “appear to think that the attribution of causality can be solved by technical means”;
- In some cases, “the degree of self-reference rises almost to the level of parody”; and
- There is “remarkably little work co-authored by non-Bank researchers from developing countries”.
The evaluators posit a number of reasons to explain the breakdowns. Most damning for Bank senior management is their finding that researchers are “under pressure from the Bank presidency and elsewhere not to say things that go directly against the broad policy line that the Bank is espousing.” Researchers are chosen by country teams who are looking for “a particular answer or a particular researcher who they know and like working with, or perhaps someone known for not rocking the boat.” The researchers themselves say it is not unusual to be told that “we should do an evaluation to prove that X programme works”.
This evidence is backed up by interviews with former Bank research heads who complain that “there was an enormous amount of interference by the public relations people, especially after Wolfensohn became president; research was not supposed to offend NGOs, nor to provide them with material they could use to criticise the Bank.” Former chief economist Joseph Stiglitz added that during the Asian financial crisis “the belief that certain policies always worked meant that the more relevant questions of when the policies worked were not addressed.”
Bank attempts to silence critics
Update 53 carried an abridged version of American University professor Robin Broad’s paper Research, knowledge and the art of ‘paradigm maintenance’: the World Bank’s Development Economics Vice-Presidency, which originally ran in the August 2006 issue of the Review of International Political Economy. After it was published, Daniel Lederman and Martin Ravallion of DEC wrote to the board of the journal calling into question the quality and accuracy of Broad’s article, labelling it as “shoddy scholarship.” The journal’s editorial board, however, defended its decision to publish Broad’s piece and offered the Bank a chance to debate Broad in the pages of the journal. With the impending release of the official evaluation, perhaps it is not surprising they turned down the offer.
“Paradigm Maintenance”, or why we can’t trust the World Bank’s research
D-day for Dollar
The examples given of the best of Bank research include several controversial choices (teacher absenteeism, project evaluation using randomised trials, and the Investment Climate and Doing Business surveys) as well as elemental statistical work such as the World Development Indicators.
When asked to point to the most flawed examples, the evaluators chose some of the highest-profile research conducted by the Bank over the past decade. Of Dollar and Burnside’s paper Aid, policies and growth, which the Bank has cited repeatedly to argue for increased support for countries which it considers to have ‘good policies’: “We think that the Bank was unwise to place so much weight on one paper whose evidence is so unconvincing.” The implications for the Bank, say the evaluators, are alarming: “once the evidence is chosen selectively without supporting argument, and empirical scepticism selectively suspended, the credibility and utility of the Bank’s research is threatened.”
Heavily slated is Dollar and Kraay’s research providing succour to Bank arguments that trade liberalising countries have seen greater poverty reduction: “Much of this line of research appears to have such deep flaws that, at present, the results cannot be regarded as remotely reliable.” International trade director Uri Dadush will undoubtedly be looking over his shoulder in the new year, as the evaluators find that, in addition to the misplaced advocacy, Bank trade work more generally has “insufficiently addressed the effects of trade on poverty”, and has been dominated by arcane computable general equilibrium models. Dissenting research conducted within the Bank, such as that by Branko Milanovic, has been routinely “ignored” (see Update 30). This repeats criticisms that were made of the Bank’s trade work last year in an evaluation conducted by the Independent Evaluation Group (see Update 50).
Other research which comes in for heavy criticism includes that on:
- Pensions and insurance: “The analytical errors referred to are those that would be well understood by a first-year graduate student in economics.” Criticism of the pro-privatisation bias of the Bank’s pension reform team echoes similar disapproval expressed in two earlier internal Bank reports: Keeping the Promise of Old Age Income Security, by the Bank’s Latin America department and last year’s evaluation of Bank assistance for pension reform by the IEG. Trade union opposition to the Bank’s pensions policy was initially rebuffed, but a conference has now been scheduled in 2008 for the Bank to review its policy.;
- Poverty mapping: “Our recommendation is that this work be put on hold until the statistical problems are resolved.”; and
- Civil war: Paul Collier’s work “lacks an appropriate conceptual and empirical framework”.
World Development Reports escape the knife
The survival of the Bank’s flagship annual report, the World Development Report (WDR), is brought into question by the evaluators’ comments. Interviews with former heads of Bank research confirmed Bank critics’ worst suspicions that WDRs “were a prime example of research where the conclusions are either predetermined or negotiated in advance.”
While the evaluators said that WDRs are valued for providing a comprehensive literature summary and for their “breadth of scholarship”, the enormously costly annual exercises tend to downplay critical policy trade-offs, overlap hugely from year to year, and are marred by poor editing and presentation. Of the 2005 WDR on the investment climate (see Update 42 , the evaluation finds it “is almost a caricature of the view that everything is important”. “If there are priorities, they are vague and constantly changing”, and the report notes that “virtually every conceivable aspect of a country’s social, political and economic institutions affects its investment climate.”
The evaluators note with concern the absence of an evaluation of the impact or usefulness of the WDRs. In their recommendations they consider going to publication every two years, but in the end, defer to the importance of the WDR as an annual high-profile vehicle for Bank research. Of the Bank’s other high-profile reports, the evaluation says “the Bank produces too many long (and sometimes unreadable) book-length reports that are ostensibly directed at policymakers, but seem very unlikely to be read by them.”
A missed opportunity?
Astonishing considering the analysis which precedes it, is the evaluators’ recommendation to increase support to Bank research through the creation of a research department. They recommend the establishment of a “research endowment fund” taken from Bank’s retained earnings – that is, taken from developing countries’ interest payments – to pay for it. Other more useful recommendations include:
- Increasing the presence of developing country researchers and support for institution-based research in developing countries;
- The establishment of a peer review mechanism for research output; and
- Improved cost accounting for research – cutting back on too many thick volume flagship reports and improving those that remain.
Not surprisingly, suggestions that there should be more funding for research have been warmly received at the Bank. Bank chief economist François Bourguignon refuted the evaluation’s description of Bank research as “undistinguished”. The Bank has said it agrees with the recommendation to build research capacity in developing countries and will ‘increase their efforts’. There was no comment on the recommendation to begin peer review of Bank research, and on the suggestion to cut its major outputs, the Bank will “explore approaches to reduce the volume of reports where possible”.
But while the evaluation has shed light on a number of concerns, other more serious problems are left unaddressed. Two DEC staff, economist Vijayendra Rao and sociologist Michael Woolcock, stress that “economics should not have (as it currently does at the Bank) a near-monopoly on determining the content and validity of development research”. Rao and Woolcock criticise the evaluators themselves over their methodology and for the complete absence of non-economists in the team. They urge the Bank to “engage with alternatives to the dominant views in development research” and expand the number of staff with non-economic training.
Even more fundamental is the belief of David Ellerman, visiting scholar at the University of California and advisor to former chief economist Joseph Stiglitz, that the root of the problem lies in the fact that the Bank taking a stand itself on the issues contradicts the institution posing as an unbiased research organisation: “If any university took an official stand on certain issues in physics, biology, or the social sciences, then it would quickly distort the future research by faculty members since any contrary findings would ’embarrass’ the university.”
Which harkens back to the question of whether or not the Bank should play the role of global development ‘knowledge bank’. On the basis of similar findings to those of the official evaluation, but derived from independent research, American University professor Robin Broad (see Update 53) concludes: Bank backers “would do far better to support independent research institutions that are stimulating a more diverse development debate”.
