Many research reports contain interesting data, but fail to draw policy-relevant conclusions from them, or to present their conclusions in a compelling fashion. Some of the most common shortcomings are:
- drawing implications that are not based on the data. (Sometimes, the recommendations could have been written before the project was undertaken.)
- misinterpreting data to draw unwarranted conclusions (e.g. if the data show that Option A is 0.1% more costly than Option B, this does not mean that Option B is clearly preferable. It means that the two options cost essentially the same and that the choice should be made on other grounds.)
- providing generic recommendations that could pertain to almost any problem (e.g. “The government should provide subsidized credit…”)
- burying worthwhile conclusions here and there within the report, rather than distilling them in a concluding section.
- drawing vague conclusions (“policy makers should take these findings into account when making decisions…”) or not drawing policy implications at all.
This paper analyzes a sample of ten articles from the environmental economics literature that are particularly good in drawing policy recommendations or policy implications from empirical data. Each article is summarized in terms of structure and content, and the things that make it particularly effective are discussed. The paper concludes by summarizing some of the elements of an effective policy paper.
Sidebar: Policy recommendations vs. policy implications
A policy recommendation is a statement that makes a specific proposal for action. (E.g. “Agency X should do the following …”). Policy implications also interpret data in ways that are useful to policymakers, but without specifying precisely what should be done. A (fictitious) example of a policy implication is:
“Our research shows that the damages from air pollution in Manila are much greater than those from water pollution.”
The statement suggests that the government should devote more attention to air pollution, but it does not recommend any specific measures.
How useful this policy implication would be depends on the context. If the finding is surprising, and the government is currently ignoring air pollution, then simply pointing out the seriousness of the problem could be very useful. But if everyone already accepts that air pollution is more serious, it is not so useful.
There is, of course, a gray area between the two. In the example above, the authors could have added “… therefore the government should devote more resources to reducing air pollution”. That is a recommendation. But it is not a very specific one – it does not recommend increasing the tax on gasoline by 35% or requiring that all new power plants use hydro-electricity instead of coal.
Policy implications are often presented in an “if … then …” format. Sometimes a recommendation can be phrased as an implication to make it seem less “aggressive”. E.g.:
“Because Option A will cost $ 1 million more than Option B, the government should adopt Option B.”
“If the government chooses option A, the cost will be $ 1 million higher.”
The working hypothesis of this review was that the most important factor in producing policy-relevant results is a good research question. If you don’t ask a good question, you won’t come up with anything interesting. It is hoped that these examples will suggest some good questions and approaches to research that others could emulate.
The ten articles were culled from a sample of 400. At each EEPSEA biannual workshop since 1993, a set of 25 background papers were provided to workshop participants – a total of about 400 by November, 2001. Each set of background papers was collected over the previous six months from journal articles, gray literature and media reports in environmental economics.
The author scanned the titles of these 400 items and found that 40 had policy-oriented titles. The remainder appeared to report data; to describe recent events; or to demonstrate a methodology – but not to make explicit policy recommendations. This small number was something of a surprise, since the 400 items had been selected for their practical value to a group of applied environmental economists. Theoretical and methodological pieces were already greatly under-represented, relative to what one would find in the average academic journal. In spite of this, only 10% of the sample were obvious policy papers! So the tendency among researchers to forego policy analysis seems to be more the rule than the exception.
Of the 40 pieces with policy-relevant titles, only 30 proved to have actual policy relevance on closer examination. Those that did not displayed weaknesses such as the following:
- “conclusions” that were actually hypotheses, and were not supported by data.
- models that used only algebra and no data. (This is a valid form of research, but is unlikely to be persuasive to policymakers.)
- vague recommendations (“it all depends…”, “more research is needed…”, “it’s all very complex…”).
- something important missing in the exposition of the research process that undermines the report’s credibility (e.g. no description of the methodology).
Of the 30 policy-relevant papers remaining, ten were selected to provide a good range of research questions and styles of research. They were judged to be effective examples of policy analysis, not because they were known to have actually had policy impact but because the reviewer found them to be persuasive. In other words, “If I were a policymaker, I would understand this paper; find it credible; get a clear idea of what action needs to be taken; and be persuaded that the recommendations are sensible”.
The ten papers are discussed in turn below. Click here to continue.
By David Glover, for 18th EEPSEA Biannual Workshop: May 21-24, 2002