Reevaluating Reinhart and Rogoff

This week one of the most cited papers and research findings in the past few years has been questioned. Reinhart and Rogoff's findings that GDP growth slows down after government debt levels reach 90% of GDP, which allegedly became the justification to use austerity as the best way to cut down the debt and reignite economic recovery, has been recalculated by three economists from University of Massachusetts  Thomas Herndon, Michael Ash and Robert Pollin. They have found coding errors in the excel spreadsheet, selective exclusion of some data, and unconventional weighting of summary statistics in the original paper. After fixing for these errors they found that GDP growth for countries above 90% public debt-to-GDP ratios is 2.1% on average, and not -0.1%, which was the initial Reinhart-Rogoff finding. This implies that it doesn't make a difference whether or not an economy is burdened with high public debt. You can read the details behind the mistakes on the Rotrybomb blog, or from Matt Yglesias, the FT, Tyler Cowen's comments, and some pretty harsh criticism from Paul Krugman. 

In comparison the findings from the two papers look something like this (data from 1945 to 2009): 

	Reinhart & Rogoff		Herndon, Ash & Pollin
Debt/GDP	Mean	Median	Mean	Median
< 30	4.1	4.2	4.2	NA
30-60	2.8	3.9	3.1	NA
60-90	2.8	2.9	3.2	NA
>90	-0.1	1.6	2.2	NA

While a table with a longer time span, going back to 1800, looks like this (HAP paper didn't look into this data): 

Reinhart & Rogoff		Herndon, Ash & Pollin
Debt/GDP	Mean	Median	Mean	Median
< 30	3.7	3.9	NA	NA
30-60	3	3.1	NA	NA
60-90	3.4	2.8	NA	NA
>90	1.7	1.9	NA	NA

Source: FT 

Both tables still yield the same conclusion: GDP growth tends to diminish as debt-to-GDP increases. The difference is that the effect isn't as severe as the initial results of RR seemed to show. 

Source: The Economist

The database used in the original paper was from Reinhart and Rogoff's brilliant and several award-winning financial history book, This Time is Different. Their paper that followed afterward concluded the following:

"First, the relationship between government debt and real GDP growth is weak for debt/GDP ratios below a threshold of 90 percent of GDP. Above 90 percent, median growth rates fall by one percent, and average growth falls considerably more. We find that the threshold for public debt is similar in advanced and emerging economies. Second, emerging markets face lower thresholds for external debt (public and private)—which is usually denominated in a foreign currency. When external debt reaches 60 percent of GDP, annual growth declines by about two percent; for higher levels, growth rates are roughly cut in half. Third, there is no apparent contemporaneous link between inflation and public debt levels for the advanced countries as a group (some countries, such as the United States, have experienced higher inflation when debt/GDP is high). The story is entirely different for emerging markets, where inflation rises sharply as debt increases."

All these findings were highly receptive because they made a lot of logical sense. As was their financial contagion mechanism, and suggestions for policymakers. However, once it came under scrutiny, Reinhart and Rogoff admitted to the coding error, but dismissed other attacks and stood firmly by their conclusions: 

"…Herndon, Ash and Pollin accurately point out the coding error that omits several countries from the averages in figure 2. Full stop. HAP are on point. The authors show our accidental omission has a fairly marginal effect on the 0-90% buckets in figure 2. However, it leads to a notable change in the average growth rate for the over 90% debt group. 

HAP go on to note some other missing debt data points, which they describe as “selective omissions”. This charge, which permeates through their paper, is one we object to in the strongest terms. The “gaps” are explained by the fact there were still gaps in our public data debt set at the time of this paper. 

Our approach has been followed in many other settings where one does not want to overly weight a small number of countries that may have their own peculiarities. 

So do where does this leave matters on debt and growth? Do Herndon et al. get dramatically different results on the relatively short post war sample they focus on? Not really. They, too, find lower growth associated with periods when debt is over 90 per cent. Put differently, growth at high debt levels is a little more than half of the growth rate at the lowest levels of debt."

You can read their full response here and here. Cowen defends them.

What are the implications of all of this? The first one is the trustworthiness of non-peer reviewed papers (like NBER working papers of AER proceedings) to be used for policy implications. On the other hand, since journal articles became more restrictive to information access, it is no wonder working papers became much more attractive and widespread. As Ryan Avent concludes, this should "convince economists that more openness is better. Retreating back behind ivory walls will only protect their ideas by helping to make them irrelevant."

This brings us to the second implication; the blogosphere and the consequential immediate reaction of economists to the research has become more important than economic journals in enhancing and stimulating economic discussions and even at influencing policy, as Krugman has noted a few years back:

"First of all, policy-oriented research was never as centered on refereed journals as we liked to imagine. A lot of the discussion always took place via Federal Reserve and IMF working papers, and even reports from the research departments of investment banks. The rise and fall of Fed policy via targeting of aggregates, for example, was not a debate played out in the pages of the JPE and the QJE. 

Second, even for more academic research, the journals ceased being a means of communication a long time ago – more than 20 years ago for sure. New research would be unveiled in seminars, circulated as NBER Working Papers, long before anything showed up in a journal. Whole literatures could flourish, mature, and grow decadent before the first article got properly published – this happened to me with target zones back in the late 1980s, where my original 1988 working paper had spawned a large derivative literature by the time it actually got published. The journals have long served as tombstones, certifications for tenure committees, rather than a forum in which ideas get argued. What the blogs have done, in a way, is open up that process." 

Couldn't agree more, journal articles have a much higher influence on academic tenure than on spurring new ideas into discussion, especially if the discussion is on economic policy. 

As for the policymakers using new economic research as their justification for austerity, I only wish they are so receptive. It has been signaled by numerous research papers, discussions, op-eds, blogs, etc. that every European country is conducting a wrong tax-based approach to austerity, but this hasn't stopped the policymakers from doing so, hasn't it? The reasons are purely political, and have little to do with economics (as I've summarized back in September). 

Finally, how have these new findings affected our feelings on the issue of debt and the recovery? Well, not much. Debt is still bad. Even if there is a reversed causality in the debt and slow growth link, there is bound to be something wrong with an economy that doubles or even quadruples its public debt in a few years while using the classical Keynesian response to the crisis. In order to avoid several decades of stagnation that seems to be a necessary outcome of such an approach (combined of course with tax-based consolidations), the focus should still stand on broad-based pro-competition, pro-market reforms. Or to be more topical, revoke the Margaret Thatcher approach.