How to go from an unknown graduate student in Economics to front page news of New York Times.

How to go from being an unknown graduate student in Economics to landing the front page of the New York Times in one week? And no, you don’t even have to win a Nobel Prize in Economics at 24. All you have to do is replicate a paper. Yes, you did read it right- just replicate a paper. All of a sudden your usual problem set assignment for Econometrics could result in Paul Krugman dedicating his NYT column to you. However…on your way to stardom you might need a bit of luck. And it might be helpful to choose to replicate a certain, how to say it nicely, erroneous paper of Harvard professors Rogoff and Reinhart at a time when fiscal austerity is quite a hot topic. This is the story of Thomas Herndon.

The original Kenneth Rogoff and Carmen Reinhart paper “Growth in Time of a Debt”, except for being one of the most cited American Economic Review papers, has been playing a significant role in the justification of global austerity measures. Everyone, from the FED president to random journalists were citing their crucial finding that countries whose debt exceeds 90 percent of their gross domestic product will experience slower growth than countries with a lower public debt-to-GDP ratio. And although most of the economic experts and policy makers were quite thrilled to be able to cite a scientific paper when explaining how austerity will be beneficial, one PhD student at the University of Massachusetts wasn’t very convinced and decided to look into their work as an assignment for an econometrics course that involved replicating the data work of a famous study. After some struggle to receive the original data from the authors, as New York Magazine reports, it took him very little time from opening the data Excel spreadsheet after receiving the e-mail with the data, to notice that there was something incoherent in the spreadsheet. Just minutes later, still sitting on his couch, he figured out that the data contained a computing error – Reinhart and Rogoff had forgotten to include a part of data on countries with high public debt-to-GDP ratios that would have altered their overall calculations (typing AVG(L30:L44) instead of AVG(L30:L49) they excluded data from Canada, Belgium, and Australia: economies that had positive growth  even during periods of high debt). The discovery of this kind of error in such a seminal work by a graduate student seems very much like a David and Goliath story.

(Brad Plumer, the Washington Post)

To quote Herndon (published as Herndon, Ash & Pollin) referring to Rogoff & Reinhart “A necessary condition for a stylized fact is accuracy”, and accuracy is something Rogoff & Reinhart lack. Herndon, Ash & Pollin found that when correctly calculated (correcting for the coding error, data omissions, and strange aggregation procedures), the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0.1 percent as published in Rogoff & Reinhart^[1].  This implies an important conclusion: average GDP growth at higher levels of public debt-to-GDP ratio is not significantly different than at lower ones. In an article in the New York Times in 2010, just after “Growth in Time of a Debt” was published,  NYT praised professors Reinhart and Rogoff for “leading and encouraging others in their field to look beyond hermetically sealed theoretical models and into the historical record”, a statement that  now seems quite ironic- maybe now they have learned that looking into historical record, although indeed very important in economic analysis, should not mean selectively choosing historical data to support one’s theory.

(Jared Bernstein, the Washington Post)

Although not addressed in the Herndon, Ash & Pollin paper, a stronger critique of the Rogoff & Reinhart paper is the question of causality, since as we all have learned in Econometrics 101 “correlation does not mean causality”. UMass Amherst professor Arindrajit Dube added to the discussion showing backward causality that is quite intuitive: countries have high debt-to-GDP ratios because they have slow growth, rather than the other way around.

But perhaps the most interesting result of this finding is the debate it stirred afterwards. It definitely made the day of a famous anti-austerian economist Paul Krugman and he is having quite a lot of fun these days with continuous columns in his blog for the New York Times^[2]. The numbers in the Rogoff & Reinhart paper go far beyond their numerical values, they have had a strong impact on the pro-austerity debate in these recent times of crisis. While fans of austerity believe that governments should cut spending in order to grow their economies, anti-austerians believe that government spending in times of economic crises can create growth and reduce unemployment, although it increases debt in the short term. As Krugman points out (in his column on the NYT, The Excel Depression) “what the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But “economic research” showed no such thing; a couple of economists made that assertion, while many others disagreed.”

Even if all the nerdy conspiracy theories can be quite entertaining, the question remains: did Rogoff & Reinhart intentionally play around with their data to support pro-austerity measures in 2010, when the debate on how to deal with the PIGS was at its peak, or was it just a random mistake by their over-employed research assistant in some late-hours Excel and Stata work? Finally, it remains to be seen how Herndon’s results will affect future austerity measures support, and whether Rogoff and Reinhart’s academic respectability will be put into question.

And what about our Thomas Herndon from the beginning of the story? As New York Magazine reports he has to deal with academic celebrity now, but has been postponing a proper celebration: “I’m going to celebrate this weekend,” he says. “But for now, I have a really gnarly problem set.”. Makes us feel a little better at least.

Maybe on a bit of a side note, let me leave you thinking with perhaps now quite an ironic anecdote. New York Times in 2010 (just after the publishing of the original paper) in an interview with Rogoff revealed that his favorite economics joke is “the one about the lamp-post”: A drunk on his way home from a bar one night realizes that he has dropped his keys. He gets down on his hands and knees and starts groping around beneath a lamp-post. A policeman asks what he’s doing. “I lost my keys in the park,” says the drunk. “Then why are you looking for them under the lamp-post?” asks the puzzled cop. “Because,” says the drunk, “that’s where the light is.”

Sources:

Carmen M. Reinhart & Kenneth S. Rogoff, 2010. “Growth in a Time of Debt,” American Economic Review, American Economic Association, vol. 100(2), pages 573-78, May.

Herndon, T., Ash, M., Pollin, R.. “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff”, Political Economy Research Institute Publications, available at – http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/

Roose, K. (The New York Magazine): The Graduate Student who shook global austerity movement. http://nymag.com/daily/intelligencer/2013/04/grad-student-who-shook-global-austerity-movement.html

Plumer, B. (The Washington Post):  Is the evidence for austerity based on an Excel spreadsheet error? – http://www.washingtonpost.com/blogs/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/

The Financial Times: http://blogs.ft.com/the-world/2013/04/reinhart-and-rogoff-your-essential-reading-list/?

The Conscience of a Liberal (Paul Krugman blog for the New York Times) – http://krugman.blogs.nytimes.com