The Myth of Robust Analysis

// UPDATE 12 april 2013: See my follow-up post, which invalidates this one.

If you mix red pigment into white paint, the paint gets red.

Yesterday we looked at Chart 2 from The Myth of 'Jobless Recoveries'. Today we consider Chart 1. Laurence Ball, Daniel Leigh and Prakash Loungani write:

Chart 1 illustrates the fit of the estimated Okun’s Law by plotting the unemployment gap (the gap between unemployment and the natural rate) against the output gap (output relative to potential). The relationship is very tight. No year is a major outlier in the graphs.

Graph #2: Ball, Leigh and Loungani's Chart 1.
Okun’s Law, 1948-2011 (Annual US data)

Yesterday, to figure out Chart 2, I reorganized a formula to make it compatible with Okun's law. Chart 2, as I understand it, compares actual unemployment to some calculation of the output gap and the natural rate of unemployment, and finds an amazingly close match for 65 years.

After re-arranging terms, Chart 2 compares the unemployment gap to the output gap. Today's focus, Chart 1, compares the unemployment gap to the output gap directly, one gap per axis. Both charts from the Econbrowser post consider the Okun's Law relation. Well, sure: The topic of that post is the validity of Okun's law.

The Econbrowser post includes both charts under the heading "U.S. evidence: Fit & Stable". To me this means that Laurence Ball of Johns Hopkins University, and Daniel Leigh and Prakash Loungani of the IMF, consider their graphs evidence of the validity of Okun's law. I do not.

Why not? Look again at what they wrote:

Chart 1 illustrates the fit of the estimated Okun’s Law by plotting the unemployment gap (the gap between unemployment and the natural rate) against the output gap (output relative to potential). The relationship is very tight.

They compare the unemployment gap to the output gap and find astonishing validity. That is to say, there is a tight relation between two gaps: the gap between actual and hypothetical unemployment, and the gap between actual and potential output. A very tight relation.

But here's the thing. To figure potential output, the Congressional Budget Office uses the unemployment gap. They take that gap and stretch it to fit over actual output. That gives them the output gap. The output gap looks like the unemployment gap by design.

Then they use actual output and the output gap to estimate potential output. So when you look at potential output, part of what you're seeing is the unemployment gap.

Just as Milton Friedman worked inflation into his "money relative to output" (by using inflation-adjusted output in the denominator), the CBO works the unemployment number into the calculation of potential output. Just as Milton Friedman's results are skewed to look like inflation, so is the CBO calculation of potential output skewed to look like unemployment.

In the case of the CBO, this is not a problem. CBO is modeling potential output, based on the natural rate of unemployment. There's nothing wrong with that. But for people who want to use potential output in other calculations, it is important to remember how potential output is figured. Otherwise you can end up doing bad arithmetic, just like Friedman did.

Laurence Ball, Daniel Leigh and Prakash Loungani take potential output and work it backwards. They show similarity between the output gap and the employment gap. But you should expect to see this similarity, just as you should expect to see similarity to inflation in Milton Friedman's numbers. But the similarity does not mean what Ball, Leigh and Loungani say it means. They say it is evidence Okun's law is valid.

It is not evidence. The output gap looks similar to the unemployment gap because the output gap has the unemployment gap figured into it: The paint is red because it has red pigment mixed in.

All they have done, really, Ball and Leigh and Loungani, is check the CBO's numbers. CBO starts with unemployment and ends up with potential output. Ball and all start with potential output and end up with unemployment numbers. If Ball's final numbers match the numbers CBO started with, it only means that nobody made a mistake in their arithmetic. It doesn't mean Okun's law is valid.

I suppose if they use Potential Output from some other source, calculated some other way, then my objection may be the thing that's not valid. That could be. But if they identified their data sources in the post, I didn't see it. And since their charts show an unbelievable validity, I cannot trust their charts.

// Update 6:26 AM 17 March 2013

In response to an email, Laurence Ball replies:

In our main results, we calculate potential output and output gaps separately from unemployment gaps and the natural rate, using the Hodrick-Prescott filter in each case. You're right that using CBO output gaps is circular--we mention that in the paper.