The Significance of Significance

William Easterly admits to being sloppy with his statistical reporting.

Aid policy was based on the premise that aid raises growth, but …{a major} study of this question was saying that this premise was false.

This quote refers to the Rajan-Subramanian paper (later published in a peer-reviewed journal) that was unable to reject the hypothesis of a zero effect of aid on growth. As I never tire of pointing out, we often get our conditional probabilities mixed up. Based on standard statistical methodology, the (1) probability of failing to reject the zero effect hypothesis is high when the effect is indeed zero. Unfortunately, the author of the quote incorrectly thinks this implies the opposite probability is high — (2) the likelihood that the effect is indeed zero when you fail to reject the hypothesis of zero. This likelihood can actually be quite low even if the first probability is high.

This is an important point, but it's not intuitive. Let me take a moment to interpret.

Suppose you and some friends are out partying but your friend Bob didn't show up. Where's Bob? It's late: Bob's probably at home. Bob being at home is your null hypothesis. (When I first learned about null hypothesis, I learned it as the theory that nothing interesting's going on. It's more complex than that but that will suit us for our purposes.)

You decide to call Bob to figure out if he can come party with you. Granted, Bob might be busy playing poker or getting drunk at his favorite bar. But he also might be home and it's a lot easier to get Bob to do something when he isn't doing anything.

If Bob tells you he's at home, you can accept the null hypothesis. Bob is indeed at home. (Technically, you never actually accept the null due to mathematical constraints but ignore that to build the intuition.) If Bob tells you he's in the gutter somewhere, at a strip club, or doing something else "interesting," you reject the null hypothesis. But if Bob doesn't pick up the phone, if it just rings and rings and rings, then you fail to reject the null hypothesis. This is not the same thing as accepting the null. Bob could be asleep in bed OR he could be in jail after having just spray painted a cop's car while wasted on vodka. You just don't know.

That confusion, that not getting an answer is the same thing as getting something boring, is the confusion William Easterly made. The Rajan-Subramanian paper didn't get statistical significance when it came to aid's relation to growth which is the same as the phone not picking up. To quote Easterly once more, "Absence of Evidence does not constitute Evidence for Absence."