PrawfsBlawg: More Problems With Hypothesis Testing (From a Mo...

The confidence level is set in advance. The resulting p-value is a random variable. To adjust the level in response to the p-value is an improper ex post move. So if the pre-set level of significance is 95% and the resulting p-value is 0.0001, the proper response is to say "The results are significant at the 95% level (p = 0.0001)." To readjust the claim to "The results are significant at the 99% level (since) p = 0.0001" is simply incorrect.

Thus: there should only be one star per table, at whatever level the analyst sets in advance.

Highlighted by cgibbons

And so it is disappointing that rarely if ever do we see a paper wrestle with the proper level of significance, such as by asking whether this is a case where a false negative is better or worse than a false positive. After all, it is not always clear that we are best served by the conservatism of the 95% confidence interval. A false positive may be worse than a false negative in criminal law ("better 10 guilty men go free..."), but a false negative may be worse in some medical situations, such as whether a particular pill works.

Highlighted by cgibbons