0882: "Significant"

[Dason]

If you're just going to lower your alpha level according to the p-value what is the point of even reporting the alpha level? To me it becomes meaningless at that point. Just report the p-value if that's all you're going to do.

To go to your example (which I admit is a better analogy than what I gave) - just because you got lucky and had a better crop of students this year doesn't mean that that's your standard. Let's reduce it down to the case of only considering accepting a single student. Your requirements say that they need to be in the top 5%. You would gladly accept anybody in the top 5%. You get lucky and the student you pick is in the top 1%. Does that mean that you should report that you only accept students in the top 1%? No - because you're not that strict and you're just lying to the public if you say you are. What's the point of even reporting alpha then?

[biodomino]

Well, my point was actually that often times .05 is used where .01 could have just as well been used. The results don't change, so what's wrong with indicating a lower probability of a type I error? If .01 could have been used but .05 was used as the indicator of significance, I see no problem in selecting a more favorable alpha considering p will be reduced accordingly.

i.e., if .01 CAN be used while maintaining statistical significance, but isn't, doesn't that reflect that 1/20 studies that report at a .05 level committing a type I error is actually a conservative estimate? If 20/20 studies report at .05 but find significance at .01, then the risk of a type I error is only 1/100 at the respectively lower p values. I don't think the lay person understands that it's the relationship between alpha and p that is telling. When you report only that p>.05 (as the comic did), but not the size of the effect, you conceal the fact that this applies only to the size of the effect, and not the standardized "validity" (for lack of a better word) of the study.

When we only report significance/insignificance according to a .05, we don't actually give any good indication of the robustness of the findings. Granted this may not have much practical significance, but I thought this is why many studies report the level at which p is significant in addition to an a priori alpha.

[sora-ten]

p<0.05 indicates the probability of being wrong (due to coincidence, not causality) is less than 5%. p>0.05 indicates the opposite. Therefore the symbol should be > for green jellybeans (biologists report exact values in this case) and < for all others (in which case p<0.05 is sufficient). We can also state p<0.01 when we want to say it's REALLY good data, but we never write p=0.00 because that can't really be true even if the computer spits it out.

tl;dr: Please exchange > and < in this comic!

[phlip]

p<0.05 indicates the probability of being wrong (due to coincidence, not causality) is less than 5%. p>0.05 indicates the opposite.

Not quite... it's the other way around: p<0.05 means that, if we were wrong, there's a less than 5% chance we'd get a false-positive result anyway. As we see in the comic: they do 20 tests, and get 19 negatives and one false positive; a 5% false-positive rate, which is what you expect setting p<0.05 as your significance threshold. Note that this doesn't translate directly directly into a "probability of being wrong"... to do that you need Bayes' theorem, and to use that you need a prior.

Therefore the symbol should be > for green jellybeans (biologists report exact values in this case) and < for all others (in which case p<0.05 is sufficient).

No, he has it the right way around - he says p>0.05 for the tests that have no link, and p<0.05 for the test that (by chance) does find a (false) link.

[Editer]

As usual, it takes a while, but science catches up with xkcd.

This article in the journal Psychological Science explicates the effects of running a lot of tests and only reporting the ones with the low p-values. (The scientists in the comic don't do this, the newspaper does, but the result is effectively the same.)

To help illustrate the problem, we conducted two experiments designed to demonstrate something false: that certain songs can change listeners’ age. Everything reported here actually happened.

I love the term the authors use, "researcher degrees of freedom".

Good commentary here, too.

A key part of the story is that, although such manipulations could be performed by a cheater, they could also seem like reasonable steps to a sincere researcher who thinks there’s an effect and wants to analyze the data a bit to understand it further.