xkcd

by **philip1201** » Wed Apr 06, 2011 6:50 am UTC

The scientists and people are actually the same people. The actual scientists were too busy playing minecraft, so the girl just undoes her hair and the guy wears Science Goggles, so the news just thought they would be scientists.
Once I was subscribed to a popular science magazine. Then I noticed they did this. Now I'm no longer subscribed*.
Also, while it's necessary for the joke, there is no way they would post the margin of error in a newspaper.

(* because my free yearly subscription ran out. See what happens when you assume causation from correlation?)

by **nico** » Wed Apr 06, 2011 6:51 am UTC

wackojacko1138 wrote:Say it with me kids:
"The p-value represents the likelihood of the evidence given the null hypothesis, not the likelihood of the null hypothesis given the evidence."
If you know, understand, and live that sentence, then you understand statistics better than most journalists and some scientists.

Thomas does not agree with you, Sir

PS: this is totally a non plausible situation. With a result like that, newspapers would have written that all candies are fatal AND increase your risk of atherosclerosis by 57%.

by **Pfhorrest** » Wed Apr 06, 2011 6:52 am UTC

So is the joke that the scientists, egged on by the annoying people, tested (all?) 20 flavors of jelly bean to conclusively show "no, there really isn't any link", but then made a typo with their inequality sign on one, and together with the number of other tests run that one (false, due to typo) result becomes much more significant than it would be taken in isolation? I confess I don't completely understand the math at work here, but I would think the first test would have to have demonstrated a link for that to work, i.e. "Hmm, jelly beans appear linked with acne. I wonder if it's just one color? [20 tests, only green shows link] Yep, seems to be just green."

by **A_S00** » Wed Apr 06, 2011 6:55 am UTC

In similar (if slightly old) news: http://www.wired.com/wiredscience/2009/09/fmrisalmon/ - Neuroscience's infamous dead fish experiment (another cautionary tale about correcting for multiple comparisons)

Re: The Bonferroni discussion above, Bonferroni correction is an ugly thing to do to your statistical power, to be avoided at all costs. If the experimental design allows it, I'm a fan of Fisher-protected t-tests. Pretty good rate of type 1 error control on the multiple comparison front, with no cost whatsoever to the power of the pairwise comparisons (the only loss of power for the experiment as a whole is from the possibility of a type 2 error when testing the significance of the multiple R...and if your multiple R is small enough that you care about your power when trying to detect it, your effect sizes are probably small enough that nobody cares).

by **VectorZero** » Wed Apr 06, 2011 6:59 am UTC

Now if the claim was "jelly beans cure acne" and the scientists were "jelly bean manufacturer in-house scienticians" ...

@pforrest: short answer, no. p<0.05 means there is a less than 5% chance the result is due to chance. If you do a single test with an a priori expectation of a result (say, by detailing a physiological pathway by which a pharmaceutical agent would act on a system), then p<0.05 is accepted as 'significantly' likely to be a genuine result (it's an arbitrary figure). If you do 20 tests, with no a priori reason to expect an outcome and post hoc pull out the 'significant' results then you are less justified in claiming statistical significance. It still might be true. It is probably due to chance.

by **nico** » Wed Apr 06, 2011 7:05 am UTC

Pfhorrest wrote:So is the joke that the scientists, egged on by the annoying people, tested (all?) 20 flavors of jelly bean to conclusively show "no, there really isn't any link", but then made a typo with their inequality sign on one, and together with the number of other tests run that one (false, due to typo) result becomes much more significant than it would be taken in isolation? I confess I don't completely understand the math at work here, but I would think the first test would have to have demonstrated a link for that to work, i.e. "Hmm, jelly beans appear linked with acne. I wonder if it's just one color? [20 tests, only green shows link] Yep, seems to be just green."

This is an example of multiple comparisons fallacy.

You may want to have a look at this:
http://en.wikipedia.org/wiki/Multiple_comparisons

The problem is that the p-value does NOT tell you that acne and green jelly beans are correlated 95% of times.
It tells you that you have <5% (1/20, the number of panels is well chosen) probability that the correlation in the set of data you gathered is just due to chance.

by **cjdrum** » Wed Apr 06, 2011 7:07 am UTC

Oops, I just skimmed through reading the colours... Then I got to the bottom frame and was like "Wait..." *checks*

:lol:

by **madock345** » Wed Apr 06, 2011 7:07 am UTC

Qwerty.55 wrote:
madock345 wrote:
ethereal_fire wrote:madock345, I like your sig : )

I enjoy recursive signatures.

I like your new recursive sig even better than before.

Me too!

by **syko_lozz** » Wed Apr 06, 2011 7:16 am UTC

ummmm.... ok I am one of those life-scientists that totally doesnt get stats. I really dont see whats wrong with this.
So thinking out loud... yeah if I was doing the experiment I'd put them all onto one graph and tell my software to do an anova (rather than a whole bunch of separate t-tests).... but what if they were all independant experiments and didnt share a common control?

of course if the p values were all close to 0.05 then yeah its dodgy but what if the green one was way below 0.05 and really clear with a really big sample and a huge fold difference? and all the others had really high p-values?

is the point just that you cant rely on p-values alone? arent we missing a whole bunch of other details about the experiment to really judge whether the newspaper article was stupid or not?

these sorts of experiments are done all the time, for example, the effect of various drugs on growth of a cell line. (drug A, no effect, try the next one, drug B, no effect, drug C, ooh effect!) but this is spread over years and it seems silly to consider it all as one huge data set.

teeeaaach meeeee!

edit- will read article, but still, teach me!

by **soveraign** » Wed Apr 06, 2011 7:23 am UTC

- Read panel 1
- Think to self, "I'm a scientist, I've heard of minecraft, what is this?"
- FOUR HOURS later, return to comic and read rest of panels.

by **paravatar** » Wed Apr 06, 2011 7:29 am UTC

thank you for this one!

yesterday I tried to explain the dangers of multiple comparisons to a coworker!

… but the comic strip lacks a proper "table one" and it gives no information if the trial was randomized…
with science like that you'll never make it to the New England Journal ;-)

by **Scarlet** » Wed Apr 06, 2011 7:48 am UTC

Oh, science!

by **Turing Machine** » Wed Apr 06, 2011 7:52 am UTC

haha SMBC was funny when it did this

so long ago

by **VectorZero** » Wed Apr 06, 2011 7:54 am UTC

Also @pfhorrest: the initial test of all jelly bean colors doesn't have to show a difference. In fact, the correlation of the green jelly beans would have to be very strong indeed to show through the noise of 19 other colors that don't have a correlation with the expected outcome.

@syko_lozz: you're right, there is insufficient information to comment fully on the validity of the study. Yet that didn't stop the journos from claiming the correlation to be proven causation, did it?

by **darkspork** » Wed Apr 06, 2011 7:57 am UTC

GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.
This has been roughly my response to everything for a week.

by **Philyphil** » Wed Apr 06, 2011 8:53 am UTC

This comic is great. This is how, often old, scientific information is actually eminated. Nice work as ever. Loving the reccursive sig floating about today too!

by **glasnt** » Wed Apr 06, 2011 9:15 am UTC

This is now my excuse for everything:

"But.. miiiiiinecraft...

"

HI JOEE

by **le_sacre** » Wed Apr 06, 2011 9:24 am UTC

Love the comic, but how could you pass up the opportunity for a "Bonferroni"-related punchline in the alt-text??

To the people (including scientists!) who say they don't get it: read it again! The way to think about p-values is to say, for p < .05, "You'd expect to get an effect at least this large just by chance 5% of of the time (if the null hypothesis were true)."

5% of the time = 1 out of 20.

Most of the brilliant, accurate, low-effort heuristics that have been evolutionarily selected into the human mind deal with single cases or very simple proportions. Probability and statistics is something a lot of us struggle to deal with intuitively. However, I think the basics can be grasped by anyone if they're taught well and early enough. In my grand agenda, stats will be a major and required part of both math and science curricula starting in grade school; it's the mathematics of decision-making, which is arguably far more vital in our world than the mathematics of mechanics or architecture, which are more specialty topics.

by **lrossouw** » Wed Apr 06, 2011 9:49 am UTC

Good stats commic.

Spoiler:

by **Tetrinity** » Wed Apr 06, 2011 10:09 am UTC

I'm no statistician, but I'm glad I knew enough to get this one. I must admit I skimmed past the colours first reading too...

And upon re-reading, noticed that they tested yellow jelly beans twice, in panels 13 and 18. Presumably a mistake on Randall's part, but I suppose there's nothing wrong (scientifically speaking, at least) with repeating the experiment.

by **spekkeh** » Wed Apr 06, 2011 10:37 am UTC

Diadem wrote:Unfortunately, this is exactly how many scientists (not all, luckily, but too many) do statistics. They really do tests on dozens of variables at once, with several different statistical tests, and then publish the ones that are interesting.

For a single experiment you should always divide your allowed margin of error by the number of variables you are testing for. So if you test n different colours of jelly beans, you must require p < 0.05/n before considering a result statistically significant. But this is often ignored.

....unless you write an article about each experiment separately, in which case you won't have to do it anymore. The comic would still apply, because basically that's what p<0.05 says, that once every 20 times you will get a false positive, so if that's your alpha, dividing it by twenty doesn't make any sense. Even in the case where you perform multiple tests on the same data (of which this is not an example), Bonferroni adjustments are pretty iffy, cf.

Nakagawa, S. (2004). A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral Ecology, 15(6), 1044-1045. doi: 10.1093/beheco/arh107.

by **graatz** » Wed Apr 06, 2011 11:06 am UTC

So in other words, 1 out of every 20 studies that use a 95% CI is wrong. The number of the wrong ones that get reported by popular media is overrepresented.

(Yes, I get the fact that the joke is that testing different jelly bean colors is essentially running the same test over and over, thus the expected 1 anomaly in 20, but it's nice to be periodically reminded how big of a number 5% actually is and how arbitrary a choice it is for hypothesis testing.)

by **eidako** » Wed Apr 06, 2011 11:09 am UTC

NASA Scientist: Minecraft Increases Your IQ

Cranium Talksalot (CNN) -- Worry that your kids play too many video games? You shouldn't. A surprising report by NASA roboticist and computer programmer Randall Munroe shows a trend between employment in intellectually demanding fields and playing games. Statisticians and chemists alike have been shown to play Minecraft, the wildly popular sandbox game by indie developer Mojang, even while at work. Shortly after the report's release, Munroe's website was abuzz with fellow scientists, engineers and academics remarking that they play the game, or if they haven't, greatly anticipate doing so. Munroe is an avid player himself. He admits to having played for days on end while recovering from a dental operation, so absorbed in the complex virtual world that he was able to completely ignore any pain.

He is not alone. Over 5.8 million users have registered on the game and 1.7 million have purchased it. There are approximately 1.3 million scientists in the United States, no small coincidence. The other 400,000? One of them just might be your child.

The next time your child wants to stay up playing Minecraft, maybe you should think twice before sending them to bed. That pixelated pickaxe just might be their ticket to an exciting career in science.

by **ksquared129** » Wed Apr 06, 2011 11:15 am UTC

chaucer345 wrote:See, what they should have done this is run a single factor ANOVA with the colors as the levels and then performed a Tukey MCP to determine which of the levels were significantly different from each other... Or would that have helped? Maybe I'm thinking of this the wrong way.

This is exactly what I thought after reading the comic. Poor scientists, they would have been done so much more quickly (to get back to Miiiiinecraft), without that mistake about the green jelly beans.

by **eidako** » Wed Apr 06, 2011 11:23 am UTC

(post merged with previous)

by **KShrike** » Wed Apr 06, 2011 11:27 am UTC

But... We're playing Minecraft (don't ever get this game. It will suck your life away with it's awesomeness)

Another statistics joke. Two in a row. He must really hate statistics this week (the practice thereof)

by **radtea** » Wed Apr 06, 2011 12:09 pm UTC

Someone wrote:

"The p-value represents the likelihood of the evidence given the null hypothesis, not the likelihood of the null hypothesis given the evidence."

Which is why it is fortunate we have Bayes' Theorem and related techniques to let us use prior information to ascertain an estimate of the plausibility of the hypothesis given the evidence, which is what people who've kept up with the world have been doing for the past decade or more.

The sad fact is that far more than 5% of papers published against the rather modest standard of p < 0.05 turn out to be reporting on results due to random chance. This comic explains why.

No real experiment exists in isolation, and if your analytical techniques do not provide ways of accounting for prior knowledge, you are using the wrong analytical techniques.

by **KShrike** » Wed Apr 06, 2011 12:24 pm UTC

darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.
This has been roughly my response to everything for a week.

Haha... me too. Last Thursday too. "What's Minecraft?...... I .... can't.... stop!!!!!!

This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Mark Twain wrote:paraphrased: There are three kinds of fibs: lies, damned lies, and statistics.

Oh, Mark Twain. If only you'd understand. Being an engineer is hard in the moral sense, really. I can prove it too (p < .05).

by **SirMustapha** » Wed Apr 06, 2011 12:35 pm UTC

darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.

And then people don't understand what I mean with pandering.

KShrike wrote:This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Why? Randall's practice of namedropping nerd culture items to cause reactions like the one above is extremely common, and this is the second time he has used Minecraft.

Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

by **jthetzel** » Wed Apr 06, 2011 1:11 pm UTC

Diadem wrote:For a single experiment you should always divide your allowed margin of error by the number of variables you are testing for. So if you test n different colours of jelly beans, you must require p < 0.05/n before considering a result statistically significant. But this is often ignored.

THE CURSE OF BONFERRONI!!!

by **Balesk Baj, Timeburner** » Wed Apr 06, 2011 1:23 pm UTC

Having learned some Statistics in my third year of high school, I love this comic and can relate to it. I believe it perfectly shows how journalists and reporters sometimes either twist their reports to make it more interesting or misinterpret them due to not understanding them much.

SirMustapha wrote:
darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.

And then people don't understand what I mean with pandering.

KShrike wrote:This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Why? Randall's practice of namedropping nerd culture items to cause reactions like the one above is extremely common, and this is the second time he has used Minecraft.

Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

I find it baffling that you are complaining about Randall's small mention of Minecraft in this comic and you're not complaining on the topic of the comic itself. But what I find stranger is that you usually don't bash comics containing scientific, mathematical, or programming elements that normal people usually don't know, and when you do, you usually:

a.) complain on small, unimportant stuff in it,
b.) point out that Randall's overall presentation of the comic is sloppy, or
c.) say that Randall only posts obscure nerdy topics to gain more attention and to make himself (and us XKCD fans) feel smarter.

</trollfeeding>

by **ribbonsofnight** » Wed Apr 06, 2011 1:32 pm UTC

SirMustapha wrote:
Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

We all love hearing from perfect trolls

by **TheDoctor** » Wed Apr 06, 2011 1:57 pm UTC

I loved reading this comic! It was my 30 second break from minecraft. Must destroy creepers.....

by **Cecilff2** » Wed Apr 06, 2011 1:59 pm UTC

Hooray statistics!

I'm a poisson distribution!

I'm a poisson distribution!

I'm a poisson distribution!

by **savanik** » Wed Apr 06, 2011 2:09 pm UTC

Cecilff2 wrote:Hooray statistics!

I'm a poisson distribution!

I'm a poisson distribution!

I'm a poisson distribution!

Hmmm... given the small sample size, I would have to conclude it's still fairly likely that you're not a poisson distribution.

by **Cecilff2** » Wed Apr 06, 2011 2:12 pm UTC

I'm totally a poisson distribution.

by **jozwa** » Wed Apr 06, 2011 2:18 pm UTC

Wait...

by **neoliminal** » Wed Apr 06, 2011 2:41 pm UTC

NEWS: Scientist find correlation between acne and minecraft!

by **Angua** » Wed Apr 06, 2011 2:49 pm UTC

Did the comic change between last night and this morning? I didn't see green jelly beans referenced until the end.

by **SpringLoaded12** » Wed Apr 06, 2011 3:03 pm UTC

I was JUST eating jelly beans not 2 minutes ago :shock:

So, the public's attempts to simplify the findings of scientific studies or to use scientific studies to "prove" their hunches, gets in the way of true science?

I remember some news report from a couple years ago saying that studies showed blue M&M's could cure spinal injuries, and further details showed that the blue dye on them had been linked to improved spinal health or something. This reminds me a lot of that, I think they're in the same boat.
And, I think personally I would put the link between flu vaccines and autism in the same boat as well.

ribbonsofnight wrote:
SirMustapha wrote:
Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

We all love hearing from perfect trolls

Welcome to the fora.
He's not trolling, that really is his opinion. Check the past 300+ comic threads, it's mostly similar reactions.

Who is online