0882: "Significant"

This forum is for the individual discussion thread that goes with each new comic.

Moderators: Moderators General, Magistrates, Prelates

Re: 0882: "Significant"

The scientists and people are actually the same people. The actual scientists were too busy playing minecraft, so the girl just undoes her hair and the guy wears Science Goggles, so the news just thought they would be scientists.
Once I was subscribed to a popular science magazine. Then I noticed they did this. Now I'm no longer subscribed*.
Also, while it's necessary for the joke, there is no way they would post the margin of error in a newspaper.

(* because my free yearly subscription ran out. See what happens when you assume causation from correlation?)
philip1201

Posts: 189
Joined: Tue Nov 03, 2009 6:16 am UTC

Re: 0882: "Significant"

wackojacko1138 wrote:Say it with me kids:
"The p-value represents the likelihood of the evidence given the null hypothesis, not the likelihood of the null hypothesis given the evidence."
If you know, understand, and live that sentence, then you understand statistics better than most journalists and some scientists.

Thomas does not agree with you, Sir

PS: this is totally a non plausible situation. With a result like that, newspapers would have written that all candies are fatal AND increase your risk of atherosclerosis by 57%.
nico

Posts: 16
Joined: Fri Aug 22, 2008 6:53 am UTC

Re: 0882: "Significant"

So is the joke that the scientists, egged on by the annoying people, tested (all?) 20 flavors of jelly bean to conclusively show "no, there really isn't any link", but then made a typo with their inequality sign on one, and together with the number of other tests run that one (false, due to typo) result becomes much more significant than it would be taken in isolation? I confess I don't completely understand the math at work here, but I would think the first test would have to have demonstrated a link for that to work, i.e. "Hmm, jelly beans appear linked with acne. I wonder if it's just one color? [20 tests, only green shows link] Yep, seems to be just green."
Forrest Cameranesi, Geek of All Trades
"I am Sam. Sam I am. I do not like trolls, flames, or spam."
The Codex Quaerendae (my philosophy) - The Chronicles of Quelouva (my fiction)

Pfhorrest

Posts: 1928
Joined: Fri Oct 30, 2009 6:11 am UTC

Re: 0882: "Significant"

In similar (if slightly old) news: http://www.wired.com/wiredscience/2009/09/fmrisalmon/ - Neuroscience's infamous dead fish experiment (another cautionary tale about correcting for multiple comparisons)

Re: The Bonferroni discussion above, Bonferroni correction is an ugly thing to do to your statistical power, to be avoided at all costs. If the experimental design allows it, I'm a fan of Fisher-protected t-tests. Pretty good rate of type 1 error control on the multiple comparison front, with no cost whatsoever to the power of the pairwise comparisons (the only loss of power for the experiment as a whole is from the possibility of a type 2 error when testing the significance of the multiple R...and if your multiple R is small enough that you care about your power when trying to detect it, your effect sizes are probably small enough that nobody cares).
A_S00

Posts: 3
Joined: Mon Sep 20, 2010 8:16 pm UTC

Re: 0882: "Significant"

Now if the claim was "jelly beans cure acne" and the scientists were "jelly bean manufacturer in-house scienticians" ...

@pforrest: short answer, no. p<0.05 means there is a less than 5% chance the result is due to chance. If you do a single test with an a priori expectation of a result (say, by detailing a physiological pathway by which a pharmaceutical agent would act on a system), then p<0.05 is accepted as 'significantly' likely to be a genuine result (it's an arbitrary figure). If you do 20 tests, with no a priori reason to expect an outcome and post hoc pull out the 'significant' results then you are less justified in claiming statistical significance. It still might be true. It is probably due to chance.
Last edited by VectorZero on Wed Apr 06, 2011 7:08 am UTC, edited 1 time in total.
Van wrote:Fireballs don't lie.

VectorZero

Posts: 466
Joined: Fri Nov 02, 2007 7:22 am UTC
Location: Darwin

Re: 0882: "Significant"

Pfhorrest wrote:So is the joke that the scientists, egged on by the annoying people, tested (all?) 20 flavors of jelly bean to conclusively show "no, there really isn't any link", but then made a typo with their inequality sign on one, and together with the number of other tests run that one (false, due to typo) result becomes much more significant than it would be taken in isolation? I confess I don't completely understand the math at work here, but I would think the first test would have to have demonstrated a link for that to work, i.e. "Hmm, jelly beans appear linked with acne. I wonder if it's just one color? [20 tests, only green shows link] Yep, seems to be just green."

This is an example of multiple comparisons fallacy.

You may want to have a look at this:
http://en.wikipedia.org/wiki/Multiple_comparisons

The problem is that the p-value does NOT tell you that acne and green jelly beans are correlated 95% of times.
It tells you that you have <5% (1/20, the number of panels is well chosen) probability that the correlation in the set of data you gathered is just due to chance.
nico

Posts: 16
Joined: Fri Aug 22, 2008 6:53 am UTC

Re: 0882: "Significant"

Oops, I just skimmed through reading the colours... Then I got to the bottom frame and was like "Wait..." *checks*

cjdrum

Posts: 113
Joined: Sun Dec 12, 2010 4:51 am UTC
Location: BACK

Re: 0882: "Significant"

Qwerty.55 wrote:

I enjoy recursive signatures.

I like your new recursive sig even better than before.

Me too!
"As the size of an Explosion increses the number of social situations it is incapable of resolving approches zero"
-Vaarsuvius, The Order of the Stick
Spoiler:

Qwerty.55 wrote:I like your new recursive sig even better than before.

Posts: 48
Joined: Mon Apr 12, 2010 12:26 am UTC
Location: My used book store near Seattle WA

Re: 0882: "Significant"

ummmm.... ok I am one of those life-scientists that totally doesnt get stats. I really dont see whats wrong with this.
So thinking out loud... yeah if I was doing the experiment I'd put them all onto one graph and tell my software to do an anova (rather than a whole bunch of separate t-tests).... but what if they were all independant experiments and didnt share a common control?

of course if the p values were all close to 0.05 then yeah its dodgy but what if the green one was way below 0.05 and really clear with a really big sample and a huge fold difference? and all the others had really high p-values?

is the point just that you cant rely on p-values alone? arent we missing a whole bunch of other details about the experiment to really judge whether the newspaper article was stupid or not?

these sorts of experiments are done all the time, for example, the effect of various drugs on growth of a cell line. (drug A, no effect, try the next one, drug B, no effect, drug C, ooh effect!) but this is spread over years and it seems silly to consider it all as one huge data set.

teeeaaach meeeee!

edit- will read article, but still, teach me!
Debate politics with a fern. If you lose, refuse to water it.

syko_lozz

Posts: 53
Joined: Fri Jan 11, 2008 5:30 am UTC
Location: Oz

Re: 0882: "Significant"

- Think to self, "I'm a scientist, I've heard of minecraft, what is this?"
soveraign

Posts: 3
Joined: Sun Mar 21, 2010 12:59 am UTC

Re: 0882: "Significant"

thank you for this one!

yesterday I tried to explain the dangers of multiple comparisons to a coworker!

… but the comic strip lacks a proper "table one" and it gives no information if the trial was randomized…
with science like that you'll never make it to the New England Journal
…is listening to the swooshing sound of deadlines passing by
paravatar

Posts: 9
Joined: Sun Oct 10, 2010 8:12 am UTC
Location: Ulm, Germany

Re: 0882: "Significant"

Oh, science!
Current project
Honours Bachelor of Science
Computational Chemistry
University of Waterloo
Sept 2007 - Apr 2012
Scarlet

Posts: 15
Joined: Wed Jan 14, 2009 12:18 am UTC

Re: 0882: "Significant"

haha SMBC was funny when it did this

so long ago
Turing Machine

Posts: 94
Joined: Fri Feb 19, 2010 5:48 am UTC

Re: 0882: "Significant"

Also @pfhorrest: the initial test of all jelly bean colors doesn't have to show a difference. In fact, the correlation of the green jelly beans would have to be very strong indeed to show through the noise of 19 other colors that don't have a correlation with the expected outcome.

@syko_lozz: you're right, there is insufficient information to comment fully on the validity of the study. Yet that didn't stop the journos from claiming the correlation to be proven causation, did it?
Van wrote:Fireballs don't lie.

VectorZero

Posts: 466
Joined: Fri Nov 02, 2007 7:22 am UTC
Location: Darwin

Re: 0882: "Significant"

GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.
This has been roughly my response to everything for a week.
Shameless Website Promotion: Gamma Energy
My new esoteric programming language: GLOBOL
An experiment to mess with Google Search results: HARDCORE PORNOGRAPHY HARDCORE PORNOGRAPHY

darkspork

Posts: 532
Joined: Tue Sep 23, 2008 12:43 am UTC
Location: Land of Trains and Suburbs

Re: 0882: "Significant"

This comic is great. This is how, often old, scientific information is actually eminated. Nice work as ever. Loving the reccursive sig floating about today too!
Philyphil

Posts: 3
Joined: Mon Jan 24, 2011 9:29 am UTC

Re: 0882: "Significant"

This is now my excuse for everything:

"But.. miiiiiinecraft... "

HI JOEE

glasnt

Posts: 518
Joined: Fri Jan 25, 2008 5:18 am UTC
Location: SQUEE!

Re: 0882: "Significant"

Love the comic, but how could you pass up the opportunity for a "Bonferroni"-related punchline in the alt-text??

To the people (including scientists!) who say they don't get it: read it again! The way to think about p-values is to say, for p < .05, "You'd expect to get an effect at least this large just by chance 5% of of the time (if the null hypothesis were true)."

5% of the time = 1 out of 20.

Most of the brilliant, accurate, low-effort heuristics that have been evolutionarily selected into the human mind deal with single cases or very simple proportions. Probability and statistics is something a lot of us struggle to deal with intuitively. However, I think the basics can be grasped by anyone if they're taught well and early enough. In my grand agenda, stats will be a major and required part of both math and science curricula starting in grade school; it's the mathematics of decision-making, which is arguably far more vital in our world than the mathematics of mechanics or architecture, which are more specialty topics.

le_sacre

Posts: 35
Joined: Wed Jun 20, 2007 11:24 pm UTC
Location: stanford, ca

Re: 0882: "Significant"

Good stats commic.
Spoiler:
For those that don't get it. Basically the p value (5% = 1/20) of the green test says there is a one in twenty chance that there is no link on green. However they effectively repeated the test 20 times so if none of the colours are linked you expect on average one false positivie for a link in 20 tests. So their results are in reality broadly as expected for the scenario where no link is made.

To really test the validity of the result they need a combined test that takes all the data into account. Though I would not know exactly what is best in this case.
lrossouw

Posts: 3
Joined: Wed Feb 03, 2010 5:10 am UTC

Re: 0882: "Significant"

I'm no statistician, but I'm glad I knew enough to get this one. I must admit I skimmed past the colours first reading too...

And upon re-reading, noticed that they tested yellow jelly beans twice, in panels 13 and 18. Presumably a mistake on Randall's part, but I suppose there's nothing wrong (scientifically speaking, at least) with repeating the experiment.
Tetrinity

Posts: 3
Joined: Wed Jan 19, 2011 7:42 am UTC

Re: 0882: "Significant"

Diadem wrote:Unfortunately, this is exactly how many scientists (not all, luckily, but too many) do statistics. They really do tests on dozens of variables at once, with several different statistical tests, and then publish the ones that are interesting.

For a single experiment you should always divide your allowed margin of error by the number of variables you are testing for. So if you test n different colours of jelly beans, you must require p < 0.05/n before considering a result statistically significant. But this is often ignored.

....unless you write an article about each experiment separately, in which case you won't have to do it anymore. The comic would still apply, because basically that's what p<0.05 says, that once every 20 times you will get a false positive, so if that's your alpha, dividing it by twenty doesn't make any sense. Even in the case where you perform multiple tests on the same data (of which this is not an example), Bonferroni adjustments are pretty iffy, cf.

Nakagawa, S. (2004). A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral Ecology, 15(6), 1044-1045. doi: 10.1093/beheco/arh107.
spekkeh

Posts: 17
Joined: Wed Nov 18, 2009 11:13 am UTC

Re: 0882: "Significant"

So in other words, 1 out of every 20 studies that use a 95% CI is wrong. The number of the wrong ones that get reported by popular media is overrepresented.

(Yes, I get the fact that the joke is that testing different jelly bean colors is essentially running the same test over and over, thus the expected 1 anomaly in 20, but it's nice to be periodically reminded how big of a number 5% actually is and how arbitrary a choice it is for hypothesis testing.)
graatz

Posts: 87
Joined: Thu Oct 29, 2009 7:24 pm UTC

Re: 0882: "Significant"

NASA Scientist: Minecraft Increases Your IQ

Cranium Talksalot (CNN) -- Worry that your kids play too many video games? You shouldn't. A surprising report by NASA roboticist and computer programmer Randall Munroe shows a trend between employment in intellectually demanding fields and playing games. Statisticians and chemists alike have been shown to play Minecraft, the wildly popular sandbox game by indie developer Mojang, even while at work. Shortly after the report's release, Munroe's website was abuzz with fellow scientists, engineers and academics remarking that they play the game, or if they haven't, greatly anticipate doing so. Munroe is an avid player himself. He admits to having played for days on end while recovering from a dental operation, so absorbed in the complex virtual world that he was able to completely ignore any pain.

He is not alone. Over 5.8 million users have registered on the game and 1.7 million have purchased it. There are approximately 1.3 million scientists in the United States, no small coincidence. The other 400,000? One of them just might be your child.

The next time your child wants to stay up playing Minecraft, maybe you should think twice before sending them to bed. That pixelated pickaxe just might be their ticket to an exciting career in science.
Last edited by eidako on Wed Apr 06, 2011 12:00 pm UTC, edited 1 time in total.
eidako

Posts: 39
Joined: Wed Apr 06, 2011 10:24 am UTC

Re: 0882: "Significant"

chaucer345 wrote:See, what they should have done this is run a single factor ANOVA with the colors as the levels and then performed a Tukey MCP to determine which of the levels were significantly different from each other... Or would that have helped? Maybe I'm thinking of this the wrong way.

This is exactly what I thought after reading the comic. Poor scientists, they would have been done so much more quickly (to get back to Miiiiinecraft), without that mistake about the green jelly beans.
ksquared129

Posts: 1
Joined: Wed Apr 06, 2011 11:12 am UTC

Re: 0882: "Significant"

(post merged with previous)
Last edited by eidako on Wed Apr 06, 2011 12:01 pm UTC, edited 1 time in total.
eidako

Posts: 39
Joined: Wed Apr 06, 2011 10:24 am UTC

Re: 0882: "Significant"

But... We're playing Minecraft (don't ever get this game. It will suck your life away with it's awesomeness)

Another statistics joke. Two in a row. He must really hate statistics this week (the practice thereof)
On 10/10/10, My Little Pony started to appeal to adult males. Ya rly!
Forget the /b/ memes, and go watch the show and see what I mean. After all, the /b/ memes exist because the show itself is so cleverly written. Check it out!

KShrike

Posts: 130
Joined: Wed Dec 15, 2010 5:47 am UTC

Re: 0882: "Significant"

Someone wrote:
"The p-value represents the likelihood of the evidence given the null hypothesis, not the likelihood of the null hypothesis given the evidence."

Which is why it is fortunate we have Bayes' Theorem and related techniques to let us use prior information to ascertain an estimate of the plausibility of the hypothesis given the evidence, which is what people who've kept up with the world have been doing for the past decade or more.

The sad fact is that far more than 5% of papers published against the rather modest standard of p < 0.05 turn out to be reporting on results due to random chance. This comic explains why.

No real experiment exists in isolation, and if your analytical techniques do not provide ways of accounting for prior knowledge, you are using the wrong analytical techniques.
Coming on Midsummer's Day to a Web Browser Near You: http://www.songsofalbion.com

Posts: 137
Joined: Mon Mar 03, 2008 8:57 pm UTC

Re: 0882: "Significant"

darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.
This has been roughly my response to everything for a week.

Haha... me too. Last Thursday too. "What's Minecraft?...... I .... can't.... stop!!!!!!

This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Mark Twain wrote:paraphrased: There are three kinds of fibs: lies, damned lies, and statistics.

Oh, Mark Twain. If only you'd understand. Being an engineer is hard in the moral sense, really. I can prove it too (p < .05).
On 10/10/10, My Little Pony started to appeal to adult males. Ya rly!
Forget the /b/ memes, and go watch the show and see what I mean. After all, the /b/ memes exist because the show itself is so cleverly written. Check it out!

KShrike

Posts: 130
Joined: Wed Dec 15, 2010 5:47 am UTC

Re: 0882: "Significant"

darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.

And then people don't understand what I mean with pandering.

KShrike wrote:This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Why? Randall's practice of namedropping nerd culture items to cause reactions like the one above is extremely common, and this is the second time he has used Minecraft.

Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

SirMustapha

Posts: 1303
Joined: Mon Jul 21, 2008 6:07 pm UTC

Re: 0882: "Significant"

Diadem wrote:For a single experiment you should always divide your allowed margin of error by the number of variables you are testing for. So if you test n different colours of jelly beans, you must require p < 0.05/n before considering a result statistically significant. But this is often ignored.

THE CURSE OF BONFERRONI!!!
jthetzel

Posts: 2
Joined: Wed Apr 06, 2011 1:08 pm UTC

Re: 0882: "Significant"

Having learned some Statistics in my third year of high school, I love this comic and can relate to it. I believe it perfectly shows how journalists and reporters sometimes either twist their reports to make it more interesting or misinterpret them due to not understanding them much.

SirMustapha wrote:
darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.

And then people don't understand what I mean with pandering.

KShrike wrote:This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Why? Randall's practice of namedropping nerd culture items to cause reactions like the one above is extremely common, and this is the second time he has used Minecraft.

Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

I find it baffling that you are complaining about Randall's small mention of Minecraft in this comic and you're not complaining on the topic of the comic itself. But what I find stranger is that you usually don't bash comics containing scientific, mathematical, or programming elements that normal people usually don't know, and when you do, you usually:

a.) complain on small, unimportant stuff in it,
b.) point out that Randall's overall presentation of the comic is sloppy, or
c.) say that Randall only posts obscure nerdy topics to gain more attention and to make himself (and us XKCD fans) feel smarter.

</trollfeeding>
Magnanimous wrote:I'd have a giant I Tetris piece shoved down my throat, after which my neck will flash and disappear.

Balesk Baj, Timeburner

Posts: 76
Joined: Sun Aug 30, 2009 11:07 am UTC
Location: Wherever parallel dimension I'm currently living in...

Re: 0882: "Significant"

SirMustapha wrote:
Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

We all love hearing from perfect trolls
ribbonsofnight

Posts: 8
Joined: Wed Apr 06, 2011 1:26 pm UTC

Re: 0882: "Significant"

I loved reading this comic! It was my 30 second break from minecraft. Must destroy creepers.....

TheDoctor

Posts: 11
Joined: Mon Apr 04, 2011 5:15 pm UTC

Re: 0882: "Significant"

Hooray statistics!

I'm a poisson distribution!

I'm a poisson distribution!

I'm a poisson distribution!
Cecilff2

Posts: 114
Joined: Fri Oct 03, 2008 9:10 pm UTC

Re: 0882: "Significant"

Cecilff2 wrote:Hooray statistics!

I'm a poisson distribution!

I'm a poisson distribution!

I'm a poisson distribution!

Hmmm... given the small sample size, I would have to conclude it's still fairly likely that you're not a poisson distribution.
"If it were up to the copyright lobby, owning a pen would be punishable by fines." ---Arancaytar

savanik

Posts: 344
Joined: Fri Jan 29, 2010 6:10 am UTC

Re: 0882: "Significant"

I'm totally a poisson distribution.
Cecilff2

Posts: 114
Joined: Fri Oct 03, 2008 9:10 pm UTC

Re: 0882: "Significant"

Wait...
jozwa

Posts: 70
Joined: Fri Sep 11, 2009 3:16 pm UTC
Location: Finland

Re: 0882: "Significant"

NEWS: Scientist find correlation between acne and minecraft!
Read My Book. Cost less than coffee. Will probabaly keep you awake longer.

neoliminal

Posts: 610
Joined: Wed Feb 18, 2009 6:39 pm UTC

Re: 0882: "Significant"

Did the comic change between last night and this morning? I didn't see green jelly beans referenced until the end.
“When we remember we are all mad, the mysteries disappear and life stands explained.” - Mark Twain

Angua
Don't call her Delphine

Posts: 3499
Joined: Tue Sep 16, 2008 12:42 pm UTC
Location: UK/St. Kitts and Nevis Occasionally, I migrate to the US for a bit

Re: 0882: "Significant"

I was JUST eating jelly beans not 2 minutes ago

So, the public's attempts to simplify the findings of scientific studies or to use scientific studies to "prove" their hunches, gets in the way of true science?

I remember some news report from a couple years ago saying that studies showed blue M&M's could cure spinal injuries, and further details showed that the blue dye on them had been linked to improved spinal health or something. This reminds me a lot of that, I think they're in the same boat.
And, I think personally I would put the link between flu vaccines and autism in the same boat as well.

ribbonsofnight wrote:
SirMustapha wrote:
Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

We all love hearing from perfect trolls

Welcome to the fora.
He's not trolling, that really is his opinion. Check the past 300+ comic threads, it's mostly similar reactions.
Last edited by SpringLoaded12 on Wed Apr 06, 2011 3:05 pm UTC, edited 1 time in total.
"It's easy to forget what a sin is in the middle of a battlefield." "Opposite over hypotenuse, dipshit."