Evaluating studies

Suppose you're on Facebook, minding your own business, when somebody posts a link to an article. This article is about a study, or, more accurately, the article uses the study to support its argument.

This could be any kind of study: a health study, a sociology study, a global warming study, a child-rearing study. Whatever. The real question is: should you believe the study?

Wonder no more. I've whipped up a handy flowchart to help you decide when you should accept the validity of a study used in an argument!

This shouldn't be this way, but far too often it is. No, not just the Republicans. No, not just the Democrats. No, not just the hippie health nuts. No, not just the medical establishment. No, not just the pseudoscientists. No, not just the scientists. Everyone. Not just your stupid opponents: even people you agree with. You. Me. We all do this.

It's not that we should unquestioningly accept studies we don't agree with, it's that we shouldn't short-circuit into unquestioningly accepting any study which supports our point.

We rightfully pick apart our opponents' supporting studies, critically examine the methodology, look for rebuttals, demand larger sample sizes, require replications, etc, but not our own. We're doing it wrong.

I'm sure there's a name for this failure in rationality, but I don't know it. "Selective scientific rigor" sounds vaguely familiar/correct, but doesn't bring up any relevant Google hits. Whatever its name, it's not a good way to figure out the truth.

If you'd like to find examples of this in action, log into Facebook and look at the comments on any article/study relating to health, parenting, or politics. That should illuminate both the "unquestioning acceptance" and "demands for selective perfect scientific rigor".

I've (extremely slowly) started trying not do this. It's incredibly hard, and I've had to give up some of my treasured beliefs, or at least the treasured arguments for my beliefs.

I fail at it more often than not, but at least it's caused me to fail more gracefully, because instead of posting the article supporting my belief, I think about it for a bit, see that the argument is full of holes which would take a long time to fill, and end up not posting the link in the first place.

Also, when one of my friends posts an article which supports one of my opinions, I'll take a minute before clicking "like" to think about the article. Many times I decide they should stop agreeing with me like that, and let it go.

Sometimes, a little less zeal is a good thing.