"Works for me", part 2: almost anecdotes

My second attempt at fighting anecdotes… with anecdotes….

Last week, I felt a cold coming on. I felt tired, and my nose and sinuses started getting gooped up. So, to ward it off, I took a bunch of vitamin C. And guess what, my symptoms went away after a day. Victory!

This is a completely true story…

… except for the part where I took a lot of vitamin C. In reality, I took echinacea. No, sorry, it was vitamin A. No, no, it was herbs. Definitely herbs. Oh wait…

Well, the truth is, I took no remedies. I did nothing unusual at all, in fact. The symptoms just went away by themselves. I thought I was getting a cold, but it never fully materialized. So what does my anecdote mean?

It means anecdotes about something that "worked for me" can't show that it actually does. Sometimes, when you think you are getting a cold, you actually aren't. Sometimes a "cold" can cure itself, with or without a remedy.

But, if I had taken vitamin C, I might believe that it cured my cold. If I had taken echinacea, that would have been the fix. If I had loaded up on rosemary, vitamin concoctions, or Airborne, one of those would have gotten the credit. In reality, though, those things might have done nothing positive: my "cold" could have just never happened, regardless.

I am not claiming with 100% certainty that those (or any other) remedies don't work. I am claiming is that if someone takes a "remedy" when they think they are getting a cold, and then a cold never happens, that by itself is no evidence at all. The "cold" might not have happened anyway.

Sometimes, it's pretty easy to show that something works. If I want to protect myself from bullets, it's fairly easy to show that a steel plate "works for me!" It's straightforward to test if bullets are stopped by a steel plate because the results are dramatic. But if I want to protect myself from getting a cold, it's a bit trickier, a bit more subtle. "I didn't get a cold" doesn't prove much, because lots of people don't get colds all the time. So one person saying "I didn't get a cold" is really not worth much. Even 1000 people saying "I didn't get a cold after taking echinacea" doesn't show much, because millions of people also didn't get colds after not taking echinacea, as well. You have nothing to compare to. Did those 1000 people have fewer colds than the people who didn't take the remedy? Without that statistic, you have nothing.

That's one reason why skeptics want large, blinded studies instead of anecdotes. Instead of a few true believers saying "It works for me", you take hundreds of people, give half the "remedy" and half get a sugar pill, and no one knows if they got the real thing. Then see who gets colds and who doesn't. Or who gets milder colds and who gets worse colds. Etc.

But without that – without something to compare your results to – it's just an anecdote.