"It works for me"… doesn't

Here, I fight the power of anecdotes with… an anecdote? Ok, it sounds silly, but bear with me…

Several years ago, I found myself severely addicted to caffeine. If I didn't have my coffee, soda, or tea by 10am or so, I'd get headaches and generally feel terrible. One weekend I bought some bottled tea on sale. Sunday morning rolls around and I cracked open a bottle or two, and proceeded with my day: my caffeine fix was acquired. Monday morning comes along and I open another bottle and the day works out about the same, except that later in the day I absentmindedly picked up the bottle and started reading the label. Huh: it is completely caffeine-free.

Two theories:

1) Placebo: I thought it had caffeine, so it prevented my headaches and other symptoms of caffeine deprivation.

2) Nocebo: My previous headaches and other symptoms were psychosomatic, and were never caused by "caffeine withdrawal" in the first place.

I find the second one a bit more compelling because it seems to me it'd be easier for my body to generate "fake" symptoms than cure real ones, but that's far from certain. I've not tried to narrow it down because that experiment would be too involved and have little payoff other than satisfying my curiosity.

Too involved? What? It seems like it'd be easy to determine if I was addicted to caffeine, right? I mean, I could just avoid caffeine for a few days and record how I felt. But the anecdote from above shows the problem with that: if I know the drink was or was not caffeinated, it could greatly effect how I feel and/or how I interpret those feelings.

To really figure it out, I'd need a much more complicated experiment:

• I'd need at least one other person (or a machine) to randomize whether I received caffeine or not that day, and keep it a secret from me.

• Ideally, this other person should also not be aware if I received caffeine or not on any particular day (ie double blind).

• I'd need to do this for a while, not just a day or a week, but several weeks at least.

• I'd need to record whether I had headaches or other symptoms, and rate the severity.

• After the experiment, I'd need to figure out if there was a strong correlation between days I felt bad and days I didn't have caffeine.

The key to the experiment would be that neither I nor the accomplice knows which days I receive and don't receive caffeine, otherwise I couldn't rule out either placebo or nocebo effects. If I know I got caffeine, I could subconsciously say a headache wasn't that bad, or if I knew I hadn't received caffeine, I might think some minor feeling was worse than it really was. Another key feature of the experiment would be multiple trials: many days of caffeine versus non, otherwise I couldn't know if a headache I had was just a random headache or as caused by caffeine withdrawal. Only with many instances of headaches after not having caffeine could I say with any degree of certainty that the headaches were caused by the lack of caffeine and not just chance.

It'd be an interesting experiment for sure, but I'm too lazy for all that. But the main value of this is as a thought experiment to show why I can't simply take someone's word when they say "Well, you should try it! It works for me!" Whatever diet, herb, or practice "works for you" might not actually be "working for you". You might just think it is helping when in reality it is irrelevant, or only part of the story.

So when someone says taking a supplement helps them with their depression or digestion or whatnot, it's not exactly that I disbelieve them, it's more that I don't have enough information to decide if it's actually helping or a placebo, nocebo, or some other factor is affecting things.

If you say that avoiding dairy, MSG, fruit, wheat, cooked food, nuts, beans, beef, artificial colors, corn, fish, conventionally grown vegetables, sugar, medicines, high-fructose corn syrup, etc etc etc makes you feel better, helped your cholesterol, keeps you regular, cures your ADHD, eases your arthritis, prevents colds, helps you lose weight, etc, etc, etc… it's the same deal: you might be correct, partially correct, or it might not have anything to do with it. I don't believe you are lying, and I don't even necessarily believe you are wrong, it's merely that saying "works for me" doesn't provide me enough evidence to say that you're right.