Health studies and you

The way many health studies are done makes them not very useful. The way they are reported on is even worse. You see this nearly every day, then you never hear about it again. Ok, you will hear about those studies again on the Dr Oz show because he blows, but generally the study behind the news headline "It's official: Bacon is good for you!" will not hold up for long. That's partially because a lot of these studies depend on self-reported data.

So, if you wanted to find out if, for example, consumption of bacon increased your risk of heart disease, here's what you do:

• Find a bunch of people who have heart disease.
• Find a bunch of people who do not have heart disease.
• Ask both groups how much bacon, on average, they ate per day in the past 20 years.
• Ask both groups what else they ate for the past 20 years.
• Also ask them if they smoked, drank, or did drugs, and how often, in the past 20 years.
• Ask them to rate, on a scale from 1-5, how healthy they are now.
• Finally, ask them how much they weigh and how much they exercised in the past 20 years.
• Publish results!

Several problems.

People are terrible at remembering stuff like that.

I know mostly what I ate in the past few days, but before that? How often did we order pizza last year?

People tend to underestimate how much they eat.

I probably eat only about 2000 calories a day. Yup, I'm just big boned.

People overestimate how much they exercise.

I get plenty of exercise. My man-boobs are genetic!

If you ask someone with cardiovascular disease how much bacon they ate, several things could occur:

1. They will accurately report that they ate bacon X number of times per week.
2. Suspecting there might be a connection, they will be defensive (either consciously or unconsciously) and rationalize away how much they really ate and report a far lower number.
3. Suspecting there might be a connection, they will fall into self-recrimination and overestimate how much they ate.

Which one happens most? Do the latter two happen frequently? Do they cancel each other out? Do I like asking myself questions?

I understand that following people around for decades recording what they eat is not feasible. But if a study relies only on self-reported data, that is a big red flag for me.

Perhaps the kind of people who eat a ton of bacon also tend to be the kind of people who eat buckets of buttered biscuits or lie about exercise?

This is another big part of the problem: confounding factors. Does the bacon cause the heart disease, or is it the white-bread biscuits we forgot to ask about? Or, more basically, suppose the heart disease is simply genetic, and it is tied to a predisposition to like the taste of bacon? Researchers try to control for these issues, and sometimes they do a very thorough job. But it's an incredibly difficult task, and takes a tremendous amount of knowledge, thought, and planning to truly eliminate all the factors which might confound your data.

Finally, the kind of people who avoid bacon are more likely to be those who do other proven healthy things too.

I call this the "health nut effect", though I'm sure it has a technical name. Think about it. If you are a self-described "health nut", you probably limit your fatty fried meats (ie bacon), but you also exercise, eat lots of vegetables, don't smoke, drink little alcohol, avoid processed foods, avoid sugary drinks and desserts, etc. Many of these things the researchers are already asking about, but you do them for real as opposed to those who report that they "exercise" but really mean they play golf once a week or walk to the mailbox every other day (I bet that counts as a 30 minute walk, right?). There are probably also lots of other small things a "health nut" does which might have small effects, but are hard to measure.

So, I should ignore studies, then?

None of this is to say you should ignore all health studies, or that we know nothing about health. We actually know quite a lot about health because of studies like these. But we know about them because the studies were replicated by other groups, then yet other groups performed different studies and experiments which lend support to the original studies, and then more research is done which fills in other pieces and supports the earlier studies, and so we end up with a pretty good idea of what is and isn't true.

Also, many times these studies are not intended to establish a fact all by themselves, but to suss out interesting things to research further. It's the media hype around a new study that's usually to blame.

The point of this is that you should take small studies with a grain of salt, especially if they use self-reported data. But also that when many replicated studies and other data point to the same conclusion, you can pay attention. So when Yahoo "Health News" (or an alternative-health guru or an advertisement) reports on a new study which says eating some tropical berry or another will reduce your risk of cancer, you can probably safely not run out and buy the expensive berries or extracts; However, you can still know that decades of research has shown that eating a lot of fruits and vegetables – and maybe not quite so much bacon – is probably good for you.