A good friend opened a conversation with me the other day: “Don’t bother taking Vitamin C when you start feeling sick because a recent study proves it has no effect.” This is the kind of suggestion that literally floods my inbox. Comments like “there is no proof that acupuncture works” and headlines that read “Kombucha—just another sugar water?” are a part of my daily life as a nutritionist, and I’m sure for you as well if you condone anything holistic. The major problem with all these statements is that they’re often wrong.
There are over 10,000 scientific studies published in the US every single month, making it basically impossible to keep up with what’s current. This amount of information being thrown at us, especially for those that try to keep up, forces us to base conclusions on a language we don’t really understand (but think we do.) We are well-intentioned humans just trying to do the right thing to be healthy, but because most of us are not statisticians, our “take-away’s” from the studies are actually assumptions, and therefore…wrong.
Today I’d like to attempt to very simply (re)introduce some facts about the scientific method, so you can begin to arm yourself with the truth as you combat the daily whirlwind of “you should’s” and “you should not’s.”
1) When a study does NOT show a correlation between 2 things, it is 100% INCORRECT to conclude that those 2 things are not correlated. All it means when no correlation is seen is that this particular trial was unable to find one. Many studies, especially if they’re not set up appropriately, will fail to show a difference in populations when in fact there is one. So, whenever you read a headline that suggests “no link,” make a note to yourself that this is not a correct conclusion. Just because you don’t see stars in the sky doesn’t mean there aren’t any.
2) When a GOOD study does show a significant difference between 2 populations, your conclusion here can be a bit more validated. If you’re starting with 2 very similar groups of people and your results over time show that a certain behavior separated one group from the other, there’s probably truth to saying that behavior will likely beget that result. The larger and more significant the difference, the more likely this is to be true.
3) Saying something showed a “significant” change in a study does NOT mean the same thing as a big change. When something is “significantly” deemed different in science, all we are saying is that more than 95% of the time you can expect to see a difference, even if the difference is small. You will see this notated on a study as a “p-value” (if a result is deemed true 95% of the time, its p-value will be 0.05.) If it is found to be true only 94% of the time (p-value 0.06), it is not deemed significant, just the same as if it’s only found to be true 10% of the time (p-value 0.9.) Obviously you can see that 10% and 94% are vastly different, and that the number 95% is somewhat arbitrary, so I hope you understand from this that “significance” in a study needs to be further looked into before you make assumptions.
4) The original scientific method suggests that in order to show something to be true, you must prove all of the following steps to be true…Almost no study I read actually does this:
- The study must start with at least two similar populations large in size, behaviors, demographics, and characteristics.
- The introduced behavior demonstrates a significant difference between the 2 groups.
- Taking away that behavior alleviates the difference in the 2 groups (they are the same again).
- Re-introducing the behavior demonstrates the same significant difference as it did before.
- Removing the behavior yet again alleviates the difference in the 2 populations again.
5) There are many different types of studies, and what we take from each kind of study should be different. Here are the most common types (definitely NOT an exhaustive list):
- Prospective studies: these start with at least 2 similar groups (think apples to apples)—the first asked to take on one behavior, the second to take on another behavior, and after a certain amount of time has gone by, the scientists measure the difference in the groups. This is how most drug trials are set up. When done correctly, these often expensive studies are the most reliable sources of science, especially if they demonstrate a large and significant difference between the groups at the end of the study. I emphasize “correctly” here because many many many prospective studies fail to set up their trials with the best integrity, tarnishing their results.
- Retrospective studies: this is when you take data from the past to find correlations. Let’s say you wanted to see whether patients with hypertension AND diabetes are more likely to have a heart attack than those with just hypertension—you might look at past hospital records of heart attack patients and note which had what conditions and base conclusions on what you’ve found. Because we are looking at the past, and we’re not really sure we are comparing apples to apples, you can probably imagine that there are limitations to any conclusions you make here. Patterns you find in a retrospective study are not conclusive; they are suggestions upon which we might want to build a prospective study around.
- Meta-analysis: this is when someone compiles information from many different past completed studies on a particular subject and essentially reads it as one. Take colds and vitamin C as an example: the data from hundreds of smaller studies have been aggregated together to make one bigger study from which to make conclusions. This kind of study can be very dangerous from which to make any conclusions, because we are most certainly NOT comparing apples to apples (we’re talking about entirely different studies being looked at as one). But as with retrospective analyses, if you see a correlation, it may warrant future research.
- Population or epidemiological studies: this is a type of research often used in public health to help suggest hypotheses for future studies. It’s used when you look at something like global heart attack rates, and you see that the US has much higher rates than Japan. You might also look at what behaviors the 2 populations engage in to see if there’s a difference. Your population/epi study might show that Japanese eat more fish, which could help fuel your hypothesis that fish-eating may ward off heart disease, but you CANNOT conclude this until you study it prospectively.
- Reviews: reviews should look at ALL the good studies done on a subject and summarize the results. These can be great if you’re trying to make a conclusion for yourself about a behavior. But, again, if the studies its reviewing aren’t any good, neither is it.
In conclusion, science isn’t perfect. It has limitations, just like everything else, and we should always take anything we read about science with a grain of salt. I hope after reading this blog you better understand (even a little bit more) what to be skeptical of, what you should and shouldn’t assume, and what to do with the information you read.
Remember, just because a study hasn’t been done, or just because a study didn’t show a correlation, doesn’t mean you should wait to introduce that behavior (or remove that behavior) if you think it can help or hurt you. This is where your intelligence must come in to play! For example, even though there has yet to be robust proof around organic being “healthier” for you…do YOU think eating pesticide-laden veggies and fruits will have an impact on your health? How about with kombucha? Even though a prestigious scientific journal hasn’t published whether it actually improves your energy or boosts your immunity, but when YOU drink it YOU feel the difference (and so do thousands of others), what conclusion will YOU make here? To sum it up, I think science is important, but we can’ t loose our own voice to the limitations in literature. Sometimes, you just have to FOLLOW YOUR GUT.
Daina Slekys, MS, MPH is a Nutritionist and co-founder of Health-Ade kombucha, in Los Angeles, CA. Email Daina your health questions she’ll be happy to help! firstname.lastname@example.org
Health-Ade kombucha is available for purchase at:
- Calabasas, CA Farmers’ Market every Saturday from 8 am - 1 pm
- Brentwood, CA Farmers’ Market every Sunday from 9 am - 2:30 pm
- Malibu, CA Farmers’ Market every Sunday from 10 am - 3 pm
- Starting Aug 18 at the NEW BEVERLY-GLEN Farmers’ Market, every Saturday from 9-2 pm.
- Every day at Cake Bakeshop in Manhattan Beach, CA.
- Every day at The Rabbit Hole in Agoura Hills, CA
- Every day at Back Door Bakery in Sunland, CA
- Every day at Thunderbolt Spiritual Books in Santa Monica, CA