When researchers look at the data they collect and run statistical tests, the main thing they are looking for is this: statistical validation that the results they obtained were highly unlikely to have popped into existence by chance.
Is what you’re seeing reliable and valid?
That is to say, scientific research is all about demonstrating relationships that are reliable and valid. Reliable is to say that if the same experiment was done again, the results would come out the same. And valid means that the effect is a “true” effect. In other words, if your experiment is showing that eating carrots turns your skin orange, you want to make sure that you aren’t taking your pictures under an orange tinted light.
Science and politics can mean a great deal of bias
Here’s an article from the Royal Society that demonstrates a huge problem facing medical science today. The political nature of doing real research has invaded to the extent that bias is in charge of everything. Bias against publishing null results (people whose research shows there was no effect of treatment usually don’t get their research published) means that we don’t hear about a lot of treatments that don’t work. This bias has been a noted problem for many decades. This article points out bias that rewards new, novel or surprising findings and shows that essentially even honest and reliable scientists (lots of data has shown that there is a discouragingly high rate of dishonesty in academic science) will end up doing “bad science” if only their novel findings are published.
Politics Make a Difference in What You’re Seeing
Medical science is still a fairly young field and there is much that is left uninvestigated. I remember clearly being told by various bigwigs in the medical publishing world that I couldn’t publish findings because they were not politically correct. In another case my publishing efforts were shut down because a particular pharmaceutical company (that supported some of the research done by the group I worked with) would be unhappy with my results (they basically showed that a behavioral intervention was superior to a pharmaceutical intervention).
The key here is replication. A valid finding should be a finding that can be repeated. Basic science isn’t showy and sexy. It’s basic and it can get boring. The scientific method isn’t the problem. The politics are what is the problem.