"To good to be true" - Criticism of scientific studies grows

It is almost ironic, the other day I posted this article Reason Why I'm Skeptical of Skepticism which criticized over reliance on many study conclusions without actual supporting or valid data within the studies.  Now, just a few days later, there is a new published "study of studies" which reinforces the idea of being skeptical of scientific study conclusions, Excess Success for Psychology Articles in the Journal Science.  This study exposes that many studies in Psychology have issues, where the declared conclusions are simply "to good to be true" based on the strength of the data.  The inference being that there may be a general problem with all fields of science.

"Not every experiment is methodologically sound, and some experiments (even if methodologically sound) do not clarify the status of a theoretical idea. There is little reason to publish such experimental results, whether they are statistically significant or not. Unfortunately, in day-to-day scientific practice it is quite easy to interpret an unsuccessful outcome as being irrelevant to the theory or as being methodologically flawed and therefore not worth reporting."

In other words, data is cherry-picked in support of the theory rather than attempting to take contrary results into account.  This is basically throwing out the Scientific Method when it doesn't result in data this supports a theory.  In other cases, data collection is just too imprecise to form a suitable theory.  Kind of like garage-in-garbage-out.

I have a feeling a growing criticism of the current system is going to force changes into the process of study publishing and utilization.

Old Perry Mason show becomes reality for Cops

What's up with all these Perry Mason style Grand Jury hearings being used to acquit police officers?  The latest case in New York is just nuts, where the accused officer was even allowed to testify, just like in the old Perry Mason show.  How are these Prosecutors allowing any defense at all during the hearing, let alone 2 hours of testimony by the accused without cross-examination (as in the Ferguson case)?  The cops are not being acquitted because of following the process. They are being acquitted because the Prosecutors are gaming the system to support the accused officers.  If a regular citizen is pulled up in front of a grand jury, they cannot present a case, they cannot challenge the presented evidence and they certainly don't get to testify for two hours without cross-examination.  Looks like it's time for other accused people to demand equal protection under the law, for equal treatment to these officers are getting during Grand Jury trails.

Reason why I'm skeptical of Skepticism

There is a plague on modern science and Skepticism.  That plague is over reliance upon research reports.  Research report is a gathering of pre-existing data (the "re" in "research"), repurposed to find patterns.  The problem is that many research reports are created to find correlations in support of predetermined conclusions (assumptions).  Research reports are often formatted as scientific studies and published in science journals along side scientific studies that use the Scientific Method.

What is the Scientific Method?  First, wonder about a phenomenon and ask a question.  Conduct research on that question.  Then, construct a hypothesis (a proposed explanation for the phenomenon). A hypothesis must be formed in such a way to be tested for being  false (falsifiability). The subsequent test must be done in such a way as to try to disprove your hypothesis.  Only after all of this can you analyze your data and form a conclusion from that data.  The final step is to share your results for others to review and check, often in the form of their own studies.  For a study to have value, its results must be replicated by others.¹

The Scientific Method is valuable because it helps eliminate incorrect explanations for a phenomenon.  For example, say someone as a hypothesis that lemons are yellow because someone paints them with yellow paint.  You can test this in any number of ways.  You can go to an orchard and watch the lemons grow, changing color as they mature.  You can buy a lemon at a store and cut it, looking for a layer of paint.  You can peel the lemon and send the skin to a lab to search for significant amounts of paint.  Any of these tests would prove the hypothesis false.  Someone else could create a new hypothesis about why lemons are yellow, knowing that the cause is not paint.

The limitation of an unscientific research report is that it only requires you to visit a grocery store to see that the lemons are yellow in order to confirm your assumption that they've been painted yellow by a person.

Research reports often lack several crucial steps compared to Scientific Method.  First, the purpose of a research report is often to collect data in support of a pre-existing assumption.  An assumption is different than a hypothesis because an assumption is not a proposed explanation for a phenomenon; rather an assumption is that there is a phenomenon ("someone is painting those lemons yellow", instead of asking "why are lemons yellow?").  Second, no test is performed in support of this assumption.  Data is unscientifically collected from different studies, reports and other sources rather than being the results of a direct test.  There is certainly no falsifiability test.  Third, the results of the collected data are often correlated to the original assumption rather than standing on their own within the conclusion.

What's wrong with correlation between data sets?  Correlation is an indicator, but it is not a identifier. The common phrase is "correlation is not causation."  It is extremely easy to correlate unrelated things.  There is a website dedicated to just this.  This is why the Scientific Method requires a falsifiability test.  It eliminates the reliance on correlation.

Now, this is not to say all research reports are bad.  A research report that uses the Scientific Method to analyze data in a way that can be demonstrated with falsifiability does have value.  But, it is very hard tell good reports apart from bad reports using quick Twitter or reddit title links.  You have to read the report to know if it has value or if it is pseudoscience.  You have to read the whole report because the report's title and conclusions often do not even match-up the data within the report.

This is where I run into issues with Skepticism.  Skepticism tends consider any conclusions of a published research report (in support of presumed consensus) to be the same as studies using the Scientific Method.  Opinions regarding concepts outside the presumed consensus are immediately rejected (even if they are published) without regard to the quality of the report or study.  I cannot count how many times I've read a promoted research report, only to find that the evidence in the report is based correlated cherry-picked data.  Blind acceptance of research report conclusions is a big problem with Skepticism, especially as it grows in popularity among Atheists and other non-religious folks.  I've seen seen sources such as "skeptic" magazines that site unscientific research reports as though they are undeniable fact.  It is a problem being exacerbated by the ease with which (mis)information flows on the Internet through various social media and various other media outlets.

Other reading

In researching this topic, I found a very interesting "study of studies" about the flaws in most studies, Why most published research findings are false. Nothing in my article here is based on this study, so take this as a completely different source.  It is worth the read, and more fuel for the fire to be skeptical, not just of Skepticism, but of anyone trying to use a "study" to promote a notion.

Bits of wisdom: Leftovers

Leftovers Laws #12: If you remember your leftovers, they are still good. If you've forgotten about them, they've gone bad.

Dead Plants

Winter is here,
Oh my dear.
My plants are dead
"I'll see you again,
next year."

The shooting star shower [Dream]

I arrived a some tower apartment building with a large grassy field to the right, forest behind, parking lot to the left and a road along the front.  Beyond the grassy field was an industrial park.  I had just arrived, meeting some people near the apartment building.  It was very late evening.  The sky wasn't completely black just yet.  The area was lit by street lights, and the color of the plants lit up by those lights where vivid greens.  As I was just hanging around, there was a bright light that flickered from behind me.  I turning around just in time to see a second burst high in the sky in the direction of the forest.  There was a large meteor streaking downward at a very steep angle.  It was bright, with a long multiple trails and pieces breaking away.  Then there was a large flash as the meteor exploded.  Moments later, I heard a big boom. Then pieces of the meteor started hitting the ground.  Everyone ran for cover in the closest building.  No one was hit.  But, as soon as it began, it was over.  I went out to to find meteorites all over the ground.  As soon as they cooled enough to grab, several people went for them.  I was able to get a few good sized pieces.  The odd thing about them is that they where all smooth, as though they'd been on a riverbed for months.  Their color was the red side of brown, with dark greyish speckles.  I marveled as to why these rocks from the sky would be so smooth after hurtling through space and slamming in the atmosphere with an explosion.