Here’s a comment on this post originally put on Facebook, by Daniel Handwerker (copied here with permission). See next comment for my reply…
“I disagree.
You say the goal of science is to understand what can be "accepted as fully established knowledge" You do not claim or explain why that goal has to be reached to publish a peer reviewed article. I'd go father and say, if the criteria for publication is that the reviewers thinks a finding should be accepted as established knowledge, then nothing would ever get through peer review. A replication might increase the likelihood a finding is reliable but it wouldn't remotely cross the high bar you're setting.
Replication, particularly the split half replications you're using in examples, only focus on two causes of irreproducibility: Statistical power & methodological clarity/processing consistency. It might help in some situations, but won't fundamentally change the core issues underlying seeking truths about a complex system.
This is still my favorite piece on this topic: https://drugmonkey.scientopia.org/2014/07/08/the-most-replicated-finding-in-drug-abuse-science/ The core message is that publishing and sharing inconsistent findings from the fundamentally same study design is what advanced scientific understanding. This example isn't a replication failure, it's advancing science through conversations on why things aren't replicating.
Fundamentally, the question is what is a peer reviewed scientific publication? If we treat scientific publications as markers of scientific truth, then we have a replication crisis every time the findings in a specific paper aren't replicated. If we treat scientific publications as part of a conversation that advances science, then the purpose of each paper should be to clearly explain what they did (so that others replicating is possible), explain why their results are plausible (i.e. statistics & methods are sufficiently robust to support claims), and limitations on their interpretations of results. Some papers might present more support for their interpretation and some might present an interesting observation paired with clear discussion of interpretability limits. Both are critical parts of scientific discourse and advancement.
There are two core trouble with the goal of making every publication reproducible. 1. It's impossible and claiming it is possible is unscientific. 2. It amplifies the file drawer effect where potentially valuable non-replications never enter scientific discourse.”
Here’s a comment on this post originally put on Facebook, by Daniel Handwerker (copied here with permission). See next comment for my reply…
“I disagree.
You say the goal of science is to understand what can be "accepted as fully established knowledge" You do not claim or explain why that goal has to be reached to publish a peer reviewed article. I'd go father and say, if the criteria for publication is that the reviewers thinks a finding should be accepted as established knowledge, then nothing would ever get through peer review. A replication might increase the likelihood a finding is reliable but it wouldn't remotely cross the high bar you're setting.
Replication, particularly the split half replications you're using in examples, only focus on two causes of irreproducibility: Statistical power & methodological clarity/processing consistency. It might help in some situations, but won't fundamentally change the core issues underlying seeking truths about a complex system.
This is still my favorite piece on this topic: https://drugmonkey.scientopia.org/2014/07/08/the-most-replicated-finding-in-drug-abuse-science/ The core message is that publishing and sharing inconsistent findings from the fundamentally same study design is what advanced scientific understanding. This example isn't a replication failure, it's advancing science through conversations on why things aren't replicating.
Fundamentally, the question is what is a peer reviewed scientific publication? If we treat scientific publications as markers of scientific truth, then we have a replication crisis every time the findings in a specific paper aren't replicated. If we treat scientific publications as part of a conversation that advances science, then the purpose of each paper should be to clearly explain what they did (so that others replicating is possible), explain why their results are plausible (i.e. statistics & methods are sufficiently robust to support claims), and limitations on their interpretations of results. Some papers might present more support for their interpretation and some might present an interesting observation paired with clear discussion of interpretability limits. Both are critical parts of scientific discourse and advancement.
There are two core trouble with the goal of making every publication reproducible. 1. It's impossible and claiming it is possible is unscientific. 2. It amplifies the file drawer effect where potentially valuable non-replications never enter scientific discourse.”