Before comments on content per se I'd argue that you should call this 'Evidence of Article Quality Index' my main point being that an article is not necessarily the result of 'a' study, sometimes a study have to be sliced sometimes they are sliced but should not (maybe another criteria to include ... )
OK let's get to it ... I agree with two out of the three premises (1) tool against science misinformation especially for general public and media and (2) this could improve the incentive structure for scientists by giving them quantifiable credit for the quality of their work. I can see a problem related to scientific literature curation: no one reads low EAQI articles, yet these can contain completely new and revolutionary ideas? then what we miss out of that .. or should those get a boost? hard to judge.
A word on replication as criteria: I'm guessing you are talking about direct replication, but how should one do it? if significant result in study one, power is 100% (prob to have a + results --https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf) so should we redo with same sample size anyway? or a priori collect and split data? establishing criteria will be hard, but not impossible (see generics)
A word on 'fully established' --> in my mind it has to do with cumulative evidence from conceptual replication ; even so a single study with many substudies as controls for many possible biases or explanations for instance (i.e. systematic replications) seems as credible or maybe more than a study with a direct replication (again establishing those criteria).
Last but not least, we need to experiment and provide evidence ... how many rater are needed to have a stable score? :-)
Before comments on content per se I'd argue that you should call this 'Evidence of Article Quality Index' my main point being that an article is not necessarily the result of 'a' study, sometimes a study have to be sliced sometimes they are sliced but should not (maybe another criteria to include ... )
OK let's get to it ... I agree with two out of the three premises (1) tool against science misinformation especially for general public and media and (2) this could improve the incentive structure for scientists by giving them quantifiable credit for the quality of their work. I can see a problem related to scientific literature curation: no one reads low EAQI articles, yet these can contain completely new and revolutionary ideas? then what we miss out of that .. or should those get a boost? hard to judge.
A word on replication as criteria: I'm guessing you are talking about direct replication, but how should one do it? if significant result in study one, power is 100% (prob to have a + results --https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf) so should we redo with same sample size anyway? or a priori collect and split data? establishing criteria will be hard, but not impossible (see generics)
A word on 'fully established' --> in my mind it has to do with cumulative evidence from conceptual replication ; even so a single study with many substudies as controls for many possible biases or explanations for instance (i.e. systematic replications) seems as credible or maybe more than a study with a direct replication (again establishing those criteria).
Last but not least, we need to experiment and provide evidence ... how many rater are needed to have a stable score? :-)
Cyril