Re: Citation analysis of author-choice OA journals

["Stevan Harnad" <amsciforum@gmail.com>]

On Thu, Aug 28, 2008 at 10:15 PM, Phil Davis <pmd8 -- 
cornell.edu> wrote:

> Stevan,
>
> Because of the sheer number of articles published by PNAS, 
> tracking the performance of each article was considered too 
> onerous. As a result, I tracked the first and last 6-month 
> cohort of articles (June-Dec 2004; and June-Dec 2006). By 
> choosing the first and last cohort, I could estimate a temporal 
> trend in the data. Please remember that PNAS was only one of 
> the 11 journals analyzed in this study, and that Gunter 
> Eysenbach's study (PLoS Biology, 2006) analyzed only a 6-month 
> cohort in one journal (PNAS, June-Dec, 2006). Granted, a full 
> dataset from PNAS would have been ideal, and I encourage you to 
> gather and share the intervening years if you feel that the 
> missing data points would change significantly the results of 
> this study. My sense is that they won't, but will challenge you 
> to prove me wrong.

Phil,

It's your call. Your preferred hypothesis about what causes the 
Open Access Advantage is Self-Selection Bias. So if you want to 
avoid Self-Selection Bias yourself, you have to design your 
studies so as to try to falsify your hypothesis, rather than 
trying only to confirm it.

Omitting the (self)-critical control conditions (the (1) 
non-randomized control in the BMJ paper and the (2) 
citing-paper-date control as well as the (3) 
self-archived-article control in the JASIST paper), omitting the 
further data-analysis (of (4) the PNAS data-set and the (5) 
journal and article citation levels in the JASIST paper), and 
omitting the further data-gathering time-base (beyond the (6) 
first year in the BMJ study) is not the way to make a credible 
case for your hypothesis (for those who did not already believe 
that the OA Advantage was just a Self-Selection Bias!).

If I had been the referee for BMJ or JASIST (having first fully 
disclosed that my own preferred hypothesis about what causes the 
Open Access Advantage is in conflict with yours), these are the 
six further tests I would have recommended, in order to avoid 
selective confirmation bias.

(It would not have been an altogether suitable reply to my 
referee report, I think, to suggest that if I thought they would 
make a difference, then I should be the one to go and do them!)

Chrs, Stevan

"Davis et al's 1-year Study of Self-Selection Bias: No Self-Archiving
Control, No OA Effect, No Conclusion"
http://openaccess.eprints.org/index.php?/archives/441-guid.html

"Confirmation Bias and the Open Access Advantage: Some Methodological
Suggestions for Davis's Citation Study"
http://openaccess.eprints.org/index.php?/archives/451-guid.html

> Davis, P. M. (2008 in press). Author-choice open access
> publishing in the biological and medical literature: a citation
> analysis. Journal of the American Society for Information Science
> and Technology. http://arxiv.org/abs/0808.2428
>
> Stevan Harnad wrote:
>
>> How does the fact that the overall sample was small and the
>> PNAS sample was large justify that the entire PNAS data-set was
>> not analyzed? (I don't contest that it should be analyzed (i)
>> within the aggregate as well as (ii) separately, and that (iii)
>> the rest should also be analyzed separately too, to avoid
>> skewing, I just don't understand why the full analyses were not
>> done and their results reported.)