Re: Essay on article metrics

[Joseph Esposito <espositoj@gmail.com>]

This is tiresome.

There is no OA Advantage not because of the merits or limitations 
of OA but because all these purported advantages and 
disadvantages are based on the pseudo-science of quantifying what 
is not entirely quantifiable.  Citation count or page views or 
downloads or whatever are useful *approximations* of some 
qualities of materials, but an approximation is not the same 
thing as the underlying value, which is subject to various 
interpretations.  You might as well base a college admissions 
program on the sole criterion that a math score of 710 is 
unquestionably better than one of 700.

These comments apply to subscription-based publishing as well as 
to OA publications.

I know the fashion is to shut down humanities departments or 
simply to starve them (or at least all of the adjuncts who work 
there) to death, but could we not reintroduce some judgment into 
this discussion?  And while we are at it, how about a simple 
experiment:  those who wish to publish with an OA service do so, 
and those who don't, don't.

Joe Esposito

On Fri, Jan 8, 2010 at 3:47 PM, Stevan Harnad <amsciforum@gmail.com> wrote:

> On Thu, Jan 7, 2010 at 8:00 PM, Joseph Esposito
> <espositoj@gmail.com> wrote:
>
>> See Phil Davis today in The Scholarly Kitchen:
>>
>> http://bit.ly/8T69j3
>>
>> Davis is developing a set of insights that an enterprising
>> editor will want to see as a book, "The Metrics of Scholarly
>> Communication." One of his strengths is his ability to explain
>> statistics to people who have no training in the field.
>
> Phil Davis: "An interesting bit of research, although I have some
> methodological concerns about how you treat the data, which may
> explain some inconsistent and counter-intuitive results, see:
> http://j.mp/8LK57u A technical response addressing the
> methodology is welcome."
>
> Thanks for the feedback. We reply to the three points of
> substance, in order of importance:
>
> (1) LOG RATIOS: We analyzed log citation ratios to adjust for
> departures from normality. Logs were used to normalize the
> citations and attenuate distortion from high values. This
> approach loses some values when the log tranformation makes the
> denominator zero, but despite these lost data, the t-test results
> were significant, and were further confirmed by our second,
> logistic regression analysis. Moed's (2007) point was about
> (non-log) ratios that were not used in this study. We used the
> ratio of log citations and not the log of citation ratios. When
> we compare log3/log2 with log30/log20, we don't compare
> percentages with percentages (60% with 14%) because the citation
> values are transformed or normalized: the higher the citations,
> the stronger the normalisation. It is highly unlikely that any of
> this would introduce a systematic bias in favor of OA, but if the
> referees of the paper should call for a "simpler and more
> elegant" analysis to make sure, we will be glad to perform it.
>
> (2) EFFECT SIZE: The size of the OA Advantage varies greatly from
> year to year and field to field. We reported this in Hajjem et al
> (2005), stressing that the important point is that there is
> virtually always a positive OA Advantage, absent only when the
> sample is too small or the effect is measured too early (as in
> Davis et al's 2008 study). The consistently bigger OA Advantage
> in physics (Brody & Harnad 2004) is almost certainly an effect of
> the Early Access factor, because in physics, unlike in most other
> disciplines (apart from computer science and economics), authors
> tend to make their unrefereed preprints OA well before
> publication. (This too might be a good practice to emulate, for
> authors desirous of greater research impact.)
>
> (3) MANDATED OA ADVANTAGE? Yes, the fact that the citation
> advantage of mandated OA was slightly greater than that of
> self-selected OA is surprising, and if it proves reliable, it is
> interesting and worthy of interpretation. We did not interpret it
> in our paper, because it was the smallest effect, and our focus
> was on testing the Self-Selection/Quality-Bias hypothesis,
> according to which mandated OA should have little or no citation
> advantage at all, if self-selection is a major contributor to the
> OA citation advantage.
>
> Our sample was 2002-2006. We are now analyzing 2007-2008. If
> there is still a statistically significant OA advantage for
> mandated OA over self-selected OA in this more recent sample too,
> a potential explanation is the inverse of the
> Self-Selection/Quality-Bias hypothesis (which, by the way, we do
> think is one of the several factors that contribute to the OA
> Advantage, alongside the other contributors: Early Advantage,
> Quality Advantage, Competitive Advantage, Download Advantage,
> Arxiv Advantage, and probably others).
>
> The Self-Selection/Quality-Bias (SSQB) consists of better authors
> being more likely to make their papers OA, and/or authors being
> more likely to make their better papers OA, because they are
> better, hence more citeable. The hypothesis we tested was that
> all or most of the widely reported OA Advantage across all fields
> and years is just due to SSQB. Our data show that it is not,
> because the OA Advantage is no smaller when it is mandated. If it
> turns out to be reliably bigger, the most likely explanation is a
> variant of the "Sitting Pretty" (SP) effect, whereby some of the
> more comfortable authors have said that the reason they do not
> make their articles OA is that they think they have enough access
> and impact already. Such authors do not self-archive
> spontaneously. But when OA is mandated, their papers reap the
> extra benefit of OA, with its Quality Advantage (for the better,
> more citeable papers). In other words, if SSQB is a bias in favor
> of OA on the part of some of the better authors, mandates reverse
> an SP bias against OA on the part of others of the better
> authors. Spontaneous, unmandated OA would be missing the papers
> of these SP authors.
>
> There may be other explanations too. But we think any explanation
> at all is premature until it is confirmed that this new mandated
> OA advantage is indeed reliable and replicable. Phil further
> singles out the fact that the mandate advantage is present in the
> middle citation ranges and not the top and bottom. Again, it
> seems premature to interpret these minor effects whose
> unreliability is unknown, but if forced to pick an interpretation
> now, we would say it was because the "Sitting Pretty" authors may
> be the middle-range authors rather than the top ones...
>
> Yassine Gargouri, Chawki Hajjem, Vincent Lariviere, Yves Gingras,
> Les Carr, Tim Brody, Stevan Harnad
>
> ***