Re: RECENT MANUAL MEASUREMENTS OF OA AND OAA

[Stevan Harnad <harnad@ecs.soton.ac.uk>]

On Wed, 11 Jan 2006, David Goodman wrote (in liblicense-l):

> Within the last few months, Stevan Harnad and his group, and we in our 
> group, have carried out together several manual measurements of OA (and 
> sometimes OAA, Open Access Advantage). The intent has been to 
> independently evaluate the accuracy of Chawki Hajjem's robot program, 
> which has been widely used by Harnad's group to out similar measurements 
> by computer.
>
> The results from these measurements were first reported in a joint 
> posting on Amsci,* referring for specifics to a simultaneously posted 
> detailed technical report,** in which the results of each of several 
> manual analyses were separately reported.
>
> * http://listserver.sigmaxi.org/sc/wa.exe?A2=ind05&L
> =american-scientist-open-access-forum&D=1&O=D&F=l&P=96445)
>
> ** "Evaluation of Algorithm Performance on Identifying OA" by Kristin
> Antelman, Nisa Bakkalbasi, David Goodman, Chawki Hajjem, Stevan Harnad (in
> alphabetical order) posted on ECS as http: eprints/ecs.soton.ac.uk/11689,
>
> From these data, both groups agreed that "In conclusion, the robot is not
> yet performing at a desirable level and future work may be needed to
> determine the causes, and improve the algorithm."

I am happy that David and his co-workers did an independent test of how 
accurately Chawki's robot detects OA. The robot over-estimates OA (i.e., 
it miscodes many non-OA articles as OA: false positives, or false OA).

Since our primary interest was and is in demonstrating the OA citation 
impact advantage, we had reasoned that any tendency to mix up OA and 
non-OA would go against us, because we were comparing the relative number 
of citations for OA and non-OA articles: the OA/non-OA citation ratio. So 
mixing up OA and non-OA would simply dilute that ratio, hence the 
detectability of any underlying OA advantage. (But more on this below.)

We were not particularly touting the robot's accuracy in and of itself, 
nor its absolute estimates of the percentage of OA articles. There are 
other estimates of %OA, and they all agree that it is roughly between 5% 
and 25%, depending on field and year. We definitely do not think that 
pinning down that absolute percentage accurately is the high priority 
research goal at this time.

In contrast, confirming the OA impact advantage (as first reported in 2001 
by Lawrence for computer science) across other disciplines *is* a high 
priority research goal today (because of its importance for motivating 
OA). And we have already confirmed that OA advantage in a number of areas 
of physics and mathematics *without the use of a robot.*

     Brody, T. and Harnad, S. (2004) Comparing the Impact of Open Access
     (OA) vs. Non-OA Articles in the Same Journals. D-Lib Magazine 10(6).
     http://eprints.ecs.soton.ac.uk/10207/

     Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S.,
     Yves, G., Charles, O., Stamerjohanns, H. and Hilf, E. (2004)
     The Access/Impact Problem and the Green and Gold Roads to Open
     Access. Serials review 30(4).
     http://eprints.ecs.soton.ac.uk/10209/

     Brody, T., Harnad, S. and Carr, L. (2005) Earlier Web Usage
     Statistics as Predictors of Later Citation Impact. Journal of the
     American Association for Information Science and Technology (JASIST).
     http://eprints.ecs.soton.ac.uk/10713/

For the OA advantage too, it is its virtually exception-free positive 
polarity that is most important today -- less so its absolute value or 
variation by year and field.

The summary of the Goodman et al. independent signal-detection analysis of 
the robot's accuracy is the following:

     This is a second signal-detection analysis of the accuracy of a
     robot in detecting open access (OA) articles (by checking by hand
     how many of the articles the robot tagged OA were really OA, and
     vice versa). A first analysis, on a smaller sample (Biology: 100
     OA, 100 non-OA), had found a detectability (d') of 2.45 and bias of
     0.52 (hits 93%, false positives 16%; Biology %OA: 14%; OA citation
     advantage: 50%). The present analysis on a larger sample (Biology:
     272 OA, 272 non-OA) found a detectability of 0.98 and bias of 0.78
     (hits 77%, false positives, 41%; Biology %OA: 16%; OA citation
     advantage: 64%). An analysis in Sociology (177 OA, 177 non-OA)
     found near-chance detectability (d' = 0.11) and an OA bias of 0.99
     (hits, 9%, false alarms, -2%; prior robot estimate Sociology %OA:
     23%; present estimate 15%). It was not possible from these data to
     estimate the Sociology OA citation advantage. CONCLUSIONS: The robot
     significantly overcodes for OA. In Biology 2002, 40% of identified
     OA was in fact OA. In Sociology 2000, only 18% of identified OA
     was in fact OA. Missed OA was lower: 12% in Biology 2002 and 14% in
     Sociology 2000. The sources of the error are impossible to determine
     from the present data, since the algorithm did not capture URLs for
     documents identified as OA. In conclusion, the robot is not yet
     performing at a desirable level and future work may be needed to
     determine the causes, and improve the algorithm.

In other words, the second test, based on the better, larger sample, finds 
a lower accuracy and a higher false-OA bias. In Biology, the robot had 
estimated 14% OA overall; the estimate based on the Goodman et al sample 
was instead 16% OA. (So the robot's *over*coding of the OA had actually 
resulted in a slight *under*estimate of %OA -- largely because the 
population proportion of OA is so low: somewhere between 5% and 25%.) The 
robot had found an average OA advantage of 50% in Biology; the Goodman et 
al sample found an OA advantage of 64%.  (Again, there was not much 
change, because the overall proportion of OA is still so low.)

Our robot's accuracy for Sociology (which we had not tested, so Goodman et 
al's was the first test) turned out to be much worse, and we are 
investigating this further. It will be important to find out why the 
robot's accuracy in detecting OA would vary from field to field.

> Our group has now prepared an overall meta-analysis of the manual 
> results from both groups. *** We are able to combine the results, as we 
> all were careful to examine the same sample base using identical 
> protocols for both the counting and the analysis. Upon testing, we found 
> a within-group inter-rater agreement of 93% and a between-groups 
> agreement of 92%.
>
> *** "Meta-analysis of OA and OAA manual determinations." David Goodman,
> Kristen Antelman, and Nisa Bakkalbasi,
> <http://eprints.rclis.org/archive/00005327/>

I am not sure about the informativeness of a "meta-analysis" based on two 
samples, from two different fields, whose main feature is that there seems 
to be a substantial difference in robot accuracy between the two fields! 
Until we determine why the robot's accuracy would differ by field, 
combining these two divergent results is like averaging over apples and 
oranges. It is trying to squeeze too much out of limited data.

Our own group is currently focusing on testing the robot's accuracy in 
Biology and Sociology (see end of this message), using a still larger 
sample of each, and looking at other correlates, such as the number of 
search-matches for each item. This is incomparably more important than 
simply increasing the robot's accuracy for its own sake, ot for trying to 
get more accurate absolute estimates of the percentage of OA articles, 
because if the robot's false-OA bias were to be large enough *and* were 
correlated with the number of search-match items (i.e., if articles that 
have more non-OA matches on the Web are more likely to be falsely coded as 
OA) then this would compromise the robot-based OA-advantage estimates.

> Between us, we analyzed a combined sample of 1198 articles in biology and
> sociology, 559 of which the robot had identified as OA, and 559 of which
> the robot had reported as non-OA.
>
> Of the 559 robot-identified OA articles , only 224 actually were OA (37%).
> Of the 559 robot-identified non-OA articles, 533 were truly non-OA (89%).
> The discriminability index, a common used figure of merit, was only 0.97.

It is not at all clear what these figures imply, if anything. What would
be of interest would be to calculate the OA citation advantage for each
field (separately, and then, if you wish, combined) based on the citation
counts for articles now correctly coded by humans as OA and non-OA in
this sample, and to compare that with the robot-based estimate.

More calculations on the robot's overall inaccuracy averaging across these two
fields is not in and of itself providing any useful information.

> (We wish to emphasize that our group's results find true OAA in biology at
> a substantial level, and we all consider OAA one of the many reasons that
> authors should publish OA.)

It would be useful to look at the OAA (OA citation advantage) for the 
Sociology sample too, but note that the right way to compare OA and non-OA 
citations is within the same journal/year. Here only one year is involved, 
and perhaps even the raw OA/non-OA citation ratio will tell us something, 
but not a lot, given that there can be journal-bias, with the OA articles 
coming from some journals and the non-OA ones coming from different 
journals: Journals do not all have the same average citation counts.

> In the many separate postings and papers from the SH group, such as ****
> and ***** done without our group's involvement, their authors refer only
> to the SH part of the small manual inter-rater reliability test. As it was
> a small and nonrandom sample, it yields an anomalous discriminability
> index of 2.45, unlike the values found for larger individual tests or for
> the combined sample. They then use that partial result by itself to prove
> the robot's accuracy.
>
> **** such as "Open Access to Research Increases Citation Impact"  by
> Chawki Hajjem, Yves Gingras, Tim Brody, Les Carr, and Stevan Harnad
> http://eprints.ecs.soton.ac.uk/11687
>
> *****: "Ten-Year Cross-Disciplinary Comparison of the Growth of Open
> Access and How it Increases Research Citation Impact" by 5. C. Hajjem, S.
> Harnad, and Y. Gingras in IEEE Data Engineering Bulletin, 2005,
> http://eprints.ecs.soton.ac.uk/11688/

No one is "proving" (or interested in proving) robot accuracy! In our 
publications to date, we cite our results to date. The Goodman et al. test 
results came out too late to be mentioned in the ***** published article, 
but they will be mentioned in the **** updated preprint (along with the 
further results from our ongoing tests).

> None of the SH group's postings or publications refer to the joint 
> report from the two groups, of which they could not have been ignorant, 
> as the report was concurrently being evaluated and reviewed by SH.

Are Goodman et al. suggesting that there has been some suppression of 
information here -- information from reports that we have co-signed and 
co-posted publicly? Or are Goodman et al. concerned that they are not 
getting sufficient credit for something?

> Considering that both the joint ecs technical report ** and the separate 
> SH group report***** were both posted on Dec .16 2005, we have here 
> perhaps the first known instance of a author posting findings on the 
> same subject, on the same day, as adjacent postings on the same list, 
> but with opposite conclusions.

One of the postings being a published postprint and the other an 
unpublished preprint! Again, what exactly is Goodman et al.'s point?

> In view of these joint results, there is good reason to consider all 
> current and earlier automated results performed using the CH algorithm 
> to be of doubtful validity. The reader may judge: merely examine the 
> graphs in the original joint Technical Report; **. They speak for 
> themselves.

No, the robot accuracy tests do not speak for themselves. Nor does the 
conclusion of Goodman et al's preprint (***) (which I am now rather 
beginning to regret having obligingly "co-signed"!):

     "In conclusion, the robot is not yet performing at a desirable level
     and future work may be needed to determine the causes, and improve
     the algorithm."

What *I* meant in agreeing with that conclusion was that we needed to find 
out why there were the big differences in the robot accuracy estimates 
(between our two samples and between the two fields). The robot's 
detection accuracy can and will be tightened, if and when it becomes clear 
that it needs to be, for our primary purpose (measuring and comparing the 
OA citation advantage across fields) or even our secondary purpose 
(estimating the relative percentage of OA by field and year), but not as 
an end in itself (i.e., just for the sake of increasing or "proving" robot 
accuracy).

The reason we are doing our analyses with a robot rather than by hand is 
to be able to cover far more fields, years and articles, more quickly, 
than it is possible to do by hand. The hand-samples are a good check on 
the accuracy of the robot's estimates, but they are not necessarily a 
level of accuracy we need to reach or even approach with the robot!

On the other hand, potential artifacts -- tending in opposite directions 
-- do need to be tested, and, if necessary, controlled for (including 
tightening the robot's accuracy):

     (1) to what extent is the OA citation "advantage" just a non-causal
     self-selection quality bias, with authors selectively self-archiving
     their higher-quality, hence higher citation-probability articles?

     (2) to what extent is the OA citation "advantage" just an artifact
     of false positives by the robot? (because there will be more false
     positives when there are more matches with the reference search from
     articles *other* than the article itself, hence more false positives
     with articles that are more cited on the web, which would make the
     robot-based outcome not an OA effect, and circular)

A third question (not about a potential artifact, but about a genuine
causal component of the OA advantage) is:

    (3) to what extent is the OA advantage an Early (preprint) Advantage
    (EA)?

For those who are interested in our ongoing analyses, I append some
further information below.

Stevan Harnad

Chawki: Here are the tests and controls that need to be done
to determine both the robot's accuracy in detecting and estimating
%OA and the causality of the observed citation advantage:

(1) When you re-do the searches in Biology and Sociology (to begin with: 
other disciplines can come later), make sure to (1a) store the number as 
well as the URLs of all retrieved sites that match the reference-query and 
(1b) make the robot check the whole list (up to at least the pre-specified 
N-item limit you used before) rather than the robot's stopping as soon as 
it thinks it has found that the item is "OA," as in your prior searches.

That way you will have, for each of your Biology and Sociology ISI 
reference articles, not only their citation counts, but also their 
query-match counts (from the search-engines) and also the number and 
ordinal position for every time the robot calls them "OA." (One item might 
have, say, k query-matches, with the 3rd, 9th and kth one judged "OA" by 
the robot, and the other k-3 judged non-OA.)

Both the number (and URLs) of query-matches and the ordinal position of 
the first "OA"-call and the total number and proportion of OA-calls will 
be important test data to make sure that our robot-based OA citation 
advantage estimate is *not* just a query-match-frequency and/or 
query-match frequency plus false alarm artifact. (The potential artifact 
is that the robot-based OA advantage is not an OA advantage at all, but 
merely a reflection of the fact that more highly cited articles are more 
likely to have online items that *cite* them, and that these online items 
are the ones the robot is *mistaking* for OA full-texts of the *cited* 
article itself.) (2) As a further check on robot accuracy, please use a 
subset of URLs for articles that we *know* to be OA (e.g., from PubMed 
Central, Google Scholar, Arxiv, CogPrints) and try both the search-engines 
(for % query-matches) and the robot (for "%OA") on them. That will give 
another estimate of the *miss* rate of the search-engines as well as of 
the robot's algorithm for OA.

(3) While you are doing this, in addition to the parameters that are 
stored with the reference (the citation count, the URLs for every 
query-match by the search, the number, proportion, and ordinal position of 
those of the matches that the robot tags as "OA"), please also store the 
citation impact factor of the *journal* in which the reference article was 
published. (We will use this to do sub-analyses to see whether the pattern 
is the same for high and low impact journals, and across disciplines; we 
will also look at it separately, for %OA among articles at different 
citation levels (1, 2-3, 4-7, 7-15, 16-31, 32-63, 64+), again within and 
across years and disciplines.)

(4) The sampling for Biology and Sociology should of course be based on 
*pairs* within the same journal/year/issue-number: Assuming that you will 
be sampling 500 pairs (i.e., 1000 items) in each discipline (1000 Biology, 
1000 Sociology), please first pick a *random* sample of 50 pairs for each 
year, and then, within each pair, pick, at *random*, one OA and one non-OA 
article per same issue. Use only the robot's *first* ordinal OA as your 
criterion for "OA" (so that you are duplicating the methodology the robot 
had used); the criterion for non-OA is, as before: none found among all of 
the search matches). If you feel you have the time, it would also be 
informative to check the 2nd or 3rd "OA" item if the robot found more than 
one. That too would be a good control datum, for evaluating the robot's 
accuracy under different conditions (number of matches; number/proportion 
of them judged "OA").

     http://eprints.ecs.soton.ac.uk/11687/
     http://eprints.ecs.soton.ac.uk/11688/
     http://eprints.ecs.soton.ac.uk/11689/

(5) Count also the number of *journals* for which the robot judges that it 
is at or near 100% OA (for those are almost certainly OA journals and not 
self-archived articles). Include them in your %OA counts, but of course 
not in your OA/NOA ratios. (It would be a good idea to check all the ISI 
journal names against the DOAJ OA journals list -- about 2000 journals -- 
to make sure you catch all the OA journals.) Keep a count also of how many 
individual journal *issues* has either 100% OA or 0% OA (and were hence 
eliminated from the OA/NOA citation ratio). Those numbers will also be 
useful for later analyses and estimates.

With these data we will be in a much better position to estimate the 
robot's accuracy and some of the factors contributing to the OA citation 
advantage.