RECENT MANUAL MEASUREMENTS OF OA AND OAA

[David Goodman <dgoodman@Princeton.EDU>]

Within the last few months, Stevan Harnad and his group, and we in our 
group, have carried out together several manual measurements of OA (and 
sometimes OAA, Open Acess Advantage). The intent has been to independently 
evaluate the accuracy of Chawki Hajjem's robot program, which has been 
widely used by Harnad's group to out similar measurements by computer.

The results from these measurements were first reported in a joint posting 
on Amsci,* referring for specifics to a simultaneously posted detailed 
technical report,** in which the results of each of several manual 
analyses were separately reported.

>From these data, both groups agreed that "In conclusion, the robot is not 
yet performing at a desirable level and future work may be needed to 
determine the causes, and improve the algorithm."

Our group has now prepared an overall meta-analysis of the manual results 
from both groups. *** We are able to combine the results, as we all were 
careful to examine the same sample base using identical protocols for both 
the counting and the analysis. Upon testing, we found a within-group 
inter-rater agreement of 93% and a between-groups agreement of 92%.

Between us, we analyzed a combined sample of 1198 articles in biology and 
sociology, 559 of which the robot had identified as OA, and 559 of which 
the robot had reported as non-OA.

Of the 559 robot-identified OA articles , only 224 actually were OA (37%).

Of the 559 robot-identified non-OA articles, 533 were truly non-OA (89%).

The discriminability index, a common used figure of merit, was only 0.97.

(We wish to emphasize that our group's results find true OAA in biology at 
a substantial level, and we all consider OAA one of the many reasons that 
authors should publish OA.)

In the many separate postings and papers from the SH group, such as **** 
and ***** done without our group's involvement, their authors refer only 
to the SH part of the small manual inter-rater reliability test. As it was 
a small and nonrandom sample, it yields an anomalous discriminability 
index of 2.45, unlike the values found for larger individual tests or for 
the combined sample. They then use that partial result by itself to prove 
the robot's accuracy.

None of the SH group's postings or publications refer to the joint report 
from the two groups, of which they could not have been ignorant, as the 
report was concurrently being evaluated and reviewed by SH.

Considering that both the joint ecs technical report ** and the separate 
SH group report***** were both posted on Dec .16 2005, we have here 
perhaps the first known instance of a author posting findings on the same 
subject, on the same day, as adjacent postings on the same list, but with 
opposite conclusions.

In view of these joint results, there is good reason to consider all 
current and earlier automated results performed using the CH algorithm to 
be of doubtful validity. The reader may judge: merely examine the graphs 
in the original joint Technical Report; **. They speak for themselves.

Dr. David Goodman
Palmer School of Library and Information Science
Long Island University     <dgoodman@liu.edu>

Kristin Antelman
North Carolina State University Libraries < kristin_antelman@ncsu.edu>

Nisa Bakkalbasi,  Yale University Library <nisa.bakkalbasi@yale.edu, >

REFERENCES

* http://listserver.sigmaxi.org/sc/wa.exe?A2=ind05&L
=american-scientist-open-access-forum&D=1&O=D&F=l&P=96445)

** "Evaluation of Algorithm Performance on Identifying OA" by Kristin 
Antelman, Nisa Bakkalbasi, David Goodman, Chawki Hajjem, Stevan Harnad (in 
alphabetical order) posted on ECS as http: eprints/ecs.soton.ac.uk/11689,

*** "Meta-analysis of OA and OAA manual determinations." David Goodman, 
Kristen Antelman, and Nisa Bakkalbasi, 
<http://eprints.rclis.org/archive/00005327/>

**** such as "Open Access to Research Increases Citation Impact"  by 
Chawki Hajjem, Yves Gingras, Tim Brody, Les Carr, and Stevan Harnad 
http://eprints.ecs.soton.ac.uk/11687

*****: "Ten-Year Cross-Disciplinary Comparison of the Growth of Open 
Access and How it Increases Research Citation Impact" by 5. C. Hajjem, S. 
Harnad, and Y. Gingras in IEEE Data Engineering Bulletin, 2005, 
http://eprints.ecs.soton.ac.uk/11688/

####