Re: The Number That's Devouring Science

[Stevan Harnad <harnad@ecs.soton.ac.uk>]

    Comment on: 
    Richard Monastersky, The Number That's Devouring Science, 
    Chronicle of Higher Education, October 1, 2005
    http://chronicle.com/weekly/v52/i08/08a01201.htm
    [text appended at the end of the comment]

        Impact Analysis in the PostGutenberg Era

Although Richard Monasterky describes a real problem -- the abuse of
journal impact factors -- its solution is so obvious one hardly required
so many words on the subject:

A journal's citation impact factor (CIF) is the average number of
citations received by articles in that journal (ISI -- somewhat
arbitrarily -- calculates CIFs on the basis of the preceding two years,
although other time-windows may also be informative; see
http://citebase.eprints.org/analysis/correlation.php )

There is an undeniable relationship between the usefulness of an article
and how many other articles use and hence cite it. Hence CIF does measure
the average usefulness of the articles in a journal. But there are three
problems with the way CIF itself is used, each of them readily
correctable:

    (1) A measure of the average usefulness of the articles in the journal
    in which a given article appears is no substitute for the actual
    usefulness of each article itself: In other words, the journal CIF is
    merely a crude and indirect measure of usefulness; each article's own
    citation count is the far more direct and accurate measure. (Using
    the CIF instead of an article's own citation count [or the average
    citation count for the author] for evaluation and comparison is
    like using the average marks for the school from which a candidate
    graduated, rather than the actual marks of the candidate.)

    (2) Whether comparing CIFs or direct article/author citation counts,
    one must always compare like with like. There is no point comparing
    either CIFs between journals in different fields, or citation
    counts for articles/authors in different fields. (Many daily users of
    CIFs for evaluation don't seem to bother with basing it on a
    normalised citation count, adjusting for different baseline citation
    levels and variability in different fields. It could easily be done,
    but it seems to be done mostly only by bibliometric specialists,
    rather than being rigorously applied by the actual daily users of
    CIFs or citation counts today.)

    (3) Both CIFs and citation counts can be distorted and abused. Authors
    can self-cite, or cite their friends; some journal editors can and do
    encourage self-citing their journal. These malpractices are deplorable,
    but most are also detectable, and then name-and-shame-able and
    correctable. ISI could do a better job policing them, but soon the
    playing field will widen, for as authors make their articles open
    access online, other harvesters -- such as citebase and citeseer
    and even google scholar -- will be able to harvest and calculate
    citation counts, and average, compare, expose, enrich and correct
    them in powerful ways that were in the inconceivable in the Gutenberg
    era:.

    http://citebase.eprints.org/
    http://citebase.eprints.org/
    http://scholar.google.com/

So, yes, CIFs are being misused and abused currently, but the cure is
already obvious -- and a wealth of powerful new resources are on the way
for measuring and analyzing research usage and impact online, including
(1) download counts, (2) co-citation counts (co-cited with, co-cited by),
(3) hub/authority ranks (authorities are highly cited papers cited by many
highly cited papers; hubs cite many authorities), (4) download/citation
correlations and other time-series analyses, (5) download growth-curve and
peak latency scores, (6) citation growth-curve and peak-latency scores,
(7) download/citation longevity scores, (8) co-text analysis (comparing
similar texts, extrapolating directional trends), and much more. It will
no longer be just CIFs and citation counts but a rich multiple regression
equation, with many weighted predictor variables based on these new
measures. And they will be available both for navigators and evaluators
online, and based not just on the current ISI database but on all of the
peer-reviewed research literature.

Meanwhile, use the direct citation counts, not the CIFs.

Some self-citations follow:

Brody,  T. (2003) Citebase Search: Autonomous Citation Database for e-print
Archives,  sinn03 Conference on Worldwide Coherent Workforce,  Satisfied Users -
New Services For Scientific Information,  Oldenburg,  Germany,  September 2003
http://eprints.ecs.soton.ac.uk/10677/

Brody,  T. (2004) Citation Analysis in the Open Access World Interactive Media
International
http://eprints.ecs.soton.ac.uk/10000/

Brody,  T. ,  Harnad,  S. and Carr,  L. (2005) Earlier Web Usage Statistics as
Predictors of Later Citation Impact. Journal of the American Association for
Information Science and Technology (JASIST,  in press). 
http://eprints.ecs.soton.ac.uk/10713/

Hajjem,  C., Gingras, Y., Brody, T., Carr, L. & Harnad, S. (2005) Across
Disciplines, Open Access Increases Citation Impact. (manuscript in preparation).
http://www.ecs.soton.ac.uk/~harnad/Temp/chawki1.doc

Hajjem,  C. (2005) Analyse de la variation de pourcentages d'articles en acc�s
libre en fonction de taux de citations 
http://www.crsc.uqam.ca/lab/chawki/ch.htm

Harnad,  S. and Brody,  T. (2004a) Comparing the Impact of Open Access (OA) vs.
Non-OA Articles in the Same Journals. D-Lib Magazine,  Vol. 10 No. 6
http://eprints.ecs.soton.ac.uk/10207/

Harnad,  S. and Brody,  T. (2004) Prior evidence that downloads predict
citations. British Medical Journal online.
http://eprints.ecs.soton.ac.uk/10206/

Harnad, S. and Carr, L. (2000) Integrating, Navigating and Analyzing
Eprint Archives Through Open Citation Linking (the OpCit Project). Current
Science 79(5):pp. 629-638.  http://eprints.ecs.soton.ac.uk/5940/

Harnad, S. , Brody, T. , Vallieres, F. , Carr, L. , Hitchcock, S. ,
Gingras, Y. , Oppenheim, C. , Stamerjohanns, H. and Hilf, E. (2004) The
Access/Impact Problem and the Green and Gold Roads to Open Access. Serials
Review, Vol. 30, No. 4, 310-314 http://eprints.ecs.soton.ac.uk/10209/

Hitchcock, S. , Brody, T. , Gutteridge, C. , Carr, L. , Hall, W. , Harnad,
S. , Bergmark, D. and Lagoze, C. (2002) Open Citation Linking: The Way
Forward. D-Lib Magazine 8(10).  http://eprints.ecs.soton.ac.uk/7717/

Hitchcock, S. , Carr, L. , Jiao, Z. , Bergmark, D. , Hall, W. , Lagoze, C.
and Harnad, S. (2000) Developing services for open eprint archives:
globalisation, integration and the impact of links. In Proceedings of the
5th ACM Conference on Digital Libraries, San Antonio, Texas, June 2000. ,
pages pp. 143-151.  http://eprints.ecs.soton.ac.uk/2860/

Hitchcock, S. , Woukeu, A. , Brody, T. , Carr, L. , Hall, W. and Harnad,
S. (2003) Evaluating Citebase, an open access Web-based citation-ranked
search and impact discovery service. Technical Report ECSTR-IAM03-005,
School of Electronics and Computer Science, University of Southampton
http://eprints.ecs.soton.ac.uk/8204/

Stevan Harnad