"Academics strike back at spurious rankings" (Nature, 31 May)

[Stevan Harnad <harnad@ecs.soton.ac.uk>]

     Academics strike back at spurious rankings
     D Butler, Nature 447, 514-515 (31 May 2007) doi:10.1038/447514b
     http://www.nature.com/nature/journal/v447/n7144/full/447514b.html

This news item in Nature lists some of the (very valid) objections to the
many unvalidated university rankings -- both subjective and objective --
that are in wide use today.

These problems are all the more reason for extending Open Access (OA)
and developing OA scientometrics, which will provide open, validatable
and calibratable metrics for research, researchers, and institutions in
each field -- a far richer, more sensitive, and more equitable spectrum
of metrics than the few, weak and unvalidated measures available today.

Some research groups that are doing relevant work on this are, in the UK:
(1) our own OA scientometrics group at Southampton (and UQaM, Canada),
and our collaborators Charles Oppenheim (Loughborough) and Arthur Sale
(Tasmania); (2) Mike Thelwall (Wolverhampton); in the US: (3) Johan
Bollen & Herbert van de Sompel at LANL; and in the Netherlands: (5)
Henk Moed & Anton van Raan (Leiden; cited in the Nature news item).

Below are excerpts from the Nature article, followed by some references.

     Universities seek reform of ratings.
     http://www.nature.com/nature/journal/v447/n7144/full/447514b.html

     [A] group of US colleges [called for a] boycott [of] the most
     influential university ranking in the United States...  Experts argue
     that these are based on dubious methodology and spurious data,
     yet they have huge influence...

     "All current university rankings are flawed to some extent; most,
     fundamentally,"

     The rankings in the U.S. News & World Report and those published by
     the British Times Higher Education Supplement (THES) depend heavily
     on surveys of thousands of experts - a system that some contest. A
     third popular ranking, by Jiao Tong University in Shanghai, China,
     is based on more quantitative measures, such as citations, numbers
     of Nobel prizewinners and publications in Nature and Science. But
     even these measures are not straightforward.

     Thomson Scientific's ISI citation data are notoriously poor for
     use in rankings; names of institutions are spelled differently from
     one article to the next, and university affiliations are sometimes
     omitted altogether. After cleaning up ISI data on all UK papers for
     such effects... the true number of papers from the University of
     Oxford, for example, [were] 40% higher than listed by ISI...

     Researchers at Leiden University in the Netherlands have similarly
     recompiled the ISI database for 400 universities: half a million
     papers per year. Their system produces various rankings based on
     different indicators. One, for example, weights citations on the
     basis of their scientific field, so that a university that does well
     in a heavily cited field doesn't get an artificial extra boost.

     The German Center for Higher Education Development (CHE) also offers
     rankings... for almost 300 German, Austrian and Swiss universities...
     the CHE is expanding the system to cover all Europe.

     The US Commission on the Future of Higher Education is considering
     creating a similar public database, which would offer competition
     to the U.S. News & World Report.

---------------------------------------------------------------------------

     Bollen, Johan and Herbert Van de Sompel. Mapping the structure of
     science through usage. Scientometrics, 69(2), 2006
     http://dx.doi.org/10.1007/s11192-006-0151-8

     Hardy, R., Oppenheim, C., Brody, T. and Hitchcock, S. (2005) Open
     Access Citation Information.
     http://eprints.ecs.soton.ac.uk/11536/

     Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003)
     Mandated online RAE CVs Linked to University Eprint
     Archives: Improving the UK Research Assessment Exercise
     whilst making it cheaper and easier. Ariadne 35.
     http://www.ecs.soton.ac.uk/~harnad/Temp/Ariadne-RAE.htm

     Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open
     Research Web: A Preview of the Optimal and the Inevitable, in Jacobs,
     N., Eds. Open Access: Key Strategic, Technical and Economic Aspects,
     chapter 21. Chandos. http://eprints.ecs.soton.ac.uk/12453/

     Harnad, S. (2007) Open Access Scientometrics and the UK Research
     Assessment Exercise. Invited Keynote, 11th Annual Meeting of the
     International Society for Scientometrics and Informetrics. Madrid,
     Spain, 25 June 2007 http://arxiv.org/abs/cs.IR/0703131

     Kousha, Kayvan and Thelwall, Mike (2006) Google Scholar Citations and
     Google Web/URL Citations: A Multi-Discipline Exploratory Analysis.
     In Proceedings International Workshop on Webometrics, Informetrics
     and Scientometrics & Seventh COLLNET Meeting, Nancy (France).
     http://eprints.rclis.org/archive/00006416/

     Moed, H.F. (2005). Citation Analysis in Research Evaluation.
     Dordrecht (Netherlands): Springer.

     van Raan, A. (2007) Bibliometric statistical properties of the 100
     largest European universities: prevalent scaling rules in the science
     system. Journal of the American Society for Information Science and
     Technology  http://www.cwts.nl/Cwts/Stat4AX-JASIST.pdf

Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/
         To join or leave the Forum or change your subscription address:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html

UNIVERSITIES: If you have adopted or plan to adopt an institutional
policy of providing Open Access to your own research article output,
please describe your policy at:
         http://www.eprints.org/signup/sign.php

UNIFIED DUAL OPEN-ACCESS-PROVISION POLICY:
     BOAI-1 ("green"): Publish your article in a suitable toll-access journal
             http://romeo.eprints.org/
OR
     BOAI-2 ("gold"): Publish your article in a open-access journal if/when
             a suitable one exists.
             http://www.doaj.org/
AND
     in BOTH cases self-archive a supplementary version of your article
             in your institutional repository.
             http://www.eprints.org/self-faq/
             http://archives.eprints.org/
             http://openaccess.eprints.org/