RE: Open access and impact factor (fwd)

[Heather Morrison <heatherm@eln.bc.ca>]

posted on behalf of Eugene Garfield  Garfield@codex.cis.upenn.edu

Dear Heather Morrison: A colleague sent me your comments since I am
not subscribed to this listserv.

Your questions are interesting and valid, but they have been discussed in
the literature of citation analysis for many years. Author, journal and
institutional impact is regular discussed in the ASIS&T listserv at
SIGMETRICS@listserv.utk.edu

Please see some additional comments/replies in CAPS below.

---------- Forwarded message ----------
Date: Wed, 10 Mar 2004 23:24:29 EST
From: Heather Morrison <heatherm@eln.bc.ca>
Reply-To: liblicense-l@lists.yale.edu
To: liblicense-l@lists.yale.edu
Subject: Re: Open access and impact factor

Interesting questions.

Is impact itself, important though it might be, really a good measure of
merit? 
THERE HAVE BEEN NUMEROUS STUDIES WHICH SHOW A HIGH CORRELATION BETWEEN
PEER JUDGMENTS AND CITATION FREQUENCY. KNOWLEDGEABLE CITATION ANALYSTS
WOULD NOT SUGGEST THAT CITATION IMPACT ALONE BE USED TO MAKE SUCH
JUDGMENTS, ESPECIALLY WHEN LOW IMPACT JOURNALS AND AUTHORS ARE INVOLVED.
HOWEVER, MY EXTENSIVE STUDIES OF NOBEL LAUREATES DEMONSTRATES THAT THEY
ARE ALMOST INVARIABLY SUPER CITATION STARS. THESE CAN BE SEEN AT MY WEB
PAGE AT WWW.EUGENEGARFIELD.ORG ESPECIALLY AT
<http://www.garfield.library.upenn.edu/prize/prize.html>

For example:  if an area in science is suddenly deemed to have economic
significance, and therefore attracts reseach funding, so that articles in
that area are suddenly cited much more frequently, does this mean their
merit has increased?

RENEWED INTEREST IN OLD RESEARCH TOPICS IS NOT UNUSUAL. WITHOUT FUNDING
THERE WOULD BE LITTLE RESEARCH SO IT IS TO BE EXPECTED THAT MORE NEW
PAPERS WILL INCREASE CITATIONS TO OLDER ONES. BUT THIS GENERALLY IS VERY
SELECTIVE. THIS MIGHT BE CONSIDERED A VARIANT ON THE PHENOMENON OF DELAYED
RECOGNITION. THE LATTER SUBJECT HAS RECENTLY BEEN DISCUSSED BY WOLFGANG
GLANZEL AND WILL BE THE TOPIC OF A FORTHCOMING COMMENTARY IN "THE
SCIENTIST."

To put it another way, if an environmental scientist writes a definitive
article about a species of mosquito which is about to become extinct, and
no one reads it because no one cares, does that prove that the article has
no merit?  Or, if the importance of the species is suddenly understood 50
years later and citations begin to appear, has merit increased? 
I THINK THAT MOST PEOPLE WOULD AGREE THAT ALMOST ANYTHING PUBLISHED HAS
"MERIT" ( ESPECIALLY IN THE AUTHOR'S EYES) BUT IF NO ONE CARES AND NO ONE
USES IT WHAT IS ITS RELEVANCE TO THIS DISCUSSION. SCHOLARSHIP IS A
MERITOCRACY BUT IT IS THE IMPACT ON OTHER SCHOLARS THAT MAKES PUBLISHED
WORK SIGNIFICANT.

If an article receives many, many citations as an interesting example of
academic fraud, does this mean it has merit? 
WHILE EXPERIENCE TELLS US THERE ARE VERY FEW EXAMPLES OF SUCH HYPOTHETICAL
CASES ALL THAT MANY CITATIONS WOULD MEAN IN SUCH A CASE THAT THERE WAS A
LOT OF INTEREST IN THAT TOPIC ON THE PART OF THE SCIENTIFIC COMMUNITY.
SUCH FRAUDS ARE INEVITABLY UNCOVERED AND HAVE LITTLE BEARING ON THE
GENERAL DESIRE TO IDENTIFY THAT SMALL PERCENTILE OF WORK WHICH HAS HIGH
IMPACT.

Do articles that are within-paradigm receive more citations than articles
reflecting the view of an emerging paradigm?  Does this reflect merit?
Could an over-reliance on impact as a measure of merit lead to increasing
conservatism within scholarly endeavours? 
I DOUBT THAT SUCH A STUDY WOULD BE POSSIBLE CONSIDERING THE DIFFICULT OF
DEFINING PARADIGMS. HOWEVER YEAR BY YEAR CO-CITATION ANALYSIS HAS
PERMITTED THE IDENTIFICATION OF EMERGING PARADIGMS, THAT IS, RESEARCH
FRONTS. IT WOULD BE EXTREMELY DIFFICULT TO DEMONSTRATE WHETHER USE OF
CITATION IMPACT INCREASES OR DECREASES CONSERVATISM. IN THESE MATTERS
"BEAUTY IS IN THE EYE OF THE BEHOLDER"

If scholarly information becomes totally open access, and citations to
scientific research are found to be much less than citations to popular
music, does this prove that popular music has greater academic merit or
importance?

ALL IT DEMONSTRATES, IF TRUE, IS THAT POPULAR MUSIC IS OF GREATER UTILITY
OR INTEREST THAN SCIENTIFIC RESEARCH.

Do articles that are written in languages which are read by fewer people
of instrinsically less value or merit than articles in more common
languages?

HAS ANYONE EVER MADE SUCH A CLAIM? HOWEVER, THAT HAS NOTHING TO DO WITH
UNIVERSAL IMPACT. IT MAY WELL TURN OUT THAT SOME CHINESE ARTICLES ARE
HEAVILY CITED BY CHINESE SCHOLARS, BUT THAT WILL NOT GET THEM READ BY
ENGLISH READERS UNLESS THEY AE TRANSLATED. OF COURSE, THAT SITUATION MAY
CHANGE IN SOME DECADES IF CHINESE BECOMES THE LINGUA FRANCA OF SCIENCE.

Impact - or usage - are easy means of measuring the value of scholarly
information, but not necessarily the best.  Wouldn't it be better to rely
on more objective means of determining merit?  That is the one of purposes
of peer review, is it not?

CITATION ANALYSIS IS OBJECTIVE. PEER REVIEW IS SUBJECTIVE. IF THERE ARE
MORE OBJECTIVE MEANS OF DETERMINING MERIT LET'S HEAR THEM. THE REASON THAT
CITATION IMPACT HAS BECOME MORE POPULAR IS PRECISELY BECAUSE ONE KNOWS
WHAT THE MEASUREMENT MEANS. IT IS A MATTER OF USING THE DATA IN A
KNOWLEDGEABLE WAY AND NOT MINDLESSLY.

If work that is important is not being read, do we bury it, cancel our
subscriptions - or educate readers?

WHO IS TO DETERMINE THAT WORK IS IMPORTANT IS NO ONE IS READING IT? THERE
IS A HUGE AMOUNT OF LITERATURE THAT IS NEVER CITED. ARE WE GOING TO PAY
MORE ATTENTION TO IT OR TO THE WORK THAT IS USED AND CITED. 
EUGENE GARFIELD
---