RE: Refereeing, how to do it

[David Goodman <dgoodman@princeton.edu>]

Scott has been willing to say from his inside experience what I did not
dare to say as an outsider. I agree with him about the average quality of
the reviewing (and possibly editing) for many journals.  And it is
certainly reasonable that people working in very broad fields-- as all
librarians inevitably do--will have particularly difficult problems.  I
deliberately picked my examples from CHOICE to be types of material where
it is relatively easier to be sure.

Scott's also right that we do need some mechanism for coping with the
inevitable cases where something does get through.  If that is what
Anthony meant also, I apologize to him.

Dr. David Goodman
Princeton University Library
and
Palmer School of Library & Information Science, Long Island University
dgoodman@princeton.edu

----- Original Message -----
From: T Scott Plutchak <tscott@uab.edu>
Date: Thursday, January 16, 2003 10:58 am
Subject: RE: Refereeing, how to do it

> David describes what is surely the ideal, and I'm sure that every 
> conscientious editor and reviewer strives to achieve it.  
> Nonetheless, as editor of the Journal of the Medical Library 
> Association, I have thought long and hard about the peer review 
> process and it is clearly not perfect.  I am less confident than 
> David is that editors and reviewers are quite as omniscient in 
> their fields as he would like them to be.  For example, in the 
> case of the JMLA, I think we have a very good review process, but 
> studies in the use of information by health professionals (for 
> example) can appear in many different sorts of publications, and 
> while I expect my reviewers to be generally familiar with such as 
> it appears in the library literature and the major health sciences 
> publications, if an author were to publish, in some state dental 
> society journal, an article on information resource use by 
> dentists, and then send that same article on to me, it is possible 
> that we would not catch the duplication.
> 
> The record of retracted publications over the last twenty 
> years, in many cases involving very reputable and high quality 
> journals, speaks for itself.  The very best review processes will 
> not catch all instances of error, fraud, carelessness, and, 
> indeed, plagiarism or copyright infringement.
> 
> The relevance to the "vanishing act" discussion is that if we do 
> not find a way to hold publishers harmless for inadvertent and 
> accidental infringement, then it will be very difficult to 
> convince corporate lawyers that they should go ahead and annotate 
> and link such articles rather than expunging them.
> 
> Those of you who are interested in looking into the peer review 
> process further may be interested in the following:
> 
> The indispensible starting point for serious study of peer review 
> is Ann C. Weller's "Editorial Peer Review: Its Strengths and 
> Weaknesses" (ASIST, 2001).  This is a systematic review of over 
> 1,100 studies of the peer review process and includes chapters on 
> Editors and Editorial Boards, Role of Reviewers, Reviewers and 
> their Biases, and Peer Review in an Electronic Environment, among 
> other topics.
> 
> In 1986, JAMA sponsored the first International Congress on Peer 
> Review in Biomedical Publication.  The proceedings of the 4th 
> Congress, which was held in Barcelona in September 2001, are 
> published as a special theme issue of JAMA, June 5, 2002, volume 
> 297, number 21.
> 
> In 2000/2001, the journal Academic Medicine underwent a thorough 
> study of their own peer review process.  The results of that 
> study, which includes very useful information on the peer review 
> process at other publications, is presented in Academic Medicine, 
> September 2001, volume 76, number 9.
> 
> And finally, Ann Weller recently brought to my attention a new 
> publication from BMJ Books, "How to survive peer review" by 
> Elizabeth Wager, Fiona Godlee and Tom Jefferson (BMJ Books, 2002). 
> I haven't seen it yet, but it comes highly recommended.
> 
> Scott
> 
> T. Scott Plutchak
> Director, Lister Hill Library of the Health Sciences
> University of Alabama at Birmingham
> tscott@uab.edu