RE: Refereeing, how to do it

["T Scott Plutchak" <tscott@uab.edu>]

David describes what is surely the ideal, and I'm sure that every
conscientious editor and reviewer strives to achieve it.  Nonetheless, as
editor of the Journal of the Medical Library Association, I have thought
long and hard about the peer review process and it is clearly not perfect.  
I am less confident than David is that editors and reviewers are quite as
omniscient in their fields as he would like them to be.  For example, in
the case of the JMLA, I think we have a very good review process, but
studies in the use of information by health professionals (for example)
can appear in many different sorts of publications, and while I expect my
reviewers to be generally familiar with such as it appears in the library
literature and the major health sciences publications, if an author were
to publish, in some state dental society journal, an article on
information resource use by dentists, and then send that same article on
to me, it is possible that we would not catch the dup! lication.  The
record of retracted publications over the last twenty years, in many cases
involving very reputable and high quality journals, speaks for itself.  
The very best review processes will not catch all instances of error,
fraud, carelessness, and, indeed, plagiarism or copyright infringement.

The relevance to the "vanishing act" discussion is that if we do not find
a way to hold publishers harmless for inadvertent and accidental
infringement, then it will be very difficult to convince corporate lawyers
that they should go ahead and annotate and link such articles rather than
expunging them.

Those of you who are interested in looking into the peer review process
further may be interested in the following:

The indispensible starting point for serious study of peer review is Ann
C. Weller's "Editorial Peer Review: Its Strengths and Weaknesses" (ASIST,
2001).  This is a systematic review of over 1,100 studies of the peer
review process and includes chapters on Editors and Editorial Boards, Role
of Reviewers, Reviewers and their Biases, and Peer Review in an Electronic
Environment, among other topics.

In 1986, JAMA sponsored the first International Congress on Peer Review in
Biomedical Publication.  The proceedings of the 4th Congress, which was
held in Barcelona in September 2001, are published as a special theme
issue of JAMA, June 5, 2002, volume 297, number 21.

In 2000/2001, the journal Academic Medicine underwent a thorough study of
their own peer review process.  The results of that study, which includes
very useful information on the peer review process at other publications,
is presented in Academic Medicine, September 2001, volume 76, number 9.

And finally, Ann Weller recently brought to my attention a new publication
from BMJ Books, "How to survive peer review" by Elizabeth Wager, Fiona
Godlee and Tom Jefferson (BMJ Books, 2002).  I haven't seen it yet, but it
comes highly recommended.

Scott

T. Scott Plutchak
Director, Lister Hill Library of the Health Sciences
University of Alabama at Birmingham
tscott@uab.edu

-----Original Message-----
From: David Goodman [mailto:dgoodman@Princeton.EDU]
Sent: Wednesday, January 15, 2003 12:16 PM
To: reedelscustomers@lists.cc.utexas.edu
Subject: Refereeing, how to do it

I feel somewhat embarrassed in explaining to Anthony a subject so closely
within his experience --especially since he personally has taught me so
much of what I know in this area.

The reviewer of an article is supposed to be an expert in the specific
subject of the paper. He consequently is expected to be aware of all
published work on the topic. Therefore, the reviewer will know if the
material has been published elsewhere or previously, and, if in doubt,
will normally re-check the literature. The review can also be expected to
be aware of the research groups likely to be working on the subject, and,
to some extent, their potential publications.

One who does not know this he should not be reviewing the article. The
editor's role is to know who is an expert in the various fields covered by
the journal. An editor without this broad range of knowledge should not be
editing the journal. The publisher is expected to know the academic world
well enough to select a suitable editor.

Naturally, there are some circumstances that this will not detect-- e.g.,
material in press, simultaneous publication, material published in obscure
sources and not indexed....  The 100% that Anthony asks for cannot be
achieved, but most of the plagiarism can. Most cases are not of this
nature, but are careless neglect of what should have been obvious.

To give a personal example, when I review an dictionary for CHOICE, I make
sure I am aware of every similar work. If any of the contents has appeared
in any accessible material I expect to recognize it. When I review an
encyclopedia it's harder because of the range of material an encyclopedia
covers, but I will at least notice duplications in coverage and text with
other similar works--and I do find them. My excellent editor R. Balay
knows the fields I can handle, and doesn't send me material I can't. If I
were to do a careless job, he'd recognize that. That's why he's so widely
known to be qualified as the reference books editor of this authoritative
publication.

For something as specific as a primary article, I'd certainly expect to be
able to do this. When I read an article on, say, bibliometrics, I will
know if the content has been published before. Now, if it's been
translated from an oriental language, or an unpublished Ph.D. thesis, I
may not know--though I do know from which universities a thesis on the
subject might be expected to originate-- but if it's been plagiarized from
any similar article or any of the standard texts I will recognize it. If
could not do this I would not be qualified to teach or do research in the
subject, and an editor who knows the workers in the field should be aware
of that. The editor should, furthermore, be able to tell from my report
that I did not do a careful job, and know not to ask me again.
--