RE: Article on peer review

["Ken Masters" <kmasters@ithealthed.com>]

Hi All

Thanks for the article which raises a host of interesting issues. 
(Although I'm not sure that your description of "post-publication 
peer review" is entirely accurate - in this instance, the 
"journal posted online four essays not yet accepted for 
publication" so the web-based review was performed prior to 
publication.)

I'm sure we all know most of the problems with standard 
peer-review: in the back of my mind is the fear caused by the 
fact that the peer-review is done by a non-random sample of 3-5 
people; hardly a convincing basis on which to determine overall 
quality.  Especially, if the paper pushes the limits, or 
questions a practice that one of the reviewers holds dear.

The model described in the paper does have a great advantage, I 
think. Not only increasing the quality of the paper, but having 
to post comments publicly, and not anonymously, might result in 
increasing the quality of the reviewers' comments.  For example, 
I'm sure we've all seen inane general review comments like "Does 
not meet scientific standards" with no further details provided. 
Or the balancing act one treads when one reviewer insists that 
more detail is required in one section, and a second reviewer 
complains that the same section is far too detailed.  Or when 
reviewers themselves don't know the referencing style of the 
journal, and insist that changes be made.  My favourite is "I 
don't understand this" with no further explanation - to which, 
under normal circumstance, the response might be "well, perhaps 
you should get to know the subject area more thoroughly." 
Instead the author has to water down the explanation so that it 
may be understood by a high school student - and then weather the 
storm of complaints when the finished article is criticised for 
being laborious and over-simplistic.  (Yes, one can defend, but 
defences always have to be so delicate for fear of upsetting 
reviewers' egos, that sometimes it's just not worth it).  Or when 
reviewers seem to be using the opportunity to display their own 
knowledge, rather than to comment on a paper with a view to 
getting a quality publication.

On the other hand, it is possible that some of these problems 
with standard peer-review would be exacerbated if there were 40, 
50, 100? peoples' comments that would have to be taken into 
account by the authors before the paper could be published.

One only has to look at the number of online comments generated 
by articles in journals like the BMJ - if the authors had to 
address all of these _before_ publication, would they have 
bothered?  (And then there are the inevitable "Me too's" 
(disguised as new insightful inputs), one-upmanship, or the 
bun-fights that start between commentators while the authors wait 
patiently).

But, in spite of my listing some complaints about peer-review, it 
still works, mostly.  Perhaps, the current system (such as that 
by the BMJ), peer-review by a small group, and then allowing 
readers' comments _after_ publication (to extend research and 
debate) is more viable.  (I think, closer to what Joe calls 
"post-publication peer review.")  Not perfect, I'll warrant, but 
it strikes the balance between getting a wide range of comments 
without delaying publication.  Of course, to be really effective, 
the article would have to be open access, but that is a personal 
prejudice, and a fight I can't have today :-).

Regards

Ken

Dr. Ken Masters
Asst. Professor: Medical Informatics
Medical Education Unit
College of Medicine & Health Sciences
Sultan Qaboos University
Sultanate of Oman
E-i-C: The Internet Journal of Medical Education


> -------- Original Message --------
> Subject: Article on peer review
> From: Joseph Esposito <espositoj@gmail.com>
> Date: Wed, August 25, 2010 2:06 am
>
> Article on peer review in the NY Times:
>
> http://j.mp/diUbLi
>
> The article describes an experiment in the humanities on "open
> peer review."  I would have called it post-publication peer
> review.
>
> Joe Esposito