Re: Another Poynder Eye-Opener on Open Access

[Stevan Harnad <amsciforum@gmail.com>]

On 2011-05-11, at 8:35 PM, <jean.claude.guedon@umontreal.ca> 
wrote:

> I said nothing about peer review, and I would also agree that peer
> review is indispensable. The "new form of judgement" that I allude to
> would be a form of peer review, but probably closer to jury review than
> to individual, isolated reviews.

Peer review is indispensable for two reasons:

(1) Peer review causes articles to be corrected and revised, 
interactively, as a *precondition* of publication.

In other words, peer review is neither just an accept/reject tag 
nor just a post-hoc grade or mark such as A, B, C, D. It is the 
result of an adjudication by experts to whose recommendations the 
author is answerable as a condition of being published. (What 
does resemble A/B/C/D is the journal hierarchy, where journal 
names and track-records attest to their quality standards. In 
other words, there are A/B/C/D journals, according to their 
quality standards. Users know this and weight articles 
accordingly.)

(2) Because it is an interactive precondition for publication, 
peer review provides a reliable quality filter for users (or at 
least a filter as reliable as the quality of the peer-reviewed 
literature today, such as it is).

When meeting a journal's known peer review standards is a 
*precondition* for publication, users are not confronted with the 
need to make do with raw, unfiltered papers. Only editors and 
referees have to read unfiltered submissions.

But the most important point to note is that peer review is 
active and answerable. Qualified but overworked peers do their 
duty to referee -- reluctantly, and selectively, depending both 
on the reputation and quality standards of the editor and journal 
inviting them to do so and on the relevance and interest of the 
submitted paper. They do so, confident that the author is 
answerable to the editor for acting upon those of their 
recommendations the editor judges to be appropriate.

It is extremely unlikely that unfiltered "publications" will find 
their qualified referees, bidden or unbidden, ready to devote 
their scarce time to reading and tagging them with a "grade," 
even though the articles are already "published," hence not 
answerable to the referee for corrections or revisions. And in 
any case, that's all too late and uncertain for the would-be 
user.

So, yes, it is indeed peer review and quality standards that are 
at issue when one speculates about replacing the current peer 
review system -- an interactive, answerable precondition for 
publications -- with an alternative post-hoc vetting and tagging 
system that has not even been tested for whether it could deliver 
a research literature of at least the quality and usability of 
the existing one.

And adopting such untested alternatives is certainly not the 
price that needs to be paid for open access to the existing peer 
reviewed research literature; for all that needs to be done there 
is to make the peer-reviewed drafts OA immediately upon 
acceptance for publication. No need to make only the raw 
unrefereed drafts OA and then wait for pot luck!

> Ranking amounts to having as many levels as there are entities being
> ranked. Levels, on the other hand, lump numbers of entities into the
> same category. Ranking favours only individualized competition; by
> contrast, levels stress thresholds of quality and do not try to identify
> the very best. Good systems, such as schools, for example, use both
> systems, and do not try to make just one approach carry the whole
> evaluation task. The granularity of grades leads to many students being
> lumped together.

You are absolutely right. With peer review, all papers published 
by a given journal share that journal's grade. Postpublication 
ranking of the already peer-reviewed and graded articles would be 
an excellent *supplement* to this system, but it is incoherent to 
imagine it as a *substitute*.

What needs to be made OA is the refereed, accepted draft, not 
just the unrefereed preprint. There is a world of difference 
between these two.

Guedon, Jean-Claude (2004) The "Green" and "Gold" Roads to Open 
Access: The Case for Mixing and Matching. Serials Review 30(4,): 
315-28  doi:10.1016/j.serrev.2004.09.005

Harnad, S. (2005) Fast-Forward on the Green Road to Open Access: 
The Case Against Mixing Up Green and Gold. Ariadne 42. 
http://eprints.ecs.soton.ac.uk/10675/

> The granularity of the impact factor is designed to rank and only rank.
> Why this is so is not entirely clear to me, but some people do seem to
> see advantages in creating a generalized atmosphere of intense
> competition. The justification may well be to extract the best out of
> everyone, but it should be considered that it also leads to cheating and
> sloppy work.

I think everyone agrees that neither journal impact factors 
(average citation counts) nor individual article or author 
citation counts (or download counts) are sufficient as metrics of 
quality, importance or influence. OA will make it possible to 
provide and collect many more metrics, including post-publication 
tagging and ranking. (User tagging of unrefereed preprints is 
also welcome.)

But again, these are supplements, not substitutes for peer 
review. Nor is there any need for resorting to untested 
substitutes for peer review (or waiting for them to be tested) in 
order to have 100% OA, today:

All that's needed for 100% OA today is for institutions and 
funders to mandate the self-archiving of the refereed, revised, 
accepted final drafts of all peer-reviewed journal articles, 
immediately upon acceptance for publication. 
http://bit.ly/eos-policy

This is the simple essence of OA today that is still being 
systematically missed by most researchers today.

> if I did not mention peer review, it was not because I dismissed
> it, but because, on the contrary, I took it for granted. My take on peer
> review is that it deals in some ways with quality, but not exclusively.
> Other dimensions such as relevance and timeliness of topics are also
> involved in the process. For this reason, I tend to describe passing the
> peer review process as being akin to being admitted across a border. In
> this case, the border is that of the scientific territory. To that
> extent, peer review is indispensable. Beyond that, I would not want to
> fall into a fetishistic mode.

But the topic of the thread on which you intervened (Richard 
Poynder's article) was the conjecture that for Open Access, peer 
review could be replaced by postpublication vetting of some sort.

> I also feel that peer review is not necessarily tied to journals. Peer
> review can be exercised by various institutionalized bodies that simply
> want to carry out some form of evaluation based on competence and
> knowledge, not on authority.

This returns squarely to what draft it is that authors need to 
make OA: The refereed, revised, accepted final draft, which 
anyone and everyone can then evaluate as they wish, by way of 
supplementing peer review -- or just the unrefereed draft, with 
the various evaluations being based on that.

Whatever "body" does the peer review, and certifies the outcome, 
is, in all relevant respects, a "journal." What it's called is 
not important; what's important is that it really provides 
answerable, interactive peer review, as a precondition for 
certification at the body's known quality level -- known to the 
usership from the "body's" established track record for quality.

> Finally, journals are not necessarily the only sites suited to
> evaluating the quality of scientific work. Up to now, they have been
> doing most of this work, but the advent of repositories exposing
> scientific work to the world will call for the practise of peer reviews
> in these new kinds of sites. "Exposing to the world" is the first
> meaning of publishing.

An institutional (or central) repository is merely an 
(open-)access-provider. It is not a peer-reviewer or publisher. 
Moreover, institutions overseeing the evaluation of their own 
output are likely to be seen as (and to become) vanity-presses, 
for obvious reasons of conflict of interest. Peer review needs to 
be done by a neutral 3rd party, not the author or the author's 
institution.

But whatever the "body" is, if it is doing answerable peer 
review, and certifying the outcome with its name and reputation, 
we already have a perfectly adequate name for that sort of entity 
and service: it is called a journal...

Stevan Harnad