Most ACM SIG conferences are single-blind: the authors don't know the
reviewers (generally PC members), but the reviewers know the
author(s)' identity. Some conferences use double-blind reviewing, in
which the identity of the authors is hidden from the reviewers,
ostensibly to improve the objectivity of the reviews. In a few others,
the identity of the authors is hidden to the reviewers, but is
revealed at the PC meeting.
ACM sponsors about 80 conferences annually; about 20% are
double-blind. In discussions with other SIG chairs, they appear to be
very satisfied with those conferences that are double-blind. However,
there are passionate voices o n both sides of the debate.
There are several advantages to double-blind reviewing. The primary
objective is to raise the quality of reviewing by removing, when
possible, bias based on the authors of the paper. This seems to occur
in several ways: it is easier for the reviewers to be honest if the
names of the authors are not staring them in the face, it is easier
for the program committee to make tough decisions if they believe that
the reviewers did not know the authors, and there is some evidence
that this process is more fair to authors with obviously female
names. It also has a major perceptual advantage, in that it makes the
process seem more objective, which encourages young authors and
less-published authors to submit papers, and reduces the perception
that there is an insiders club. It seems to raise the quality of the
conference; many conferences employing double-blind reviewing are
considered to be the top conference in their area, e.g., CHI, DAC,
ICSA, SIGCOMM and SOSP, though it is not clear which way the causality
goes. Finally, there is evidence that prospective authors overwhelming
want double-blind reviewing.
Double-blind reviewing comes with some disadvantages. It is an extra
burden on authors, who must revise their papers with care to hide
their identity, and on the program chair. It is often not hard for
reviewers to guess the authors, circumventing the process. Well-known
authors can use tricks to reveal themselves (such as whitening out a
bibliographic entry but leaving it in alphabetical order). Calls for
papers need to be carefully written to preclude (as much as possible)
such tricks. Note however that even when the authors can be guessed,
the guess is not certain, and there are sometimes surprises. Some
argue that reviewing is already fair, and that double-blind reviewing
won't increase the quality of reviewing. Some go the other direction
and argue that author identity provides valuable clues as to the
quality of the research, and that double-blind reviewing teaches young
people that only ideas matter, rather than that persuasion and author
credibility are important components in the process of science.
After discussions with many within SIGMOD, the majority of whom have
requested double-blind reviewing, I've decided to have the SIGMOD
conference try this for two years, after which the experiment will be
evaluated, and a decision made to continue or to revert to the current
method of single-blind reviewing. We'll start with the SIGMOD'01
conference. It will be interesting to see if the advantages in fact
outweigh the disadvantages.
Speaking of conferences, I hope that you have registered for the SIGMOD'00 conference,
to be held in Dallas on May 14-19, with the co-convened PODS'00
conference and several workshops before and after. When you come,
don't forget to bring a spare database textbook to donate, helping
disseminate the ideas our community has originated and nurtured. See
you in Texas!
| Rick Snodgrass |
| January, 2000 |
|
